smoketest #6 of cerowrt is go for testing

smoketest #6 of cerowrt is go for testing

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 3469 bytes --]

GET IT AT:

http://huchra.bufferbloat.net/~cerowrt/cerowrt-wndr3700/


Noted thus far:

0) We have a priority 3 issue with ntp starting up. NTP times out too early
it keeps restarting, and restarting, and restarting... (it's on a 10 second
timer) This is related to the dnssec issue I'm well aware of. It looks like
60 second timer would work better)

(dnssec IS disabled by default in this build, so ntp isn't even needed
and does eventually get the correct time if you kill off named.montime - and
the right answer is to patch ntp to just issue the first queries for

ntp -g option

with the auth-needed bit off... I have NO idea how to do that with dnssec.

ntp even tries ipv6 but doesn't get out of gatech... (gatech's problem, not
me, I hope!)

oot@OpenWrt:/etc/xinetd.d# ntpdc -n -p
     remote           local      st poll reach  delay   offset    disp
=======================================================================
=69.167.160.102  192.168.22.222   2   64  377 0.05031 -0.001173 0.05679
=207.171.17.42   192.168.22.222  16 1024    0 0.00000  0.000000 3.99217
=66.96.110.10    192.168.22.222   2   64  377 0.07285 -0.004067 0.04944
=2001:1868:213:5 2002:82cf:6116: 16 1024    0 0.00000  0.000000 3.99217
=2001:470:8c8e:0 2002:82cf:6116: 16 1024    0 0.00000  0.000000 3.99217
=173.203.122.111 192.168.22.222   3   64  377 0.04329 -0.004989 0.06305
*66.228.39.48    192.168.22.222   2   64  377 0.02226 -0.003478 0.03444



1) uftp: built, entirely untested

2) netserver: small bug in xinetd - wrong directory for it, had to move it:

root@OpenWrt:/etc/xinetd.d# mv ../xinet.d/netserver .

(hint: I am too tired and dangers to fix this in git myself)

d@cruithne:~$ netperf 172.30.42.33
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost
(127.0.0.1) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.00    10394.62


3) And ditg guys.? THANK YOU  FOR THE FLASH!

With a full build with all the major extra tools installed (your new stuff,
plus bismark, plus openvpn and iperf), flash requirements have been reduced
by nearly 8% from the last build, thanks to your work. THANK YOU.

/dev/root                14848     10028      4820  68% /
tmpfs                    30888       436     30452   1% /tmp
tmpfs                      512         0       512   0% /dev


4) Debloat: disabled. We ran into a problem with reducing buffers via
ethtool to low levels on the gige port. It would get 'stuck' after 7 packets
or so, when the ge00 driver was reduced to 4 packets.

so that portion of the debloating stuff is *entirely disabled* right now.

I'm bugged we had to do this for this smoke test, and I intend to fix it as
soon as I have some brain cells in the morning. Since we now have a
reproducable test case for the problem, we're in great shape, for that, or
so I hope.

I'm goin to bed.

please test web, polipo, wired, wireless, encrypted wireless, 5 ghz, 2.4ghz,
the internet, ipv6, your tools, my tools, whatever you can think of, AND
FILE BUGS, and let me know if any other kittens are eaten by 1PM saturday
afternoon EST, at which I'm hoping to pull together RC-1 for wider use.

Thanks for the smoketesting.


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 3951 bytes --]

Re: smoketest #6 of cerowrt is go for testing

[Dave Hart]

On Sat, Jul 16, 2011 at 02:40 UTC, Dave Taht <dave.taht@gmail.com> wrote:
> 0) We have a priority 3 issue with ntp starting up. NTP times out too early
> it keeps restarting, and restarting, and restarting... (it's on a 10 second
> timer) This is related to the dnssec issue I'm well aware of. It looks like
> 60 second timer would work better)

10 seconds is too small, but sometimes one minute might be too small a
timeout, too.  If possible, it may make sense to start ntpd, then
start time-step-tolerant daemons, then use the ntp-wait script to
block until ntpd's first clock sync, then start time-step-sensitive
daemons.

> (dnssec IS disabled by default in this build, so ntp isn't even needed
> and does eventually get the correct time if you kill off named.montime - and
> the right answer is to patch ntp to just issue the first queries for
>
> ntp -g option
>
> with the auth-needed bit off... I have NO idea how to do that with dnssec.

ntpd doesn't request dnssec verification -- it simply uses the
provided getaddrinfo().  Any tweaking of the resolver to avoid
requiring DNSSEC validation of queries will be outside the ntpd source
code.

Cheers,
Dave Hart

Re: smoketest #6 of cerowrt is go for testing

[Rick Jones]

>
> ntpd doesn't request dnssec verification -- it simply uses the
> provided getaddrinfo().  Any tweaking of the resolver to avoid
> requiring DNSSEC validation of queries will be outside the ntpd source
> code.
>

If you configure ntpd with bare IP addresses rather than names, will the 
getaddrinfo() return without attempting any DNS in the first place?

rick jones

Re: smoketest #6 of cerowrt is go for testing

[Dave Hart]

On Sun, Jul 17, 2011 at 00:02 UTC, Rick Jones <rick.jones2@hp.com> wrote:
> If you configure ntpd with bare IP addresses rather than names, will the
> getaddrinfo() return without attempting any DNS in the first place?

Yes, basically.  ntpd might not even call getaddrinfo() in that case
(it may use inet_pton() or similar to convert the IP address to binary
representation).  At any rate, using only numeric IPv4 or IPv6
addresses will avoid any DNS lookups.

Cheers,
Dave Hart

Re: smoketest #6 of cerowrt is go for testing

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 3125 bytes --]

On Sat, Jul 16, 2011 at 10:35 PM, Dave Hart <
davehart_gmail_exchange_tee@davehart.net> wrote:

> On Sun, Jul 17, 2011 at 00:02 UTC, Rick Jones <rick.jones2@hp.com> wrote:
> > If you configure ntpd with bare IP addresses rather than names, will the
> > getaddrinfo() return without attempting any DNS in the first place?
>
> Yes, basically.  ntpd might not even call getaddrinfo() in that case
> (it may use inet_pton() or similar to convert the IP address to binary
> representation).  At any rate, using only numeric IPv4 or IPv6
> addresses will avoid any DNS lookups.
>

While there is one group that is finally providing ntp time via anycast -
which is a good solution to a large extent! - there is only the one (small)
group doing so, rather than the needed '3'.

http://news.ntppool.org/2011/03/expanding-the-anycast-dns-serv.html

And I'm reluctant, given the sordid history of hard coding ntp IP addresses,


http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse

to hard code *any* until far more anycast servers are online.

To take a step backwards on this, there are extensive notes on the circular
dependencies between time and dnssec logged here.

http://www.bufferbloat.net/issues/205

I'd implemented a hack to try to address these circular depenencies last
week in the named-latest package repo, while also coping with

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2464

I think I addressed the latter issue, but *good*. :) The 'fix' for the
ntp/dnssec/bind/network dependencies seems to have some problems, however,
notably really slow startup in general.


To step further back on this:

I had implemented ntp (with 7 contacted servers in the conf file!) in the
first place due to the "cosmic background bufferbloat detector" idea
extensively discussed on the comp.protocols.ntp newsgroup, and because I
wanted to be able to compare large sample data sets against known-to-be
accurate time, with a large deployment of client routers that had a
configuration I could trust to be accurate, talking to a yet-to-be-deployed
string of ntp servers (via hopefully a helpful operator) that could work on
this with us.


We had implemented dnssec in the first place because we wanted more people
to be using it, and ironing out problems (among other things, I planned to
use it to ensure valid updates to the routers), and because of nonsense
about DNS censorship happing all over the world, such as the recent
shenanagans in Australia.

once all these circular dependencies are resolved on boot, which doesn't
always happen and seems to take minutes, regardless, dnssec works pretty
darn good. Seeing it actually work at all after a decade of discussion makes
me really, really happy, but making it work *well*, somehow, would be best.

It's also my hope to implement this fix to bind, in the next rc release of
cerowrt.

http://www.isc.org/community/blog/201107/major-improvement-bind-9-startup-performance


> Cheers,
> Dave Hart
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 4250 bytes --]

Fwd: smoketest #6 of cerowrt is go for testing

[Dave Hart]

On Sun, Jul 17, 2011 at 12:34 UTC, Dave Taht <dave.taht@gmail.com> wrote:
> On Sat, Jul 16, 2011 at 10:35 PM, Dave Hart
> <davehart_gmail_exchange_tee@davehart.net> wrote:
>>
>> On Sun, Jul 17, 2011 at 00:02 UTC, Rick Jones <rick.jones2@hp.com> wrote:
>> > If you configure ntpd with bare IP addresses rather than names, will the
>> > getaddrinfo() return without attempting any DNS in the first place?
>>
>> Yes, basically.  ntpd might not even call getaddrinfo() in that case
>> (it may use inet_pton() or similar to convert the IP address to binary
>> representation).  At any rate, using only numeric IPv4 or IPv6
>> addresses will avoid any DNS lookups.
>
> While there is one group that is finally providing ntp time via anycast -
> which is a good solution to a large extent! - there is only the one (small)
> group doing so, rather than the needed '3'.
>
> http://news.ntppool.org/2011/03/expanding-the-anycast-dns-serv.html

You misread the announcement:  What is anycast is the DNS service for
pool.ntp.org provided by (among others) a.ntpns.org, which is the
anycast IP now served from 3 sites.  Anycast is being used to spread
the load of the custom DNS server software used for *.pool.ntp.org.
NTP time service is not being anycast by the pool operators.

I do know of one operator who provides NTP time service over anycasted
IP addresses, but it is not a good solution in general.  For simple
clients it makes no difference, but for full NTP, a single IP address
is assumed to represent a single oscillator reached via a single
network path.  ntpd keeps an eight-deep history for each configured
server of the delay, root dispersion, and apparent offset.  NTP's
clock filter algorithm chooses the lowest-delay sample from this
history, which helps remove the majority of error caused by network
delay variation -- the lowest delay sample represents the best guess
at the actual path round-trip time.  Using an anycast IP address as a
ntpd time source violates the assumption and results in samples from
more than one oscillator reached via more than one path being compared
as if they represented a single clock and path.

> And I'm reluctant, given the sordid history of hard coding ntp IP addresses,
>
> http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse
>
> to hard code *any* until far more anycast servers are online.

If you are using well-behaved NTP clients and reasonable
configurations, each server can handle many thousands of clients, each
of which is polling every 1024 seconds once settled in.  For
bootstrapping, where you need to set the time in order to enable
DNSSEC in order to enable normal operation using pool.ntp.org, you can
reasonably get by with a handful of hardcoded NTP server IPs, assuming
you are doing a one-shot NTP query during startup then switching to
DNS and pool.ntp.org for ongoing clock sync.

> To take a step backwards on this, there are extensive notes on the circular
> dependencies between time and dnssec logged here.
>
> http://www.bufferbloat.net/issues/205
>
> I'd implemented a hack to try to address these circular depenencies last
> week in the named-latest package repo, while also coping with
>
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2464
>
> I think I addressed the latter issue, but *good*. :) The 'fix' for the
> ntp/dnssec/bind/network dependencies seems to have some problems, however,
> notably really slow startup in general.
>
>
> To step further back on this:
>
> I had implemented ntp (with 7 contacted servers in the conf file!) in the

IMO 7 is not an extreme number of NTP servers to associate with.  I
help operate a pool NTP server which has 15 associations configured
using maxpoll 8 (256s).  A handful of those are for monitoring
purposes, but the majority are there to provide a wide variety of
stable oscillators, in hopes of keeping better time by giving NTP the
information it needs to ignore transient outliers.  That still amounts
to an average of about 4 queries to lower-stratum "upstream" sources
per minute, which is trivial compared to the 20,000 client NTP queries
it serves each minute (currently averaging 363 pkts/sec).

> first place due to the "cosmic background bufferbloat detector" idea
> extensively discussed on the comp.protocols.ntp newsgroup, and because I
> wanted to be able to compare large sample data sets against known-to-be
> accurate time, with a large deployment of client routers that had a
> configuration I could trust to be accurate, talking to a yet-to-be-deployed
> string of ntp servers (via hopefully a helpful operator) that could work on
> this with us.

Even better would be comparing against a local NTP server set up with
a GPS + PPS (pulse-per-second).  Using off-the-shelf components, the
hardware cost in the US is between $30 and $100.   $30 gets you a Sure
GPS that has PPS but requires good eyes and a steady soldering hand to
bring it from the PCB to the RS-232 connection.  For less than $70 you
can get a Garmin GPS 18x LVC, which requires less accurate soldering
to provide 5VDC power (often via USB) and a DB-9 connector for its
bare-lead 232 connection.  For $100 you can get a ready-to-use 18x LVC
with DB-9 serial and USB power cables provided by a third party.

Cheers,
Dave Hart

Re: smoketest #6 of cerowrt is go for testing

[Evan Hunt]

DNSSEC doesn't need atomic clock accuracy.  Accurate to within an hour
is more than sufficient.

A bog-simple anycasted UDP time/daytime server could get the job done
beautifully.  Even SNTP is overkill for this.  We could use a time
server to get the clock into the right ballpark, then start named,
then start ntpd to set the clock for real.

Or NTP could make a leap-of-faith on startup and query for server
addresses with DNSSEC checking disabled.  There's no standard API for
that yet, though (dnssec-tools.org has proposed one, val_getaddrinfo()
and val_getnameinfo(), and there's a reference implementation, but I
don't think it's made it out into any distributions yet).

Or, named can run with the dnssec-accept-expired option set to yes,
causing it to ignore signature expiration times, then switch the
option off after the date has been set correctly--but this is
cumbersome (and leaves named open to replay attacks during the
initialization interval).

In the absence of a standard DNSSEC-aware gethostinfo() API, I like
the anycasted time server idea.  I'll see if I can interest anyone at
ISC in providing such a service.  (And maybe the ntpns.org name
servers could provide it, since they're anycasted already?  I'm not
sure whom to ask.)

Evan

On Sun, Jul 17, 2011 at 8:50 AM, Dave Hart
<davehart_gmail_exchange_tee@davehart.net> wrote:
> On Sun, Jul 17, 2011 at 12:34 UTC, Dave Taht <dave.taht@gmail.com> wrote:
>> On Sat, Jul 16, 2011 at 10:35 PM, Dave Hart
>> <davehart_gmail_exchange_tee@davehart.net> wrote:
>>>
>>> On Sun, Jul 17, 2011 at 00:02 UTC, Rick Jones <rick.jones2@hp.com> wrote:
>>> > If you configure ntpd with bare IP addresses rather than names, will the
>>> > getaddrinfo() return without attempting any DNS in the first place?
>>>
>>> Yes, basically.  ntpd might not even call getaddrinfo() in that case
>>> (it may use inet_pton() or similar to convert the IP address to binary
>>> representation).  At any rate, using only numeric IPv4 or IPv6
>>> addresses will avoid any DNS lookups.
>>
>> While there is one group that is finally providing ntp time via anycast -
>> which is a good solution to a large extent! - there is only the one (small)
>> group doing so, rather than the needed '3'.
>>
>> http://news.ntppool.org/2011/03/expanding-the-anycast-dns-serv.html
>
> You misread the announcement:  What is anycast is the DNS service for
> pool.ntp.org provided by (among others) a.ntpns.org, which is the
> anycast IP now served from 3 sites.  Anycast is being used to spread
> the load of the custom DNS server software used for *.pool.ntp.org.
> NTP time service is not being anycast by the pool operators.
>
> I do know of one operator who provides NTP time service over anycasted
> IP addresses, but it is not a good solution in general.  For simple
> clients it makes no difference, but for full NTP, a single IP address
> is assumed to represent a single oscillator reached via a single
> network path.  ntpd keeps an eight-deep history for each configured
> server of the delay, root dispersion, and apparent offset.  NTP's
> clock filter algorithm chooses the lowest-delay sample from this
> history, which helps remove the majority of error caused by network
> delay variation -- the lowest delay sample represents the best guess
> at the actual path round-trip time.  Using an anycast IP address as a
> ntpd time source violates the assumption and results in samples from
> more than one oscillator reached via more than one path being compared
> as if they represented a single clock and path.
>
>> And I'm reluctant, given the sordid history of hard coding ntp IP addresses,
>>
>> http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse
>>
>> to hard code *any* until far more anycast servers are online.
>
> If you are using well-behaved NTP clients and reasonable
> configurations, each server can handle many thousands of clients, each
> of which is polling every 1024 seconds once settled in.  For
> bootstrapping, where you need to set the time in order to enable
> DNSSEC in order to enable normal operation using pool.ntp.org, you can
> reasonably get by with a handful of hardcoded NTP server IPs, assuming
> you are doing a one-shot NTP query during startup then switching to
> DNS and pool.ntp.org for ongoing clock sync.
>
>> To take a step backwards on this, there are extensive notes on the circular
>> dependencies between time and dnssec logged here.
>>
>> http://www.bufferbloat.net/issues/205
>>
>> I'd implemented a hack to try to address these circular depenencies last
>> week in the named-latest package repo, while also coping with
>>
>> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2464
>>
>> I think I addressed the latter issue, but *good*. :) The 'fix' for the
>> ntp/dnssec/bind/network dependencies seems to have some problems, however,
>> notably really slow startup in general.
>>
>>
>> To step further back on this:
>>
>> I had implemented ntp (with 7 contacted servers in the conf file!) in the
>
> IMO 7 is not an extreme number of NTP servers to associate with.  I
> help operate a pool NTP server which has 15 associations configured
> using maxpoll 8 (256s).  A handful of those are for monitoring
> purposes, but the majority are there to provide a wide variety of
> stable oscillators, in hopes of keeping better time by giving NTP the
> information it needs to ignore transient outliers.  That still amounts
> to an average of about 4 queries to lower-stratum "upstream" sources
> per minute, which is trivial compared to the 20,000 client NTP queries
> it serves each minute (currently averaging 363 pkts/sec).
>
>> first place due to the "cosmic background bufferbloat detector" idea
>> extensively discussed on the comp.protocols.ntp newsgroup, and because I
>> wanted to be able to compare large sample data sets against known-to-be
>> accurate time, with a large deployment of client routers that had a
>> configuration I could trust to be accurate, talking to a yet-to-be-deployed
>> string of ntp servers (via hopefully a helpful operator) that could work on
>> this with us.
>
> Even better would be comparing against a local NTP server set up with
> a GPS + PPS (pulse-per-second).  Using off-the-shelf components, the
> hardware cost in the US is between $30 and $100.   $30 gets you a Sure
> GPS that has PPS but requires good eyes and a steady soldering hand to
> bring it from the PCB to the RS-232 connection.  For less than $70 you
> can get a Garmin GPS 18x LVC, which requires less accurate soldering
> to provide 5VDC power (often via USB) and a DB-9 connector for its
> bare-lead 232 connection.  For $100 you can get a ready-to-use 18x LVC
> with DB-9 serial and USB power cables provided by a third party.
>
> Cheers,
> Dave Hart
>

Re: smoketest #6 of cerowrt is go for testing

[Rick Jones]

On 07/16/2011 09:35 PM, Dave Hart wrote:
> On Sun, Jul 17, 2011 at 00:02 UTC, Rick Jones<rick.jones2@hp.com>  wrote:
>> If you configure ntpd with bare IP addresses rather than names, will the
>> getaddrinfo() return without attempting any DNS in the first place?
>
> Yes, basically.  ntpd might not even call getaddrinfo() in that case
> (it may use inet_pton() or similar to convert the IP address to binary
> representation).  At any rate, using only numeric IPv4 or IPv6
> addresses will avoid any DNS lookups.

If it does still call getaddrinfo(), like netperf does, and if it does 
happen to set AI_ADDRCONFIG, like netperf does, then the getaddrinfo() 
call for a bare IPv6 address will fail if there are only link-scope IPv6 
addresses assigned and the getaddrinfo() is the same as Ubuntu's.  This 
seems to be a change in the linux space since 2008, although I cannot 
say that conclusively because in 2008, netperf wasn't setting 
AI_ADDRCONFIG. But I have vague recollections of the calls working under 
linux since AI_ADDRCONFIG was set in netperf.

rick jones

Re: smoketest #6 of cerowrt is go for testing

[Rick Jones]

> 2) netserver: small bug in xinetd - wrong directory for it, had to move it:
>
> root@OpenWrt:/etc/xinetd.d# mv ../xinet.d/netserver .
>
> (hint: I am too tired and dangers to fix this in git myself)
>
> d@cruithne:~$ netperf 172.30.42.33
> MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> localhost (127.0.0.1) port 0 AF_INET
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
>   87380  16384  16384    10.00    10394.62

Notice that the test banner talks about going to 127.0.0.1 rather than 
172.40.42.33 - this stems from missing the -H required in front of the 
IP address.  The command line should be:

netperf -H  172.30.42.33

happy benchmarking,

rick jones