From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-iy0-f171.google.com (mail-iy0-f171.google.com [209.85.210.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id D8850201146 for ; Wed, 21 Sep 2011 17:18:02 -0700 (PDT) Received: by iagv1 with SMTP id v1so4042118iag.16 for ; Wed, 21 Sep 2011 17:18:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=aYHC+aMBfeGe2t2ipP0wfILeqXtADgmraLMD5O7QUIo=; b=XRZJuTjMwMhGAMMiE6Wh7lHDW6Rfv7om3NyOkvG1YAB7/SM+25+FF5bR7GLlwf+q4g xgx/s1zLlhaulOAqB5vPiqZvCywkueVm5JlIfYFMlYo0YsyxZKd2mQgSZXfDZyhVRvH3 4VqbJT1oNgt91b9YlleJHS6pLtRxdwwn7vwDE= MIME-Version: 1.0 Received: by 10.42.75.8 with SMTP id y8mr974931icj.67.1316650681710; Wed, 21 Sep 2011 17:18:01 -0700 (PDT) Received: by 10.43.132.8 with HTTP; Wed, 21 Sep 2011 17:18:01 -0700 (PDT) In-Reply-To: <20110921230205.2275820C2E5@snark.thyrsus.com> References: <20110921230205.2275820C2E5@snark.thyrsus.com> Date: Wed, 21 Sep 2011 17:18:01 -0700 Message-ID: Subject: Re: Preliminary results of using GPS to look for clock skew From: Dave Taht To: Eric Raymond Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Hal Murray , bloat-devel@lists.bufferbloat.net X-BeenThere: bloat-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Developers working on AQM, device drivers, and networking stacks" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2011 00:18:03 -0000 Eric: It is comforting to know that ntp is working well in your case, and, using = GPS, we have a verifiable means with decent error bars of checking against ntp's algos independently! Two ideas here: 1) Run the router WITHOUT ntp enabled at all (and/or testing against CLOCK_REALTIME) It would be good to know how much the base clock drift is, without correction. 2) ReRun all tests under load (example: netperf -l 3600 -H the_router) The followon test to this is to actually start collecting and parsing ntp rawstats statistics, which can be easily turned on and collected on the router. It's getting-those-stats-somewhere coupled with the need to periodically delete these statistic files that's a problem at the moment, and really only the former... The bufferbloat signal (if it exists) is in the noise that ntp is currently (successfully in your case) rejecting. There are a couple rawstats parsers floating about, I have part of one, hal has another. I committed a major overdesign sin in mine by wanting to put it all into a postgres db, ran into major data representation problems (time on postgres is different than time inside of ntp), and put the work aside (it's on github in the same pieces I left it in) To enable rawstats collection on the router, modify /etc/ntp.conf to contai= n: statsdir /tmp statistics rawstats filegen rawstats file rawstats type day enable and restart ntp on a system protected by apparmor - like ubuntu - it's mildly trickier as you need to add a whereverthelogdiris/rawstats* rwl to the /etc/apparmor.d/usr.sbin.ntpd The final bit of the cbbd is to actually collect port numbers - so stuff on ephemeral ports is known to be from natted devices and stuff on 123 but I'm getting way ahead of myself here. On Wed, Sep 21, 2011 at 4:02 PM, Eric Raymond wrote= : > As a step toward implementation of Dave's Cosmic Background > Bufferbloat Detector, I took on the job of employing GPS as a reliable > local time source to check whether an NTP-assisted system clock has > significant skew from GPS. > > Towards this end, I've just spent a couple of days rebuilding and > improving the latency-profiling machinery in GPSD. =A0What this allows > me to do is collate the following pieces of data: > > * The GPS's time of a fix in properly leapsecond-corrected UTC (Call this= T) > > * The NTP-corrected system time at which the first burst of > =A0fix data from the device wakes up gpsd's select(2). =A0Call this > =A0S (Start of reporting cycle) > > * When the daemon has received and processed the entire burst of packets > =A0constituting a reporting cycle and is about to ship JSON to the client= . > =A0Call this E (End of reporting cycle). > > * When the client recieves the report. =A0(R =3D receipt time). > > What's interesting is to plot the deltas S - T, E - S, and R - S as a > stacked-impulse graph. =A0I can do this at will now. =A0More importantly, > so can anyone with my software and GPSD. =A0We have a working GPSD port t= o > CeroWRT, so our test routers can be instrumented. > > The entire height R-T approximates the entire fix latency. I say > "approximates" because, of course, T is on a different timebase than > R, S, and E, and NTP's 10ms fuzz is an issue. =A0But S-T gives us an > idea of the atomic-clock-vs-system-clock time. > > There's nothing surprising about my setup - FIOS to an N600 running a > recent OpenWRT build to a 2.66GHz Intel Core Duo 2 machine running Ubuntu > 10.04 LTS. > > Over runs of 20-100 fixes, S-T (the clock-skew figure) ranges from 66 > to 76 milliseconds. =A0Total latency R-T is steady at about 0.38 > seconds. =A0R-S vanishes - it's close to 0.9 milliseconds. =A0Total fix > latency is completely dominated by E-S, the time for gpsd to receive > and process the fix data from the GPS. =A0And I'm not seeing a lot of > bursty variation in these measurements. > > I'm going to investigate further. =A0In particular, I'm going to try to > either measure or compute how much of E-S is I/O time. =A0My suspicion is= that > almost all of it is, and that gpsd's actual computation time is negligibl= e > by comparison. > > But preliminary indications are that NTP is in fact doing a pretty good > job of keeping my system clock conditioned. I'm not seeing bufferbloat > effects on time service. > > However, I'd welcome having my code and assumptions checked. This kind > of profiling is tricky work, with large vulnerabilities to small > mistakes in detail. That's the main reason I'm describing these > results as 'preliminary'; some skeptical review is needed to soliudify > and verify them. > -- > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Eric= S. Raymond > > Let us hope our weapons are never needed --but do not forget what > the common people knew when they demanded the Bill of Rights: An > armed citizenry is the first defense, the best defense, and the > final defense against tyranny. > =A0 If guns are outlawed, only the government will have guns. Only > the police, the secret police, the military, the hired servants of > our rulers. =A0Only the government -- and a few outlaws. =A0I intend to > be among the outlaws. > =A0 =A0 =A0 =A0-- Edward Abbey, "Abbey's Road", 1979 > --=20 Dave T=E4ht SKYPE: davetaht US Tel: 1-239-829-5608 http://the-edge.blogspot.com