From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DD3FD3B2A4; Fri, 13 Jan 2023 02:40:17 -0500 (EST) Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id D6CB41B252; Thu, 12 Jan 2023 23:40:16 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com D6CB41B252 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1673595616; bh=9T7Xmh3BsGCObznJBiwWFxFRqLW5j4FO31l/3BMWbKE=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=JLcY9xDsmheTCFjys6Ai2GvEyJCfB/mGIvKD9UmP9f9OienRgqg19AggeQtNInFRx a8kzLzERdCkJiUlNJNKx5QHF61866v3J3iOmQw6bYZcynzeBndYKvJi/BZFjqI4GrB iUzMb5wOOVJYKPsII5dD9SK5z6z3vR3PgC2VYUww= MIME-Version: 1.0 Date: Thu, 12 Jan 2023 23:40:16 -0800 From: rjmcmahon To: dickroy@alum.mit.edu Cc: 'Sebastian Moeller' , "'Rodney W. Grimes'" , mike.reynolds@netforecast.com, 'libreqos' , "'David P. Reed'" , 'Rpm' , 'bloat' In-Reply-To: References: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> <5C6EFF20C9F745AC87DA199EE7ABBAD9@SRA6> Message-ID: <59f7d0465d74fa6b72587895a23edd4b@rjmcmahon.com> X-Sender: rjmcmahon@rjmcmahon.com Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Rpm] [Starlink] Researchers Seeking Probe Volunteers in USA X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2023 07:40:18 -0000 Hi RR, I believe quality GPS chips compensate for relativity in pulse per second which is needed to get position accuracy. Bob > Hi Sebastian (et. al.), > > [I'll comment up here instead of inline.] > > Let me start by saying that I have not been intimately involved with > the IEEE 1588 effort (PTP), however I was involved in the 802.11 > efforts along a similar vein, just adding the wireless first hop > component and it's effects on PTP. > > What was apparent from the outset was that there was a lack of > understanding what the terms "to synchronize" or "to be synchronized" > actually mean. It's not trivial … because we live in a > (approximately, that's another story!) 4-D space-time continuum where > the Lorentz metric plays a critical role. Therein, simultaneity (aka > "things happening at the same time") means the "distance" between two > such events is zero and that distance is given by sqrt(x^2 + y^2 + z^2 > - (ct)^2) and the "thing happening" can be the tick of a clock > somewhere. Now since everything is relative (time with respect to > what? / location with respect to where?) it's pretty easy to see that > "if you don't know where you are, you can't know what time it is!" > (English sailors of the 18th century knew this well!) Add to this the > fact that if everything were stationary, nothing would happen (as > Einstein said "Nothing happens until something moves!"), special > relativity also pays a role. Clocks on GPS satellites run approx. > 7usecs/day slower than those on earth due to their "speed" (8700 mph > roughly)! Then add the consequence that without mass we wouldn't exist > (in these forms at leastJ), and gravitational effects (aka General > Relativity) come into play. Those turn out to make clocks on GPS > satellites run 45usec/day faster than those on earth! The net effect > is that GPS clocks run about 38usec/day faster than clocks on earth. > So what does it mean to "synchronize to GPS"? Point is: it's a > non-trivial question with a very complicated answer. The reason it is > important to get all this right is that the "what that ties time and > space together" is the speed of light and that turns out to be a > "foot-per-nanosecond" in a vacuum (roughly 300m/usec). This means if > I am uncertain about my location to say 300 meters, then I also am not > sure what time it is to a usec AND vice-versa! > > All that said, the simplest explanation of synchronization is > probably: Two clocks are synchronized if, when they are brought > (slowly) into physical proximity ("sat next to each other") in the > same (quasi-)inertial frame and the same gravitational potential (not > so obvious BTW … see the FYI below!), an observer of both would say > "they are keeping time identically". Since this experiment is rarely > possible, one can never be "sure" that his clock is synchronized to > any other clock elsewhere. And what does it mean to say they "were > synchronized" when brought together, but now they are not because they > are now in different gravitational potentials! (FYI, there are land > mine detectors being developed on this very principle! I know someone > who actually worked on such a project!) > > This all gets even more complicated when dealing with large networks > of networks in which the "speed of information transmission" can vary > depending on the medium (cf. coaxial cables versus fiber versus > microwave links!) In fact, the atmosphere is one of those media and > variations therein result in the need for "GPS corrections" (cf. RTCM > GPS correction messages, RTK, etc.) in order to get to sub-nsec/cm > accuracy. Point is if you have a set of nodes distributed across the > country all with GPS and all "synchronized to GPS time", and a second > identical set of nodes (with no GPS) instead connected with a network > of cables and fiber links, all of different lengths and composition > using different carrier frequencies (dielectric constants vary with > frequency!) "synchronized" to some clock somewhere using NTP or PTP), > the synchronization of the two sets will be different unless a common > reference clock is used AND all the above effects are taken into > account, and good luck with that! J > > In conclusion, if anyone tells you that clock synchronization in > communication networks is simple ("Just use GPS!"), you should feel > free to chuckle (under your breath if necessaryJ) > > Cheers, > > RR > > -----Original Message----- > From: Sebastian Moeller [mailto:moeller0@gmx.de] > Sent: Thursday, January 12, 2023 12:23 AM > To: Dick Roy > Cc: Rodney W. Grimes; mike.reynolds@netforecast.com; libreqos; David > P. Reed; Rpm; rjmcmahon; bloat > Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in > USA > > Hi RR, > >> On Jan 11, 2023, at 22:46, Dick Roy wrote: > >> > >> > >> > >> -----Original Message----- > >> From: Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On > Behalf Of Sebastian Moeller via Starlink > >> Sent: Wednesday, January 11, 2023 12:01 PM > >> To: Rodney W. Grimes > >> Cc: Dave Taht via Starlink; mike.reynolds@netforecast.com; libreqos; > David P. Reed; Rpm; rjmcmahon; bloat > >> Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers > in USA > >> > >> Hi Rodney, > >> > >> > >> > >> > >> > On Jan 11, 2023, at 19:32, Rodney W. Grimes > wrote: > >> > > >> > Hello, > >> > > >> > Yall can call me crazy if you want.. but... see below [RWG] > >> >> Hi Bib, > >> >> > >> >> > >> >>> On Jan 9, 2023, at 20:13, rjmcmahon via Starlink > wrote: > >> >>> > >> >>> My biggest barrier is the lack of clock sync by the devices, > i.e. very limited support for PTP in data centers and in end devices. > This limits the ability to measure one way delays (OWD) and most > assume that OWD is 1/2 and RTT which typically is a mistake. We know > this intuitively with airplane flight times or even car commute times > where the one way time is not 1/2 a round trip time. Google maps & > directions provide a time estimate for the one way link. It doesn't > compute a round trip and divide by two. > >> >>> > >> >>> For those that can get clock sync working, the iperf 2 > --trip-times options is useful. > >> >> > >> >> [SM] +1; and yet even with unsynchronized clocks one can try > to measure how latency changes under load and that can be done per > direction. Sure this is far inferior to real reliably measured OWDs, > but if life/the internet deals you lemons.... > >> > > >> > [RWG] iperf2/iperf3, etc are already moving large amounts of data > back and forth, for that matter any rate test, why not abuse some of > that data and add the fundemental NTP clock sync data and > bidirectionally pass each others concept of "current time". IIRC (its > been 25 years since I worked on NTP at this level) you *should* be > able to get a fairly accurate clock delta between each end, and then > use that info and time stamps in the data stream to compute OWD's. > You need to put 4 time stamps in the packet, and with that you can > compute "offset". > >> [RR] For this to work at a reasonable level of accuracy, the > timestamping circuits on both ends need to be deterministic and > repeatable as I recall. Any uncertainty in that process adds to > synchronization errors/uncertainties. > >> > >> [SM] Nice idea. I would guess that all timeslot based access > technologies (so starlink, docsis, GPON, LTE?) all distribute "high > quality time" carefully to the "modems", so maybe all that would be > needed is to expose that high quality time to the LAN side of those > modems, dressed up as NTP server? > >> [RR] It's not that simple! Distributing "high-quality time", i.e. > "synchronizing all clocks" does not solve the communication problem in > synchronous slotted MAC/PHYs! > > [SM] I happily believe you, but the same idea of "time slot" > needs to be shared by all nodes, no? So the clockss need to be > reasonably similar rate, aka synchronized (see below). > >> All the technologies you mentioned above are essentially P2P, not > intended for broadcast. Point is, there is a point controller (aka > PoC) often called a base station (eNodeB, gNodeB, …) that actually > "controls everything that is necessary to control" at the UE including > time, frequency and sampling time offsets, and these are critical to > get right if you want to communicate, and they are ALL subject to the > laws of physics (cf. the speed of light)! Turns out that what is > necessary for the system to function anywhere near capacity, is for > all the clocks governing transmissions from the UEs to be > "unsynchronized" such that all the UE transmissions arrive at the PoC > at the same (prescribed) time! > > [SM] Fair enough. I would call clocks that are "in sync" albeit > with individual offsets as synchronized, but I am a layman and that > might sound offensively wrong to experts in the field. But even > without the naming my point is that all systems that depend on some > idea of shared time-base are halfway there of exposing that time to > end users, by "translating it into an NTP time source at the modem. > >> For some technologies, in particular 5G!, these considerations are > ESSENTIAL. Feel free to scour the 3GPP LTE 5G RLC and PHY specs if you > don't believe me! J > > [SM Far be it from me not to believe you, so thanks for the > pointers. Yet, I still think that unless different nodes of a shared > segment move at significantly different speeds, that there should be a > common "tick-duration" for all clocks even if each clock runs at an > offset... (I naively would try to implement something like that by > trying to fully synchronize clocks and maintain a local offset value > to convert from "absolute" time to "network" time, but likely because > coming from the outside I am blissfully unaware of the detail > challenges that need to be solved). > > Regards & Thanks > > Sebastian > >> > >> > >> > > >> >> > >> >> > >> >>> > >> >>> --trip-times > >> >>> enable the measurement of end to end write to read latencies > (client and server clocks must be synchronized) > >> > [RWG] --clock-skew > >> > enable the measurement of the wall clock difference between > sender and receiver > >> > > >> >> > >> >> [SM] Sweet! > >> >> > >> >> Regards > >> >> Sebastian > >> >> > >> >>> > >> >>> Bob > >> >>>> I have many kvetches about the new latency under load tests > being > >> >>>> designed and distributed over the past year. I am delighted! > that they > >> >>>> are happening, but most really need third party evaluation, and > > >> >>>> calibration, and a solid explanation of what network > pathologies they > >> >>>> do and don't cover. Also a RED team attitude towards them, as > well as > >> >>>> thinking hard about what you are not measuring (operations > research). > >> >>>> I actually rather love the new cloudflare speedtest, because it > tests > >> >>>> a single TCP connection, rather than dozens, and at the same > time folk > >> >>>> are complaining that it doesn't find the actual "speed!". > yet... the > >> >>>> test itself more closely emulates a user experience than > speedtest.net > >> >>>> does. I am personally pretty convinced that the fewer numbers > of flows > >> >>>> that a web page opens improves the likelihood of a good user > >> >>>> experience, but lack data on it. > >> >>>> To try to tackle the evaluation and calibration part, I've > reached out > >> >>>> to all the new test designers in the hope that we could get > together > >> >>>> and produce a report of what each new test is actually doing. > I've > >> >>>> tweeted, linked in, emailed, and spammed every measurement list > I know > >> >>>> of, and only to some response, please reach out to other test > designer > >> >>>> folks and have them join the rpm email list? > >> >>>> My principal kvetches in the new tests so far are: > >> >>>> 0) None of the tests last long enough. > >> >>>> Ideally there should be a mode where they at least run to "time > of > >> >>>> first loss", or periodically, just run longer than the > >> >>>> industry-stupid^H^H^H^H^H^Hstandard 20 seconds. There be > dragons > >> >>>> there! It's really bad science to optimize the internet for 20 > >> >>>> seconds. It's like optimizing a car, to handle well, for just > 20 > >> >>>> seconds. > >> >>>> 1) Not testing up + down + ping at the same time > >> >>>> None of the new tests actually test the same thing that the > infamous > >> >>>> rrul test does - all the others still test up, then down, and > ping. It > >> >>>> was/remains my hope that the simpler parts of the flent test > suite - > >> >>>> such as the tcp_up_squarewave tests, the rrul test, and the > rtt_fair > >> >>>> tests would provide calibration to the test designers. > >> >>>> we've got zillions of flent results in the archive published > here: > >> >>>> https://blog.cerowrt.org/post/found_in_flent/ > >> >>>> ps. Misinformation about iperf 2 impacts my ability to do this. > > >> >>> > >> >>>> The new tests have all added up + ping and down + ping, but not > up + > >> >>>> down + ping. Why?? > >> >>>> The behaviors of what happens in that case are really > non-intuitive, I > >> >>>> know, but... it's just one more phase to add to any one of > those new > >> >>>> tests. I'd be deliriously happy if someone(s) new to the field > >> >>>> started doing that, even optionally, and boggled at how it > defeated > >> >>>> their assumptions. > >> >>>> Among other things that would show... > >> >>>> It's the home router industry's dirty secret than darn few > "gigabit" > >> >>>> home routers can actually forward in both directions at a > gigabit. I'd > >> >>>> like to smash that perception thoroughly, but given our > starting point > >> >>>> is a gigabit router was a "gigabit switch" - and historically > been > >> >>>> something that couldn't even forward at 200Mbit - we have a > long way > >> >>>> to go there. > >> >>>> Only in the past year have non-x86 home routers appeared that > could > >> >>>> actually do a gbit in both directions. > >> >>>> 2) Few are actually testing within-stream latency > >> >>>> Apple's rpm project is making a stab in that direction. It > looks > >> >>>> highly likely, that with a little more work, crusader and > >> >>>> go-responsiveness can finally start sampling the tcp RTT, loss > and > >> >>>> markings, more directly. As for the rest... sampling TCP_INFO > on > >> >>>> windows, and Linux, at least, always appeared simple to me, but > I'm > >> >>>> discovering how hard it is by delving deep into the rust behind > > >> >>>> crusader. > >> >>>> the goresponsiveness thing is also IMHO running WAY too many > streams > >> >>>> at the same time, I guess motivated by an attempt to have the > test > >> >>>> complete quickly? > >> >>>> B) To try and tackle the validation problem:ps. Misinformation > about iperf 2 impacts my ability to do this. > >> >>> > >> >>>> In the libreqos.io project we've established a testbed where > tests can > >> >>>> be plunked through various ISP plan network emulations. It's > here: > >> >>>> https://payne.taht.net (run bandwidth test for what's currently > hooked > >> >>>> up) > >> >>>> We could rather use an AS number and at least a ipv4/24 and > ipv6/48 to > >> >>>> leverage with that, so I don't have to nat the various > emulations. > >> >>>> (and funding, anyone got funding?) Or, as the code is GPLv2 > licensed, > >> >>>> to see more test designers setup a testbed like this to > calibrate > >> >>>> their own stuff. > >> >>>> Presently we're able to test: > >> >>>> flent > >> >>>> netperf > >> >>>> iperf2 > >> >>>> iperf3 > >> >>>> speedtest-cli > >> >>>> crusader > >> >>>> the broadband forum udp based test: > >> >>>> https://github.com/BroadbandForum/obudpst > >> >>>> trexx > >> >>>> There's also a virtual machine setup that we can remotely drive > a web > >> >>>> browser from (but I didn't want to nat the results to the > world) to > awhile >> >>>> test other web services. > >> >>>> _______________________________________________ > >> >>>> Rpm mailing list > >> >>>> Rpm@lists.bufferbloat.net > >> >>>> https://lists.bufferbloat.net/listinfo/rpm > >> >>> _______________________________________________ > >> >>> Starlink mailing list > >> >>> Starlink@lists.bufferbloat.net > >> >>> https://lists.bufferbloat.net/listinfo/starlink > >> >> > >> >> _______________________________________________ > >> >> Starlink mailing list > >> >> Starlink@lists.bufferbloat.net > >> >> https://lists.bufferbloat.net/listinfo/starlink > >> > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/starlink