From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 541263B2A4; Fri, 13 Jan 2023 02:33:42 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1673595210; bh=5fV9P3qW4Rc9n0w+528RwSHuqnErOQFcx//qTeZtwkg=; h=X-UI-Sender-Class:Date:From:To:CC:Subject:In-Reply-To:References; b=uUDMKweKHKkBg55hGdf7zjJPj3iVxRnB9JluevejZSMKu9u2+oWjIKqQz0wgwzEGO qaJzMQeMkWzRdUCx34fDT6k37ptSscpDcxSUo4DaqOmbTfk3TYFCKaRczoBGx0yBtS +GalkzXe3K8BhJVt/iTVvJ8Om8mYMbnmHWDynCv35tV+NQ1115/VU85tsMCHVoRfPp Figi9Z2nuuttjPR41uRKJWMyw7nh2nWmm3OyIRWBdjKketW5UDsJaOKE3vucMzkk6F vjNZgWwlmpVMeO4QE+4mziK6kXndese3CwjiZSx74mChcPJRSrOpduFnF3DjKv6LBS WGYJV/8WIShLw== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from [127.0.0.1] ([80.187.108.196]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MyKDU-1oXM9O0tGW-00yhsE; Fri, 13 Jan 2023 08:33:30 +0100 Date: Fri, 13 Jan 2023 08:33:27 +0100 From: Sebastian Moeller To: dickroy@alum.mit.edu, Dick Roy CC: "'Rodney W. Grimes'" , mike.reynolds@netforecast.com, 'libreqos' , "'David P. Reed'" , 'Rpm' , 'rjmcmahon' , 'bloat' User-Agent: K-9 Mail for Android In-Reply-To: References: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> <5C6EFF20C9F745AC87DA199EE7ABBAD9@SRA6> Message-ID: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=----MQN5KYWMISO0XDTY8BMHBDNTZ6JHEL Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:RIa0Ri0Sn5uF0zCSfthH0CBSXBj8Cv99wV9W6Jn7dJ6VIvFo3Gd pDFSpNuvT1mDbbIuqo7s0nwbiO+TXLT+maUrW8a8oxRb6U2tExYNJgPvsK72MQEXyOvI30s f6YbwaMbe1STWxRPUsfv6ywdlsw3Y9RmfdYLUyS2VzID5LOzCfB2EA1heweMSyDDuFdpq91 j0Dx+TpvFKEEkNSbSkhRQ== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:OXKFjQmb8Wk=;ekakThQa9Hfs6xEeVFopjz9T/Ts fzOkqJ0GpTN5AaUoR41iZ1GP1mKeXz15dglpSFWUteLaTvvKUGi/iVjTYHwpetZjn8+U9hsgH CQ6uII71YkUYCWgBaf3QsO02RefvteioT+IqLJdjBVPDuBnhLkEtvi+CV5GsccYclv5eOwCgg aaCL1j5T0W4lq42WdukA0j2h6nHRZLzDMkrKY9aRmohh103oc1TOhPEkKJ/Ys5Ub0LQ2kdRI/ djT1UhhNnPeRCz8aC6nvxYsCFIfEv9Mclf4WKLehglAINTn/RfTJgT033cqmoGYaoKrT1Odkr UmVFT18LjDVKoI5x6BAeK146hisuoQn/prmROk/Xq+xMaqr/nfi5l867pSMoqZgun2Zp76nkt Vn2f0rEQQO8t+xTz5exnPtyUbAVtQ0TuRwIWltbh4KTekUTCMG3Nu9EHb0JuVvXqwFwtA+5wF sgM8jSsrCGKhOWElHZJo/ITpv9K12qc77RW7KJ/muoMuYI4L5d6XAC/K+TZ2hSQhnI1FxyrG5 Ci9gNKT3ERhW2JKkn0BmwS4nEkIFTE7FVPQPpzBi2pqJu8/JW8tzgqfbqKxyhur0idqx8cs4o CdMcto5WLBMemJ9FKAl+CdgTYCvF5Z0xZeRoxSpOTtVGQa3MhFLztj1r1oaq7frPU36SNq44B GRf0sKtO95F/lqc0zPTB4h1xOvZerdsA5oZErhBiUSCEekDr6BcgMLTlVxNI4xizEa/KX3svr YubzgJDTdmfqvnvppVDZdZ4fi0Je+opFkd1UW8nKCIlx8/2wTy/mM+hu1cOovI9wuePTLRSrA IHu4H3sg+zigqySWSNu8LVT8cD4ConS9UZ/XaErMSZq45SzUYzdftbzBFye1QAGvqSYZ3k0pu 5YjpfJB56hjqgbXkW7r4hwJrjFF0q/cgib6JOZ0zarGwVDYqRp2n0FBgciMo3MF1dQ0xvEOke 4TXT//NcchfvvQTtrzbwNtwNpLI= Subject: Re: [LibreQoS] [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2023 07:33:42 -0000 ------MQN5KYWMISO0XDTY8BMHBDNTZ6JHEL Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi RR, Thanks for the detailed response below, since my point is somewhat orthogo= nal I opted for top-posting=2E Let me take a step back here and rephrase, synchronising clocks within an = acceptable range to be useful is not rocket science nor witchcraft=2E For m= easuring internet traffic 'millisecond' range seems acceptable, local netwo= rks can probably profit from finer time resolution=2E So I am not after e= =2Eg=2E clock synchronisation to participate in SDH/SONET=2E Heck in the to= y project I am active in, we operate on load dependent delay deltas so we e= ven ignore different time offsets and are tolerant to (mildly) different ti= ckrates and clock skew, but it would certainly be nice to have some accepta= ble measure of UTC from endpoints to be able to interpret timestamps as 'ab= solute'=2E Mind you I am fine with them not being veridical absolute, but j= ust good enough for my measurement purpose and I guess that should be withi= n the range of the achievable=2E Heck, if all servers we query timestamps o= f would be NTP-'synchronized' and would follow the RFC recommendation to re= port timestamps in milliseconds past midnight UTC I would be happy=2E Regards Sebsstian On 12 January 2023 21:39:21 CET, Dick Roy wrote= : >Hi Sebastian (et=2E al=2E), > >=20 > >[I'll comment up here instead of inline=2E] =20 > >=20 > >Let me start by saying that I have not been intimately involved with the >IEEE 1588 effort (PTP), however I was involved in the 802=2E11 efforts al= ong a >similar vein, just adding the wireless first hop component and it's effec= ts >on PTP=2E =20 > >=20 > >What was apparent from the outset was that there was a lack of understand= ing >what the terms "to synchronize" or "to be synchronized" actually mean=2E = It's >not trivial =2E because we live in a (approximately, that's another story= !) >4-D space-time continuum where the Lorentz metric plays a critical role= =2E >Therein, simultaneity (aka "things happening at the same time") means the >"distance" between two such events is zero and that distance is given by >sqrt(x^2 + y^2 + z^2 - (ct)^2) and the "thing happening" can be the tick = of >a clock somewhere=2E Now since everything is relative (time with respect = to >what? / location with respect to where?) it's pretty easy to see that "if >you don't know where you are, you can't know what time it is!" (English >sailors of the 18th century knew this well!) Add to this the fact that if >everything were stationary, nothing would happen (as Einstein said "Nothi= ng >happens until something moves!"), special relativity also pays a role=2E >Clocks on GPS satellites run approx=2E 7usecs/day slower than those on ea= rth >due to their "speed" (8700 mph roughly)! Then add the consequence that >without mass we wouldn't exist (in these forms at least:-)), and >gravitational effects (aka General Relativity) come into play=2E Those tu= rn >out to make clocks on GPS satellites run 45usec/day faster than those on >earth! The net effect is that GPS clocks run about 38usec/day faster tha= n >clocks on earth=2E So what does it mean to "synchronize to GPS"? Point = is: >it's a non-trivial question with a very complicated answer=2E The reason= it >is important to get all this right is that the "what that ties time and >space together" is the speed of light and that turns out to be a >"foot-per-nanosecond" in a vacuum (roughly 300m/usec)=2E This means if I= am >uncertain about my location to say 300 meters, then I also am not sure wh= at >time it is to a usec AND vice-versa!=20 > >=20 > >All that said, the simplest explanation of synchronization is probably: T= wo >clocks are synchronized if, when they are brought (slowly) into physical >proximity ("sat next to each other") in the same (quasi-)inertial frame a= nd >the same gravitational potential (not so obvious BTW =2E see the FYI belo= w!), >an observer of both would say "they are keeping time identically"=2E Sinc= e >this experiment is rarely possible, one can never be "sure" that his cloc= k >is synchronized to any other clock elsewhere=2E And what does it mean to = say >they "were synchronized" when brought together, but now they are not beca= use >they are now in different gravitational potentials! (FYI, there are land >mine detectors being developed on this very principle! I know someone who >actually worked on such a project!)=20 > >=20 > >This all gets even more complicated when dealing with large networks of >networks in which the "speed of information transmission" can vary depend= ing >on the medium (cf=2E coaxial cables versus fiber versus microwave links!)= In >fact, the atmosphere is one of those media and variations therein result = in >the need for "GPS corrections" (cf=2E RTCM GPS correction messages, RTK, = etc=2E) >in order to get to sub-nsec/cm accuracy=2E Point is if you have a set of >nodes distributed across the country all with GPS and all "synchronized t= o >GPS time", and a second identical set of nodes (with no GPS) instead >connected with a network of cables and fiber links, all of different leng= ths >and composition using different carrier frequencies (dielectric constants >vary with frequency!) "synchronized" to some clock somewhere using NTP or >PTP), the synchronization of the two sets will be different unless a comm= on >reference clock is used AND all the above effects are taken into account, >and good luck with that! :-)=20 > >=20 > >In conclusion, if anyone tells you that clock synchronization in >communication networks is simple ("Just use GPS!"), you should feel free = to >chuckle (under your breath if necessary:-))=20 > >=20 > >Cheers, > >=20 > >RR > >=20 > >=20 > > =20 > >=20 > >=20 > >=20 > >-----Original Message----- >From: Sebastian Moeller [mailto:moeller0@gmx=2Ede]=20 >Sent: Thursday, January 12, 2023 12:23 AM >To: Dick Roy >Cc: Rodney W=2E Grimes; mike=2Ereynolds@netforecast=2Ecom; libreqos; Davi= d P=2E >Reed; Rpm; rjmcmahon; bloat >Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA > >=20 > >Hi RR, > >=20 > >=20 > >> On Jan 11, 2023, at 22:46, Dick Roy wrote: > >>=20 > >> =20 > >> =20 > >> -----Original Message----- > >> From: Starlink [mailto:starlink-bounces@lists=2Ebufferbloat=2Enet] On B= ehalf >Of Sebastian Moeller via Starlink > >> Sent: Wednesday, January 11, 2023 12:01 PM > >> To: Rodney W=2E Grimes > >> Cc: Dave Taht via Starlink; mike=2Ereynolds@netforecast=2Ecom; libreqos= ; David >P=2E Reed; Rpm; rjmcmahon; bloat > >> Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in U= SA > >> =20 > >> Hi Rodney, > >> =20 > >> =20 > >> =20 > >> =20 > >> > On Jan 11, 2023, at 19:32, Rodney W=2E Grimes >wrote: > >> >=20 > >> > Hello, > >> >=20 > >> > Yall can call me crazy if you want=2E=2E but=2E=2E=2E see below [= RWG] > >> >> Hi Bib, > >> >>=20 > >> >>=20 > >> >>> On Jan 9, 2023, at 20:13, rjmcmahon via Starlink > wrote: > >> >>>=20 > >> >>> My biggest barrier is the lack of clock sync by the devices, i=2Ee= =2E very >limited support for PTP in data centers and in end devices=2E This limits= the >ability to measure one way delays (OWD) and most assume that OWD is 1/2 a= nd >RTT which typically is a mistake=2E We know this intuitively with airplan= e >flight times or even car commute times where the one way time is not 1/2 = a >round trip time=2E Google maps & directions provide a time estimate for t= he >one way link=2E It doesn't compute a round trip and divide by two=2E > >> >>>=20 > >> >>> For those that can get clock sync working, the iperf 2 --trip-times >options is useful=2E > >> >>=20 > >> >> [SM] +1; and yet even with unsynchronized clocks one can try to >measure how latency changes under load and that can be done per direction= =2E >Sure this is far inferior to real reliably measured OWDs, but if life/the >internet deals you lemons=2E=2E=2E=2E > >> >=20 > >> > [RWG] iperf2/iperf3, etc are already moving large amounts of data bac= k >and forth, for that matter any rate test, why not abuse some of that data >and add the fundemental NTP clock sync data and bidirectionally pass each >others concept of "current time"=2E IIRC (its been 25 years since I work= ed on >NTP at this level) you *should* be able to get a fairly accurate clock de= lta >between each end, and then use that info and time stamps in the data stre= am >to compute OWD's=2E You need to put 4 time stamps in the packet, and wit= h >that you can compute "offset"=2E > >> [RR] For this to work at a reasonable level of accuracy, the timestampi= ng >circuits on both ends need to be deterministic and repeatable as I recall= =2E >Any uncertainty in that process adds to synchronization >errors/uncertainties=2E > >> =20 > >> [SM] Nice idea=2E I would guess that all timeslot based access >technologies (so starlink, docsis, GPON, LTE?) all distribute "high quali= ty >time" carefully to the "modems", so maybe all that would be needed is to >expose that high quality time to the LAN side of those modems, dressed up= as >NTP server? > >> [RR] It's not that simple! Distributing "high-quality time", i=2Ee=2E >"synchronizing all clocks" does not solve the communication problem in >synchronous slotted MAC/PHYs! > >=20 > > [SM] I happily believe you, but the same idea of "time slot" needs = to >be shared by all nodes, no? So the clockss need to be reasonably similar >rate, aka synchronized (see below)=2E > >=20 > >=20 > >> All the technologies you mentioned above are essentially P2P, not >intended for broadcast=2E Point is, there is a point controller (aka PoC= ) >often called a base station (eNodeB, gNodeB, =2E) that actually "controls >everything that is necessary to control" at the UE including time, freque= ncy >and sampling time offsets, and these are critical to get right if you wan= t >to communicate, and they are ALL subject to the laws of physics (cf=2E th= e >speed of light)! Turns out that what is necessary for the system to funct= ion >anywhere near capacity, is for all the clocks governing transmissions fro= m >the UEs to be "unsynchronized" such that all the UE transmissions arrive = at >the PoC at the same (prescribed) time! > >=20 > > [SM] Fair enough=2E I would call clocks that are "in sync" albeit w= ith >individual offsets as synchronized, but I am a layman and that might soun= d >offensively wrong to experts in the field=2E But even without the naming = my >point is that all systems that depend on some idea of shared time-base ar= e >halfway there of exposing that time to end users, by "translating it into= an >NTP time source at the modem=2E > >=20 > >=20 > >> For some technologies, in particular 5G!, these considerations are >ESSENTIAL=2E Feel free to scour the 3GPP LTE 5G RLC and PHY specs if you = don't >believe me! J =20 > >=20 > > [SM Far be it from me not to believe you, so thanks for the pointer= s=2E >Yet, I still think that unless different nodes of a shared segment move a= t >significantly different speeds, that there should be a common >"tick-duration" for all clocks even if each clock runs at an offset=2E=2E= =2E (I >naively would try to implement something like that by trying to fully >synchronize clocks and maintain a local offset value to convert from >"absolute" time to "network" time, but likely because coming from the >outside I am blissfully unaware of the detail challenges that need to be >solved)=2E > >=20 > >Regards & Thanks > > Sebastian > >=20 > >=20 > >> =20 > >> =20 > >> >=20 > >> >>=20 > >> >>=20 > >> >>>=20 > >> >>> --trip-times > >> >>> enable the measurement of end to end write to read latencies (clien= t >and server clocks must be synchronized) > >> > [RWG] --clock-skew > >> > enable the measurement of the wall clock difference between sende= r >and receiver > >> >=20 > >> >>=20 > >> >> [SM] Sweet! > >> >>=20 > >> >> Regards > >> >> Sebastian > >> >>=20 > >> >>>=20 > >> >>> Bob > >> >>>> I have many kvetches about the new latency under load tests being > >> >>>> designed and distributed over the past year=2E I am delighted! tha= t >they > >> >>>> are happening, but most really need third party evaluation, and > >> >>>> calibration, and a solid explanation of what network pathologies t= hey > >> >>>> do and don't cover=2E Also a RED team attitude towards them, as we= ll as > >> >>>> thinking hard about what you are not measuring (operations researc= h)=2E > >> >>>> I actually rather love the new cloudflare speedtest, because it te= sts > >> >>>> a single TCP connection, rather than dozens, and at the same time >folk > >> >>>> are complaining that it doesn't find the actual "speed!"=2E yet=2E= =2E=2E the > >> >>>> test itself more closely emulates a user experience than >speedtest=2Enet > >> >>>> does=2E I am personally pretty convinced that the fewer numbers of >flows > >> >>>> that a web page opens improves the likelihood of a good user > >> >>>> experience, but lack data on it=2E > >> >>>> To try to tackle the evaluation and calibration part, I've reached >out > >> >>>> to all the new test designers in the hope that we could get togeth= er > >> >>>> and produce a report of what each new test is actually doing=2E I'= ve > >> >>>> tweeted, linked in, emailed, and spammed every measurement list I >know > >> >>>> of, and only to some response, please reach out to other test >designer > >> >>>> folks and have them join the rpm email list? > >> >>>> My principal kvetches in the new tests so far are: > >> >>>> 0) None of the tests last long enough=2E > >> >>>> Ideally there should be a mode where they at least run to "time of > >> >>>> first loss", or periodically, just run longer than the > >> >>>> industry-stupid^H^H^H^H^H^Hstandard 20 seconds=2E There be dragons > >> >>>> there! It's really bad science to optimize the internet for 20 > >> >>>> seconds=2E It's like optimizing a car, to handle well, for just 20 > >> >>>> seconds=2E > >> >>>> 1) Not testing up + down + ping at the same time > >> >>>> None of the new tests actually test the same thing that the infamo= us > >> >>>> rrul test does - all the others still test up, then down, and ping= =2E >It > >> >>>> was/remains my hope that the simpler parts of the flent test suite= - > >> >>>> such as the tcp_up_squarewave tests, the rrul test, and the rtt_fa= ir > >> >>>> tests would provide calibration to the test designers=2E > >> >>>> we've got zillions of flent results in the archive published here: > >> >>>> https://blog=2Ecerowrt=2Eorg/post/found_in_flent/ > >> >>>> ps=2E Misinformation about iperf 2 impacts my ability to do this= =2E > >> >>>=20 > >> >>>> The new tests have all added up + ping and down + ping, but not up= + > >> >>>> down + ping=2E Why?? > >> >>>> The behaviors of what happens in that case are really non-intuitiv= e, >I > >> >>>> know, but=2E=2E=2E it's just one more phase to add to any one of t= hose new > >> >>>> tests=2E I'd be deliriously happy if someone(s) new to the field > >> >>>> started doing that, even optionally, and boggled at how it defeate= d > >> >>>> their assumptions=2E > >> >>>> Among other things that would show=2E=2E=2E > >> >>>> It's the home router industry's dirty secret than darn few "gigabi= t" > >> >>>> home routers can actually forward in both directions at a gigabit= =2E >I'd > >> >>>> like to smash that perception thoroughly, but given our starting >point > >> >>>> is a gigabit router was a "gigabit switch" - and historically been > >> >>>> something that couldn't even forward at 200Mbit - we have a long w= ay > >> >>>> to go there=2E > >> >>>> Only in the past year have non-x86 home routers appeared that coul= d > >> >>>> actually do a gbit in both directions=2E > >> >>>> 2) Few are actually testing within-stream latency > >> >>>> Apple's rpm project is making a stab in that direction=2E It looks > >> >>>> highly likely, that with a little more work, crusader and > >> >>>> go-responsiveness can finally start sampling the tcp RTT, loss and > >> >>>> markings, more directly=2E As for the rest=2E=2E=2E sampling TCP_I= NFO on > >> >>>> windows, and Linux, at least, always appeared simple to me, but I'= m > >> >>>> discovering how hard it is by delving deep into the rust behind > >> >>>> crusader=2E > >> >>>> the goresponsiveness thing is also IMHO running WAY too many strea= ms > >> >>>> at the same time, I guess motivated by an attempt to have the test > >> >>>> complete quickly? > >> >>>> B) To try and tackle the validation problem:ps=2E Misinformation a= bout >iperf 2 impacts my ability to do this=2E > >> >>>=20 > >> >>>> In the libreqos=2Eio project we've established a testbed where tes= ts >can > >> >>>> be plunked through various ISP plan network emulations=2E It's her= e: > >> >>>> https://payne=2Etaht=2Enet (run bandwidth test for what's currentl= y >hooked > >> >>>> up) > >> >>>> We could rather use an AS number and at least a ipv4/24 and ipv6/4= 8 >to > >> >>>> leverage with that, so I don't have to nat the various emulations= =2E > >> >>>> (and funding, anyone got funding?) Or, as the code is GPLv2 licens= ed, > >> >>>> to see more test designers setup a testbed like this to calibrate > >> >>>> their own stuff=2E > >> >>>> Presently we're able to test: > >> >>>> flent > >> >>>> netperf > >> >>>> iperf2 > >> >>>> iperf3 > >> >>>> speedtest-cli > >> >>>> crusader > >> >>>> the broadband forum udp based test: > >> >>>> https://github=2Ecom/BroadbandForum/obudpst > >> >>>> trexx > >> >>>> There's also a virtual machine setup that we can remotely drive a = web > >> >>>> browser from (but I didn't want to nat the results to the world) t= o > >> >>>> test other web services=2E > >> >>>> _______________________________________________ > >> >>>> Rpm mailing list > >> >>>> Rpm@lists=2Ebufferbloat=2Enet > >> >>>> https://lists=2Ebufferbloat=2Enet/listinfo/rpm > >> >>> _______________________________________________ > >> >>> Starlink mailing list > >> >>> Starlink@lists=2Ebufferbloat=2Enet > >> >>> https://lists=2Ebufferbloat=2Enet/listinfo/starlink > >> >>=20 > >> >> _______________________________________________ > >> >> Starlink mailing list > >> >> Starlink@lists=2Ebufferbloat=2Enet > >> >> https://lists=2Ebufferbloat=2Enet/listinfo/starlink > >> =20 > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists=2Ebufferbloat=2Enet > >> https://lists=2Ebufferbloat=2Enet/listinfo/starlink > --=20 Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E ------MQN5KYWMISO0XDTY8BMHBDNTZ6JHEL Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi RR,

T= hanks for the detailed response below, since my point is somewhat orthogona= l I opted for top-posting=2E
Let me take a step back here and rephrase, = synchronising clocks within an acceptable range to be useful is not rocket = science nor witchcraft=2E For measuring internet traffic 'millisecond' rang= e seems acceptable, local networks can probably profit from finer time reso= lution=2E So I am not after e=2Eg=2E clock synchronisation to participate i= n SDH/SONET=2E Heck in the toy project I am active in, we operate on load d= ependent delay deltas so we even ignore different time offsets and are tole= rant to (mildly) different tickrates and clock skew, but it would certainly= be nice to have some acceptable measure of UTC from endpoints to be able t= o interpret timestamps as 'absolute'=2E Mind you I am fine with them not be= ing veridical absolute, but just good enough for my measurement purpose and= I guess that should be within the range of the achievable=2E Heck, if all = servers we query timestamps of would be NTP-'synchronized' and would follow= the RFC recommendation to report timestamps in milliseconds past midnight = UTC I would be happy=2E

Regards
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Sebsstian

On 12 January 2023 21= :39:21 CET, Dick Roy <dickroy@alum=2Emit=2Eedu> wrote:

Hi Sebastian (et=2E al=2E),

 

[I=E2=80=99ll comment up here instead of inline=2E]  <= /o:p>

 

Let me start by saying that I have not been intimately involved = with the IEEE 1588 effort (PTP), however I was involved in the 802=2E11 efforts= along a similar vein, just adding the wireless first hop component and it=E2=80= =99s effects on PTP=2E  

 

What was apparent from the outset was that there was a lack of understanding what the terms =E2=80=9Cto synchronize=E2=80=9D or =E2=80=9C= to be synchronized=E2=80=9D actually mean=2E  It=E2=80=99s not trivial =E2= =80=A6 because we live in a (approximately, that=E2=80=99s another story!) 4-D space-time continuum where the Lorentz metric plays a critical role=2E  Therein, simultaneity (aka =E2=80=9Cthings happening at the same time=E2=80=9D) mea= ns the =E2=80=9Cdistance=E2=80=9D between two such events is zero and that distance is given by sqrt(x^2 + y= ^2 + z^2 =E2=80=93 (ct)^2) and the =E2= =80=9Cthing happening=E2=80=9D can be the tick of a clock somewhere=2E Now since every= thing is relative (time with respect to what? / location with respect to where?) it= =E2=80=99s pretty easy to see that =E2=80=9Cif you don=E2=80=99t know where you are, = you can=E2=80=99t know what time it is!=E2=80=9D (English sailors of the 18th cen= tury knew this well!) Add to this the fact that if everything were stationary, nothing would happen (as Einstein said =E2=80=9CNothing happens until some= thing moves!=E2=80=9D), special relativity also pays a role=2E  Clocks on G= PS satellites run approx=2E 7usecs/day slower than those on earth due to thei= r =E2=80=9Cspeed=E2=80=9D (8700 mph roughly)! Then add the consequence that without mass we wouldn= =E2=80=99t exist (in these forms at leastJ), and gravitational effect= s (aka General Relativity) come into play=2E Those turn out to make clocks on GPS satelli= tes run 45usec/day faster than those on earth!  The net effect is that GPS cl= ocks run about 38usec/day faster than clocks on earth=2E  So what does it = mean to =E2=80=9Csynchronize to GPS=E2=80=9D?  Point is: it=E2=80=99s a non-t= rivial question with a very complicated answer=2E  The reason it is importan= t to get all this right is that the =E2=80=9Cwhat that ties time and space toge= ther=E2=80=9D is the speed of light and that turns out to be a =E2=80=9Cfoot-per-nanosec= ond=E2=80=9D in a vacuum (roughly 300m/usec)=2E  This means if I am uncertain abou= t my location to say 300 meters, then I also am not sure what time it is to a u= sec AND vice-versa!

 

All that said, the simplest explanation of synchronization is pr= obably: Two clocks are synchronized if, when they are brought (slowly) into physic= al proximity (=E2=80=9Csat next to each other=E2=80=9D) in the same (quasi-)i= nertial frame and the same gravitational potential (not so obvious BTW =E2=80=A6 s= ee the FYI below!), an observer of both would say =E2=80=9Cthey are keeping time identically=E2=80=9D=2E Since this experiment is rarely possible, one can = never be =E2=80=9Csure=E2=80=9D that his clock is synchronized to any other clock elsewhere=2E And what do= es it mean to say they =E2=80=9Cwere synchronized=E2=80=9D when brought together= , but now they are not because they are now in different gravitational potentials! (= FYI, there are land mine detectors being developed on this very principle! I kn= ow someone who actually worked on such a project!) <= /p>

 

This all gets even more complicated when dealing with large netw= orks of networks in which the =E2=80=9Cspeed of information transmission=E2=80=9D = can vary depending on the medium (cf=2E coaxial cables versus fiber versus microwav= e links!) In fact, the atmosphere is one of those media and variations therein resul= t in the need for =E2=80=9CGPS corrections=E2=80=9D (cf=2E RTCM GPS correction = messages, RTK, etc=2E) in order to get to sub-nsec/cm accuracy=2E  Point is if you h= ave a set of nodes distributed across the country all with GPS and all =E2=80=9Csync= hronized to GPS time=E2=80=9D, and a second identical set of nodes (with no GPS) in= stead connected with a network of cables and fiber links, all of different lengths and composition using different carrier frequencies (dielectric constants vary= with frequency!) =E2=80=9Csynchronized=E2=80=9D to some clock somewhere using N= TP or PTP), the synchronization of the two sets will be different unless a commo= n reference clock is used AND all the above effects are taken into account, = and good luck with that! J

 

In conclusion, if anyone tells you that clock synchronization in communication networks is simple (=E2=80=9CJust use GPS!=E2=80=9D), you sh= ould feel free to chuckle (under your breath if necessaryJ)

 

Cheers,

 

RR

 

 

  

 

 

 

-----Original Message-----
From: Sebastian Moeller [mailto:moeller0@gmx=2Ede]
Sent: Thursday, January 12, 2023 12:23 AM
To: Dick Roy
Cc: Rodney W=2E Grimes; mike=2Ereynolds@netforecast=2Ecom; libreqos; David P=2E Reed; Rpm; rjmcmahon; bl= oat
Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA

 

Hi RR,

 

 

> On Jan 11, 2023, at 22:46, Dick Roy <dickroy@alum=2Emit=2Eedu> wrote:

>

> 

> 

> -----Original Message-----

> From: Starlink [mailto:starlink-bounces@lists=2Ebufferbloat= =2Enet] On Behalf Of Sebastian Moeller via Starlink

> Sent: Wednesday, January 11, 2023 12:01 PM

> To: Rodney W=2E Grimes

> Cc: Dave Taht via Starlink; mike=2Ereynolds@netforecast=2Ec= om; libreqos; David P=2E Reed; Rp= m; rjmcmahon; bloat

> Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Vol= unteers in USA<= /st1:country-region>

> 

> Hi Rodney,

> 

> 

> 

> 

> > On Jan 11, 2023, at 19:32, Rodney W=2E Grimes <starlink@gndrsh=2Ednsmgr=2Enet> wrote:

> >

> > Hello,

> >

> >     Yall can call me crazy if you = want=2E=2E but=2E=2E=2E see below [RWG]

> >> Hi Bib,

> >>

> >>

> >>> On Jan 9, 2023, at 20:13, rjmcmahon via Starli= nk <starlink@lists=2Ebufferbloat=2Enet> wrote:=

> >>>

> >>> My biggest barrier is the lack of clock sync b= y the devices, i=2Ee=2E very limited support for PTP in data centers and in end = devices=2E This limits the ability to measure one way delays (OWD) and most assume th= at OWD is 1/2 and RTT which typically is a mistake=2E We know this intuitivel= y with airplane flight times or even car commute times where the one way time is = not 1/2 a round trip time=2E Google maps & directions provide a time estim= ate for the one way link=2E It doesn't compute a round trip and divide by two=2E

> >>>

> >>> For those that can get clock sync working, the= iperf 2 --trip-times options is useful=2E

> >>

> >>    [SM] +1; and yet even with unsyn= chronized clocks one can try to measure how latency changes under load and that can = be done per direction=2E Sure this is far inferior to real reliably measured = OWDs, but if life/the internet deals you lemons=2E=2E=2E=2E

> >

> > [RWG] iperf2/iperf3, etc are already moving large amou= nts of data back and forth, for that matter any rate test, why not abuse some of = that data and add the fundemental NTP clock sync data and bidirectionally pass = each others concept of "current time"=2E  IIRC (its been 25 years since I worked on NTP at this level) you *should* be able to get a fairly accura= te clock delta between each end, and then use that info and time stamps in th= e data stream to compute OWD's=2E  You need to put 4 time stamps in the packet, and with that you can compute "offset"=2E=

> [RR] For this to work at a reasonable level of accuracy, th= e timestamping circuits on both ends need to be deterministic and repeatable= as I recall=2E Any uncertainty in that process adds to synchronization errors/uncertainties=2E

> 

>       [SM] Nice idea=2E I wou= ld guess that all timeslot based access technologies (so starlink, docsis, GPON, LT= E?) all distribute "high quality time" carefully to the "modems", so maybe all that would be needed is to expose that high quality time to the LAN side of those modems, dressed up as NTP server?

> [RR] It=E2=80=99s not that simple!  Distributing =E2=80=9Chigh-quality time=E2=80=9D, i=2Ee=2E =E2=80=9Csynchronizing all c= locks=E2=80=9D does not solve the communication problem in synchronous slotted MAC/PHYs!<= o:p>

 

      [SM] I happily believe you, but t= he same idea of "time slot" needs to be shared by all nodes, no? So the clockss need to be reasonably similar rate, aka synchronized (see below)= =2E

 

 

>  All the technologies you mentioned above are essentia= lly P2P, not intended for broadcast=2E  Point is, there is a point contro= ller (aka PoC) often called a base station (eNodeB, gNodeB, =E2=80=A6) that act= ually =E2=80=9Ccontrols everything that is necessary to control=E2=80=9D at the = UE including time, frequency and sampling time offsets, and these are critica= l to get right if you want to communicate, and they are ALL subject to the laws= of physics (cf=2E the speed of light)! Turns out that what is necessary for t= he system to function anywhere near capacity, is for all the clocks governing transmissions from the UEs to be =E2=80=9Cunsynchronized=E2=80=9D such tha= t all the UE transmissions arrive at the PoC at the same (prescribed) time!

 

      [SM] Fair enough=2E I would call = clocks that are "in sync" albeit with individual offsets as synchronized, but I am a layman and that might sound offensively wrong to experts in the field=2E But even without the naming my point is that all systems that dep= end on some idea of shared time-base are halfway there of exposing that time to e= nd users, by "translating it into an NTP time source at the modem=2E

 

 

> For some technologies, in particular 5G!, these considerati= ons are ESSENTIAL=2E Feel free to scour the 3GPP LTE 5G RLC and PHY specs if you don=E2=80=99t believe me! J  

 

      [SM Far be it from me not to beli= eve you, so thanks for the pointers=2E Yet, I still think that unless differen= t nodes of a shared segment move at significantly different speeds, that there sho= uld be a common "tick-duration" for all clocks even if each clock runs at an offset=2E=2E=2E (I naively would try to implement something like that b= y trying to fully synchronize clocks and maintain a local offset value to convert from "absolute" time to "network" time, but likely because coming from the outside I am blissfully unaware of the detail challenges t= hat need to be solved)=2E

 

Regards & Thanks

      Sebastian

 

 

> 

> 

> >

> >>

> >>

> >>>

> >>> --trip-times

> >>> enable the measurement of end to end write to = read latencies (client and server clocks must be synchronized)

> > [RWG] --clock-skew

> >     enable the measurement of the = wall clock difference between sender and receiver

> >

> >>

> >>    [SM] Sweet!

> >>

> >> Regards

> >>    Sebastian

> >>

> >>>

> >>> Bob

> >>>> I have many kvetches about the new latency= under load tests being

> >>>> designed and distributed over the past yea= r=2E I am delighted! that they

> >>>> are happening, but most really need third = party evaluation, and

> >>>> calibration, and a solid explanation of wh= at network pathologies they

> >>>> do and don't cover=2E Also a RED team atti= tude towards them, as well as

> >>>> thinking hard about what you are not measu= ring (operations research)=2E

> >>>> I actually rather love the new cloudflare speedtest, because it tests

> >>>> a single TCP connection, rather than dozen= s, and at the same time folk

> >>>> are complaining that it doesn't find the a= ctual "speed!"=2E yet=2E=2E=2E the

> >>>> test itself more closely emulates a user experience than speedtest=2Enet

> >>>> does=2E I am personally pretty convinced t= hat the fewer numbers of flows

> >>>> that a web page opens improves the likelih= ood of a good user

> >>>> experience, but lack data on it=2E

> >>>> To try to tackle the evaluation and calibr= ation part, I've reached out

> >>>> to all the new test designers in the hope = that we could get together

> >>>> and produce a report of what each new test= is actually doing=2E I've

> >>>> tweeted, linked in, emailed, and spammed e= very measurement list I know

> >>>> of, and only to some response, please reac= h out to other test designer

> >>>> folks and have them join the rpm email lis= t?

> >>>> My principal kvetches in the new tests so = far are:

> >>>> 0) None of the tests last long enough=2E

> >>>> Ideally there should be a mode where they = at least run to "time of

> >>>> first loss", or periodically, just run longer than the

> >>>> industry-stupid^H^H^H^H^H^Hstandard 20 sec= onds=2E There be dragons

> >>>> there! It's really bad science to optimize= the internet for 20

> >>>> seconds=2E It's like optimizing a car, to = handle well, for just 20

> >>>> seconds=2E

> >>>> 1) Not testing up + down + ping at the sam= e time

> >>>> None of the new tests actually test the sa= me thing that the infamous

> >>>> rrul test does - all the others still test= up, then down, and ping=2E It

> >>>> was/remains my hope that the simpler parts= of the flent test suite -

> >>>> such as the tcp_up_squarewave tests, the r= rul test, and the rtt_fair

> >>>> tests would provide calibration to the tes= t designers=2E

> >>>> we've got zillions of flent results in the archive published here:

> >>>> https://blog=2Ecerowrt=2Eorg/post/found_in= _flent/

> >>>> ps=2E Misinformation about iperf 2 impacts= my ability to do this=2E

> >>>

> >>>> The new tests have all added up + ping and= down + ping, but not up +

> >>>> down + ping=2E Why??

> >>>> The behaviors of what happens in that case= are really non-intuitive, I

> >>>> know, but=2E=2E=2E it's just one more phas= e to add to any one of those new

> >>>> tests=2E I'd be deliriously happy if someo= ne(s) new to the field

> >>>> started doing that, even optionally, and b= oggled at how it defeated

> >>>> their assumptions=2E

> >>>> Among other things that would show=2E=2E= =2E

> >>>> It's the home router industry's dirty secr= et than darn few "gigabit"

> >>>> home routers can actually forward in both directions at a gigabit=2E I'd

> >>>> like to smash that perception thoroughly, = but given our starting point

> >>>> is a gigabit router was a "gigabit switch" - and historically been

> >>>> something that couldn't even forward at 20= 0Mbit - we have a long way

> >>>> to go there=2E

> >>>> Only in the past year have non-x86 home ro= uters appeared that could

> >>>> actually do a gbit in both directions=2E

> >>>> 2) Few are actually testing within-stream = latency

> >>>> Apple's rpm project is making a stab in th= at direction=2E It looks

> >>>> highly likely, that with a little more wor= k, crusader and

> >>>> go-responsiveness can finally start sampli= ng the tcp RTT, loss and

> >>>> markings, more directly=2E As for the rest= =2E=2E=2E sampling TCP_INFO on

> >>>> windows, and Linux, at least, always appea= red simple to me, but I'm

> >>>> discovering how hard it is by delving deep= into the rust behind

> >>>> crusader=2E

> >>>> the goresponsiveness thing is also IMHO ru= nning WAY too many streams

> >>>> at the same time, I guess motivated by an = attempt to have the test

> >>>> complete quickly?=

> >>>> B) To try and tackle the validation proble= m:ps=2E Misinformation about iperf 2 impacts my ability to do this=2E

> >>>

> >>>> In the libreqos=2Eio project we've establi= shed a testbed where tests can

> >>>> be plunked through various ISP plan networ= k emulations=2E It's here:

> >>>> https://payne=2Etaht=2Enet (run bandwidth = test for what's currently hooked

> >>>> up)

> >>>> We could rather use an AS number and at le= ast a ipv4/24 and ipv6/48 to

> >>>> leverage with that, so I don't have to nat= the various emulations=2E

> >>>> (and funding, anyone got funding?) Or, as = the code is GPLv2 licensed,

> >>>> to see more test designers setup a testbed= like this to calibrate

> >>>> their own stuff=2E

> >>>> Presently we're able to test:

> >>>> flent

> >>>> netperf

> >>>> iperf2

> >>>> iperf3

> >>>> speedtest-cli

> >>>> crusader

> >>>> the broadband forum udp based test:

> >>>> https://github=2Ecom/BroadbandForum/obudps= t

> >>>> trexx

> >>>> There's also a virtual machine setup that = we can remotely drive a web

> >>>> browser from (but I didn't want to nat the results to the world) to

> >>>> test other web services=2E

> >>>> __________________________________________= _____

> >>>> Rpm mailing list<= /p>

> >>>> Rpm@lists=2Ebufferbloat=2Enet

> >>>> https://lists=2Ebufferbloat=2Enet/listinfo= /rpm

> >>> ______________________________________________= _

> >>> Starlink mailing list=

> >>> Starlink@lists=2Ebufferbloat=2Enet<= /span>

> >>> https://lists=2Ebufferbloat=2Enet/listinfo/sta= rlink

> >>

> >> _______________________________________________

> >> Starlink mailing list

> >> Starlink@lists=2Ebufferbloat=2Enet

> >> https://lists=2Ebufferbloat=2Enet/listinfo/starlin= k

> 

> _______________________________________________<= /span>

> Starlink mailing list

> Starlink@lists=2Ebufferbloat=2Enet=

> https://lists=2Ebufferbloat=2Enet/listinfo/starlink

--
Sent from my And= roid device with K-9 Mail=2E Please excuse my brevity=2E
------MQN5KYWMISO0XDTY8BMHBDNTZ6JHEL--