From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp120.iad3a.emailsrvr.com (smtp120.iad3a.emailsrvr.com [173.203.187.120]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 20A3A3CB47; Tue, 10 Jan 2023 12:36:58 -0500 (EST) Received: from app36.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp24.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 727201F722; Tue, 10 Jan 2023 12:36:57 -0500 (EST) Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app36.wa-webapps.iad3a (Postfix) with ESMTP id 5AF956007D; Tue, 10 Jan 2023 12:36:57 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Tue, 10 Jan 2023 12:36:57 -0500 (EST) X-Auth-ID: dpreed@deepplum.com Date: Tue, 10 Jan 2023 12:36:57 -0500 (EST) From: "David P. Reed" To: "rjmcmahon" Cc: "Dave Taht" , "Livingood, Jason" , "Rpm" , mike.reynolds@netforecast.com, "libreqos" , starlink@lists.bufferbloat.net, "bloat" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20230110123657000000_95242" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: <412c00f23a6cfef61ecbf0fd9b6f3069@rjmcmahon.com> References: <1672786712.106922180@apps.rackspace.com> <412c00f23a6cfef61ecbf0fd9b6f3069@rjmcmahon.com> X-Client-IP: 209.6.168.128 Message-ID: <1673372217.368810879@apps.rackspace.com> X-Mailer: webmail/19.0.22-RC X-Classification-ID: f25f3dac-3d51-4737-8fac-0a34cb61984d-1-1 Subject: Re: [Rpm] [Starlink] Researchers Seeking Probe Volunteers in USA X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2023 17:36:58 -0000 ------=_20230110123657000000_95242 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AOn time-sync: Every smartphone sold today can have their clocks synced, = both in rate and count value, using GPS that every smartphone has.=0A =0ASo= I think the problem of no clock sync is based on the fact that NTP and PTP= are so very, very ancient. And the tooling (iperf and netperf) don't have = much ambition (mcmahon being the exception).=0A =0AI speak on this based on= my experience implementing nanosecond accuracy clock sync among multiple c= omputers in a datacenter at TidalScale (not using PTP, because the hardware= is so unstandardized and the driver support for PTP is terrible, even in L= inux), and also using clock sync to test software defined radio transceiver= s in my hobby radios. You can do better than GPS, but frankly, GPS is enoug= h to get a stable synchronized clock outside the datacenter context.=0A =0A= The real issue is that the gear that ISP's provide for SMB and residential = access is pretty cheap and minimal - they don't provide GPS-accurate clock = timestamps. They could, but this is typical industry penny-pinching.=0A =0A= Maybe Comcast Research might fund development of devices that are inexpensi= ve, use GPS timesync, and provide end-to-end performance characterization t= ools in the form of a $100 SBC plus a good performance testing suite?=0A = =0AThis would let us characterize Starlink, too.=0A =0A =0AOn Monday, Janua= ry 9, 2023 2:13pm, "rjmcmahon" said:=0A=0A=0A=0A>= My biggest barrier is the lack of clock sync by the devices, i.e. very=0A>= limited support for PTP in data centers and in end devices. This limits=0A= > the ability to measure one way delays (OWD) and most assume that OWD is= =0A> 1/2 and RTT which typically is a mistake. We know this intuitively wit= h=0A> airplane flight times or even car commute times where the one way tim= e=0A> is not 1/2 a round trip time. Google maps & directions provide a time= =0A> estimate for the one way link. It doesn't compute a round trip and=0A>= divide by two.=0A> =0A> For those that can get clock sync working, the ipe= rf 2 --trip-times=0A> options is useful.=0A> =0A> --trip-times=0A> enable t= he measurement of end to end write to read latencies (client=0A> and server= clocks must be synchronized)=0A> =0A> Bob=0A> > I have many kvetches about= the new latency under load tests being=0A> > designed and distributed over= the past year. I am delighted! that they=0A> > are happening, but most rea= lly need third party evaluation, and=0A> > calibration, and a solid explana= tion of what network pathologies they=0A> > do and don't cover. Also a RED = team attitude towards them, as well as=0A> > thinking hard about what you a= re not measuring (operations research).=0A> >=0A> > I actually rather love = the new cloudflare speedtest, because it tests=0A> > a single TCP connectio= n, rather than dozens, and at the same time folk=0A> > are complaining that= it doesn't find the actual "speed!". yet... the=0A> > test itself more clo= sely emulates a user experience than speedtest.net=0A> > does. I am persona= lly pretty convinced that the fewer numbers of flows=0A> > that a web page = opens improves the likelihood of a good user=0A> > experience, but lack dat= a on it.=0A> >=0A> > To try to tackle the evaluation and calibration part, = I've reached out=0A> > to all the new test designers in the hope that we co= uld get together=0A> > and produce a report of what each new test is actual= ly doing. I've=0A> > tweeted, linked in, emailed, and spammed every measure= ment list I know=0A> > of, and only to some response, please reach out to o= ther test designer=0A> > folks and have them join the rpm email list?=0A> >= =0A> > My principal kvetches in the new tests so far are:=0A> >=0A> > 0) No= ne of the tests last long enough.=0A> >=0A> > Ideally there should be a mod= e where they at least run to "time of=0A> > first loss", or periodically, j= ust run longer than the=0A> > industry-stupid^H^H^H^H^H^Hstandard 20 second= s. There be dragons=0A> > there! It's really bad science to optimize the in= ternet for 20=0A> > seconds. It's like optimizing a car, to handle well, fo= r just 20=0A> > seconds.=0A> >=0A> > 1) Not testing up + down + ping at the= same time=0A> >=0A> > None of the new tests actually test the same thing t= hat the infamous=0A> > rrul test does - all the others still test up, then = down, and ping. It=0A> > was/remains my hope that the simpler parts of the = flent test suite -=0A> > such as the tcp_up_squarewave tests, the rrul test= , and the rtt_fair=0A> > tests would provide calibration to the test design= ers.=0A> >=0A> > we've got zillions of flent results in the archive publish= ed here:=0A> > https://blog.cerowrt.org/post/found_in_flent/=0A> > ps. Misi= nformation about iperf 2 impacts my ability to do this.=0A> =0A> > The new = tests have all added up + ping and down + ping, but not up +=0A> > down + p= ing. Why??=0A> >=0A> > The behaviors of what happens in that case are reall= y non-intuitive, I=0A> > know, but... it's just one more phase to add to an= y one of those new=0A> > tests. I'd be deliriously happy if someone(s) new = to the field=0A> > started doing that, even optionally, and boggled at how = it defeated=0A> > their assumptions.=0A> >=0A> > Among other things that wo= uld show...=0A> >=0A> > It's the home router industry's dirty secret than d= arn few "gigabit"=0A> > home routers can actually forward in both direction= s at a gigabit. I'd=0A> > like to smash that perception thoroughly, but giv= en our starting point=0A> > is a gigabit router was a "gigabit switch" - an= d historically been=0A> > something that couldn't even forward at 200Mbit -= we have a long way=0A> > to go there.=0A> >=0A> > Only in the past year ha= ve non-x86 home routers appeared that could=0A> > actually do a gbit in bot= h directions.=0A> >=0A> > 2) Few are actually testing within-stream latency= =0A> >=0A> > Apple's rpm project is making a stab in that direction. It loo= ks=0A> > highly likely, that with a little more work, crusader and=0A> > go= -responsiveness can finally start sampling the tcp RTT, loss and=0A> > mark= ings, more directly. As for the rest... sampling TCP_INFO on=0A> > windows,= and Linux, at least, always appeared simple to me, but I'm=0A> > discoveri= ng how hard it is by delving deep into the rust behind=0A> > crusader.=0A> = >=0A> > the goresponsiveness thing is also IMHO running WAY too many stream= s=0A> > at the same time, I guess motivated by an attempt to have the test= =0A> > complete quickly?=0A> >=0A> > B) To try and tackle the validation pr= oblem:ps. Misinformation about=0A> > iperf 2 impacts my ability to do this.= =0A> =0A> >=0A> > In the libreqos.io project we've established a testbed wh= ere tests can=0A> > be plunked through various ISP plan network emulations.= It's here:=0A> > https://payne.taht.net (run bandwidth test for what's cur= rently hooked=0A> > up)=0A> >=0A> > We could rather use an AS number and at= least a ipv4/24 and ipv6/48 to=0A> > leverage with that, so I don't have t= o nat the various emulations.=0A> > (and funding, anyone got funding?) Or, = as the code is GPLv2 licensed,=0A> > to see more test designers setup a tes= tbed like this to calibrate=0A> > their own stuff.=0A> >=0A> > Presently we= 're able to test:=0A> > flent=0A> > netperf=0A> > iperf2=0A> > iperf3=0A> >= speedtest-cli=0A> > crusader=0A> > the broadband forum udp based test:=0A>= > https://github.com/BroadbandForum/obudpst=0A> > trexx=0A> >=0A> > There'= s also a virtual machine setup that we can remotely drive a web=0A> > brows= er from (but I didn't want to nat the results to the world) to=0A> > test o= ther web services.=0A> > _______________________________________________=0A= > > Rpm mailing list=0A> > Rpm@lists.bufferbloat.net=0A> > https://lists.bu= fferbloat.net/listinfo/rpm=0A> ------=_20230110123657000000_95242 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On time-sync: Every sm= artphone sold today can have their clocks synced, both in rate and count va= lue, using GPS that every smartphone has.

=0A

 =

=0A

So I think the problem of no clock sync is base= d on the fact that NTP and PTP are so very, very ancient. And the tooling (= iperf and netperf) don't have much ambition (mcmahon being the exception).<= /p>=0A

 

=0A

I speak on thi= s based on my experience implementing nanosecond accuracy clock sync among = multiple computers in a datacenter at TidalScale (not using PTP, because th= e hardware is so unstandardized and the driver support for PTP is terrible,= even in Linux), and also using clock sync to test software defined radio t= ransceivers in my hobby radios. You can do better than GPS, but frankly, GP= S is enough to get a stable synchronized clock outside the datacenter conte= xt.

=0A

 

=0A

The real i= ssue is that the gear that ISP's provide for SMB and residential access is = pretty cheap and minimal - they don't provide GPS-accurate clock timestamps= . They could, but this is typical industry penny-pinching.

=0A

 

=0A

Maybe Comcast Research might= fund development of devices that are inexpensive, use GPS timesync, and pr= ovide end-to-end performance characterization tools in the form of a $100 S= BC plus a good performance testing suite?

=0A

 =

=0A

This would let us characterize Starlink, too.=0A

 

=0A

 

=0AOn Monday, January 9, 2023 2:13pm, "rjmcmahon" <rjmcm= ahon@rjmcmahon.com> said:

=0A
=0A

> My biggest barrier is the lack of clock s= ync by the devices, i.e. very
> limited support for PTP in data cen= ters and in end devices. This limits
> the ability to measure one w= ay delays (OWD) and most assume that OWD is
> 1/2 and RTT which typ= ically is a mistake. We know this intuitively with
> airplane fligh= t times or even car commute times where the one way time
> is not 1= /2 a round trip time. Google maps & directions provide a time
>= estimate for the one way link. It doesn't compute a round trip and
&g= t; divide by two.
>
> For those that can get clock sync wo= rking, the iperf 2 --trip-times
> options is useful.
>
> --trip-times
> enable the measurement of end to end write to= read latencies (client
> and server clocks must be synchronized)>
> Bob
> > I have many kvetches about the new l= atency under load tests being
> > designed and distributed over = the past year. I am delighted! that they
> > are happening, but = most really need third party evaluation, and
> > calibration, an= d a solid explanation of what network pathologies they
> > do an= d don't cover. Also a RED team attitude towards them, as well as
> = > thinking hard about what you are not measuring (operations research).<= br />> >
> > I actually rather love the new cloudflare spe= edtest, because it tests
> > a single TCP connection, rather tha= n dozens, and at the same time folk
> > are complaining that it = doesn't find the actual "speed!". yet... the
> > test itself mor= e closely emulates a user experience than speedtest.net
> > does= . I am personally pretty convinced that the fewer numbers of flows
>= ; > that a web page opens improves the likelihood of a good user
&g= t; > experience, but lack data on it.
> >
> > To t= ry to tackle the evaluation and calibration part, I've reached out
>= ; > to all the new test designers in the hope that we could get together=
> > and produce a report of what each new test is actually doin= g. I've
> > tweeted, linked in, emailed, and spammed every measu= rement list I know
> > of, and only to some response, please rea= ch out to other test designer
> > folks and have them join the r= pm email list?
> >
> > My principal kvetches in the n= ew tests so far are:
> >
> > 0) None of the tests las= t long enough.
> >
> > Ideally there should be a mode= where they at least run to "time of
> > first loss", or periodi= cally, just run longer than the
> > industry-stupid^H^H^H^H^H^Hs= tandard 20 seconds. There be dragons
> > there! It's really bad = science to optimize the internet for 20
> > seconds. It's like o= ptimizing a car, to handle well, for just 20
> > seconds.
&= gt; >
> > 1) Not testing up + down + ping at the same time> >
> > None of the new tests actually test the same th= ing that the infamous
> > rrul test does - all the others still = test up, then down, and ping. It
> > was/remains my hope that th= e simpler parts of the flent test suite -
> > such as the tcp_up= _squarewave tests, the rrul test, and the rtt_fair
> > tests wou= ld provide calibration to the test designers.
> >
> >= we've got zillions of flent results in the archive published here:
&g= t; > https://blog.cerowrt.org/post/found_in_flent/
> > ps. Mi= sinformation about iperf 2 impacts my ability to do this.
>
&= gt; > The new tests have all added up + ping and down + ping, but not up= +
> > down + ping. Why??
> >
> > The beha= viors of what happens in that case are really non-intuitive, I
> &g= t; know, but... it's just one more phase to add to any one of those new
> > tests. I'd be deliriously happy if someone(s) new to the field<= br />> > started doing that, even optionally, and boggled at how it d= efeated
> > their assumptions.
> >
> > Amo= ng other things that would show...
> >
> > It's the h= ome router industry's dirty secret than darn few "gigabit"
> > h= ome routers can actually forward in both directions at a gigabit. I'd
= > > like to smash that perception thoroughly, but given our starting = point
> > is a gigabit router was a "gigabit switch" - and histo= rically been
> > something that couldn't even forward at 200Mbit= - we have a long way
> > to go there.
> >
> = > Only in the past year have non-x86 home routers appeared that could> > actually do a gbit in both directions.
> >
>= ; > 2) Few are actually testing within-stream latency
> >
> > Apple's rpm project is making a stab in that direction. It look= s
> > highly likely, that with a little more work, crusader and<= br />> > go-responsiveness can finally start sampling the tcp RTT, lo= ss and
> > markings, more directly. As for the rest... sampling = TCP_INFO on
> > windows, and Linux, at least, always appeared si= mple to me, but I'm
> > discovering how hard it is by delving de= ep into the rust behind
> > crusader.
> >
> &= gt; the goresponsiveness thing is also IMHO running WAY too many streams> > at the same time, I guess motivated by an attempt to have the = test
> > complete quickly?
> >
> > B) To t= ry and tackle the validation problem:ps. Misinformation about
> >= ; iperf 2 impacts my ability to do this.
>
> >
>= ; > In the libreqos.io project we've established a testbed where tests c= an
> > be plunked through various ISP plan network emulations. I= t's here:
> > https://payne.taht.net (run bandwidth test for wha= t's currently hooked
> > up)
> >
> > We co= uld rather use an AS number and at least a ipv4/24 and ipv6/48 to
>= > leverage with that, so I don't have to nat the various emulations.> > (and funding, anyone got funding?) Or, as the code is GPLv2 li= censed,
> > to see more test designers setup a testbed like this= to calibrate
> > their own stuff.
> >
> >= Presently we're able to test:
> > flent
> > netperf<= br />> > iperf2
> > iperf3
> > speedtest-cli> > crusader
> > the broadband forum udp based test:> > https://github.com/BroadbandForum/obudpst
> > trexx=
> >
> > There's also a virtual machine setup that we= can remotely drive a web
> > browser from (but I didn't want to= nat the results to the world) to
> > test other web services.> > _______________________________________________
> >= ; Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > = https://lists.bufferbloat.net/listinfo/rpm
>

=0A
 ------=_20230110123657000000_95242--