From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bosmailout08.eigbox.net (bosmailout08.eigbox.net [66.96.185.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3ECEF3B2A4; Thu, 12 Jan 2023 16:57:46 -0500 (EST) Received: from [10.20.15.3] (helo=bosmailscan03.eigbox.net) by bosmailout08.eigbox.net with esmtp (Exim) id 1pG5Zx-0001uM-Pw; Thu, 12 Jan 2023 16:57:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=alum.mit.edu; s=dkim; h=Sender:Content-Type:MIME-Version:Message-ID:Date: Subject:In-Reply-To:References:Cc:To:From:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=MZfxxiBYM72OMzcnokmHVtcb3Bcax7nEdrhDaavzAUA=; b=kkTUN+cBLGoBgxV8mUSNDm7RP1 cqIEsfIcSHWOhQhW4j36PrCK8Pw2V57tNnc8rS7lLVIXP9XJrGEImb601eYdoxclkhpzxjwcP58tx QLe9YTZq3W3UqoTfY5dCA71Io2IodRWGfuIIkzYnkuZRwyBh2FlKj8JbNtt14B39r6qVVT022citR LR17R1wTlomJEwgrtXsurr8RPTTLKIYpHxq6Ck6WdTnNTPwJ4obq5Yq0nCVX/f7jLFDpWPzWPeoxq /C1dTDA30A6ykl3Q/h20td0FlIv3HjHb+O12TFvy5VfAONtfmOS5sv1g0LkHgs7XFjaufhc9m9v7L Aqy+VWnw==; Received: from [10.115.3.32] (helo=bosimpout12) by bosmailscan03.eigbox.net with esmtp (Exim) id 1pG5Zx-0007uW-Fn; Thu, 12 Jan 2023 16:57:45 -0500 Received: from bosauthsmtp19.yourhostingaccount.com ([10.20.18.19]) by bosimpout12 with id 7xxi290050QhFXN01xxljo; Thu, 12 Jan 2023 16:57:45 -0500 X-Authority-Analysis: v=2.3 cv=d4VuNSrE c=1 sm=1 tr=0 a=9UqFsMnAB6EOkiq4MrOclQ==:117 a=nIEF4cAZMyOU5h9mcfI6lg==:17 a=RvmDmJFTN0MA:10 a=6ulraYUaiNAA:10 a=r77TgQKjGQsHNAKrUKIA:9 a=Hc3_fNfnAAAA:8 a=kurRqvosAAAA:8 a=56-wrF8OAAAA:8 a=usUTcz4nAAAA:8 a=PMT70qdeAAAA:8 a=U7_ZthDDAAAA:8 a=Y9VdyaxHAAAA:8 a=NEAV23lmAAAA:8 a=r9QlLRt6a8zRwcwhV14A:9 a=CjuIK1q_8ugA:10 a=SSmOFEACAAAA:8 a=M6ZQNEhNAAAA:8 a=wC-Gk3RfTKlgzXYI9cwA:9 a=xz8dw0HNfjJcON7V:21 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=p1fTsZBjJ6T-B58s-fGi:22 a=kbxRQ_lfPIoQnHsAj2-A:22 a=D-cbWEhQhE2WxCfyCVJz:22 a=MqnEBYhnR1GEXjMu-uAJ:22 a=9JDB30BYp0WKznqG0cJI:22 a=Mi1b8AHvSdhFQvMrORL1:22 a=04czSPJNoGzj1I3jXP3C:22 Received: from c-67-180-86-211.hsd1.ca.comcast.net ([67.180.86.211]:59273 helo=SRA6) by bosauthsmtp19.eigbox.net with esmtpa (Exim) id 1pG5Zt-0006sm-Le; Thu, 12 Jan 2023 16:57:42 -0500 Reply-To: From: "Dick Roy" To: "'Robert McMahon'" , "'Sebastian Moeller'" Cc: , "'libreqos'" , "'David P. Reed'" , "'Rpm'" , "'bloat'" References: <202301111832.30BIWevV030127@gndrsh.dnsmgr.net> <44D8ADEB-8F89-4477-BCBD-C597A888A83E@gmx.de> <752678c9-9fdf-4abe-9915-564e3989f4d8@rjmcmahon.com> In-Reply-To: <752678c9-9fdf-4abe-9915-564e3989f4d8@rjmcmahon.com> Date: Thu, 12 Jan 2023 13:57:32 -0800 Organization: SRA Message-ID: <177980E5906F412288C57A60C82DF9D1@SRA6> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0643_01D9268D.D6AC7190" X-Mailer: Microsoft Office Outlook 11 Thread-Index: AdkmrlGjsgU61K61QaCe9IDoFqI4VQAIn3Ow X-MimeOLE: Produced By Microsoft MimeOLE X-EN-UserInfo: f809475445fb8041985048e338e1a001:931c98230c6409dcc37fa7e93b490c27 X-EN-AuthUser: dickroy@intellicommunications.com Sender: "Dick Roy" X-EN-OrigIP: 67.180.86.211 X-EN-OrigHost: c-67-180-86-211.hsd1.ca.comcast.net Subject: Re: [LibreQoS] [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2023 21:57:46 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0643_01D9268D.D6AC7190 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit FYI . https://www.fiercewireless.com/tech/cbrs-based-fwa-beats-starlink-performanc e-madden Nothing earth-shaking :-) RR _____ From: Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On Behalf Of Robert McMahon via Starlink Sent: Thursday, January 12, 2023 9:50 AM To: Sebastian Moeller Cc: Dave Taht via Starlink; mike.reynolds@netforecast.com; libreqos; David P. Reed; Rpm; bloat Subject: Re: [Starlink] [Rpm] Researchers Seeking Probe Volunteers in USA Hi Sebastien, You make a good point. What I did was issue a warning if the tool found it was being CPU limited vs i/o limited. This indicates the i/o test likely is inaccurate from an i/o perspective, and the results are suspect. It does this crudely by comparing the cpu thread doing stats against the traffic threads doing i/o, which thread is waiting on the others. There is no attempt to assess the cpu load itself. So it's designed with a singular purpose of making sure i/o threads only block on syscalls of write and read. I probably should revisit this both in design and implementation. Thanks for bringing it up and all input is truly appreciated. Bob On Jan 12, 2023, at 12:14 AM, Sebastian Moeller wrote: Hi Bob, On Jan 11, 2023, at 21:09, rjmcmahon wrote: Iperf 2 is designed to measure network i/o. Note: It doesn't have to move large amounts of data. It can support data profiles that don't drive TCP's CCA as an example. Two things I've been asked for and avoided: 1) Integrate clock sync into iperf's test traffic [SM] This I understand, measurement conditions can be unsuited for tight time synchronization... 2) Measure and output CPU usages [SM] This one puzzles me, as far as I understand the only way to properly diagnose network issues is to rule out other things like CPU overload that can have symptoms similar to network issues. As an example, the cake qdisc will if CPU cycles become tight first increases its internal queueing and jitter (not consciously, it is just an observation that once cake does not get access to the CPU as timely as it wants, queuing latency and variability increases) and then later also shows reduced throughput, so similar things that can happen along an e2e network path for completely different reasons, e.g. lower level retransmissions or a variable rate link. So i would think that checking the CPU load at least coarse would be within the scope of network testing tools, no? Regards Sebastian I think both of these are outside the scope of a tool designed to test network i/o over sockets, rather these should be developed & validated independently of a network i/o tool. Clock error really isn't about amount/frequency of traffic but rather getting a periodic high-quality reference. I tend to use GPS pulse per second to lock the local system oscillator to. As David says, most every modern handheld computer has the GPS chips to do this already. So to me it seems more of a policy choice between data center operators and device mfgs and less of a technical issue. Bob Hello, Yall can call me crazy if you want.. but... see below [RWG] Hi Bib, On Jan 9, 2023, at 20:13, rjmcmahon via Starlink wrote: My biggest barrier is the lack of clock sync by the devices, i.e. very limited support for PTP in data centers and in end devices. This limits the ability to measure one way delays (OWD) and most assume that OWD is 1/2 and RTT which typically is a mistake. We know this intuitively with airplane flight times or even car commute times where the one way time is not 1/2 a round trip time. Google maps & directions provide a time estimate for the one way link. It doesn't compute a round trip and divide by two. For those that can get clock sync working, the iperf 2 --trip-times options is useful. [SM] +1; and yet even with unsynchronized clocks one can try to measure how latency changes under load and that can be done per direction. Sure this is far inferior to real reliably measured OWDs, but if life/the internet deals you lemons.... [RWG] iperf2/iperf3, etc are already moving large amounts of data back and forth, for that matter any rate test, why not abuse some of that data and add the fundemental NTP clock sync data and bidirectionally pass each others concept of "current time". IIRC (its been 25 years since I worked on NTP at this level) you *should* be able to get a fairly accurate clock delta between each end, and then use that info and time stamps in the data stream to compute OWD's. You need to put 4 time stamps in the packet, and with that you can compute "offset". --trip-times enable the measurement of end to end write to read latencies (client and server clocks must be synchronized) [RWG] --clock-skew enable the measurement of the wall clock difference between sender and receiver [SM] Sweet! Regards Sebastian Bob I have many kvetches about the new latency under load tests being designed and distributed over the past year. I am delighted! that they are happening, but most really need third party evaluation, and calibration, and a solid explanation of what network pathologies they do and don't cover. Also a RED team attitude towards them, as well as thinking hard about what you are not measuring (operations research). I actually rather love the new cloudflare speedtest, because it tests a single TCP connection, rather than dozens, and at the same time folk are complaining that it doesn't find the actual "speed!". yet... the test itself more closely emulates a user experience than speedtest.net does. I am personally pretty convinced that the fewer numbers of flows that a web page opens improves the likelihood of a good user experience, but lack data on it. To try to tackle the evaluation and calibration part, I've reached out to all the new test designers in the hope that we could get together and produce a report of what each new test is actually doing. I've tweeted, linked in, emailed, and spammed every measurement list I know of, and only to some response, please reach out to other test designer folks and have them join the rpm email list? My principal kvetches in the new tests so far are: 0) None of the tests last long enough. Ideally there should be a mode where they at least run to "time of first loss", or periodically, just run longer than the industry-stupid^H^H^H^H^H^Hstandard 20 seconds. There be dragons there! It's really bad science to optimize the internet for 20 seconds. It's like optimizing a car, to handle well, for just 20 seconds. 1) Not testing up + down + ping at the same time None of the new tests actually test the same thing that the infamous rrul test does - all the others still test up, then down, and ping. It was/remains my hope that the simpler parts of the flent test suite - such as the tcp_up_squarewave tests, the rrul test, and the rtt_fair tests would provide calibration to the test designers. we've got zillions of flent results in the archive published here: https://blog.cerowrt.org/post/found_in_flent/ ps. Misinformation about iperf 2 impacts my ability to do this. The new tests have all added up + ping and down + ping, but not up + down + ping. Why?? The behaviors of what happens in that case are really non-intuitive, I know, but... it's just one more phase to add to any one of those new tests. I'd be deliriously happy if someone(s) new to the field started doing that, even optionally, and boggled at how it defeated their assumptions. Among other things that would show... It's the home router industry's dirty secret than darn few "gigabit" home routers can actually forward in both directions at a gigabit. I'd like to smash that perception thoroughly, but given our starting point is a gigabit router was a "gigabit switch" - and historically been something that couldn't even forward at 200Mbit - we have a long way to go there. Only in the past year have non-x86 home routers appeared that could actually do a gbit in both directions. 2) Few are actually testing within-stream latency Apple's rpm project is making a stab in that direction. It looks highly likely, that with a little more work, crusader and go-responsiveness can finally start sampling the tcp RTT, loss and markings, more directly. As for the rest... sampling TCP_INFO on windows, and Linux, at least, always appeared simple to me, but I'm discovering how hard it is by delving deep into the rust behind crusader. the goresponsiveness thing is also IMHO running WAY too many streams at the same time, I guess motivated by an attempt to have the test complete quickly? B) To try and tackle the validation problem:ps. Misinformation about iperf 2 impacts my ability to do this. In the libreqos.io project we've established a testbed where tests can be plunked through various ISP plan network emulations. It's here: https://payne.taht.net (run bandwidth test for what's currently hooked up) We could rather use an AS number and at least a ipv4/24 and ipv6/48 to leverage with that, so I don't have to nat the various emulations. (and funding, anyone got funding?) Or, as the code is GPLv2 licensed, to see more test designers setup a testbed like this to calibrate their own stuff. Presently we're able to test: flent netperf iperf2 iperf3 speedtest-cli crusader the broadband forum udp based test: https://github.com/BroadbandForum/obudpst trexx There's also a virtual machine setup that we can remotely drive a web browser from (but I didn't want to nat the results to the world) to test other web services. _____ Rpm mailing list Rpm@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/rpm _____ Starlink mailing list Starlink@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/starlink _____ Starlink mailing list Starlink@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/starlink ------=_NextPart_000_0643_01D9268D.D6AC7190 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

FYI = …

 

https://www.fiercewireless.com/tech/cbrs-based-fwa-b= eats-starlink-performance-madden

 

Nothing earth-shaking = J


RR

 

From: = Starlink [mailto:starlink-bounces@lists.bufferbloat.net] On Behalf Of Robert McMahon via Starlink
Sent: Thursday, January = 12, 2023 9:50 AM
To: Sebastian Moeller
Cc: Dave Taht via = Starlink; mike.reynolds@netforecast.com; libreqos; David P. Reed; Rpm; bloat
Subject: Re: [Starlink] = [Rpm] Researchers Seeking Probe Volunteers in USA=

 

Hi = Sebastien,

You make a = good point. What I did was issue a warning if the tool found it was being CPU = limited vs i/o limited. This indicates the i/o test likely is inaccurate from an = i/o perspective, and the results are suspect. It does this crudely by = comparing the cpu thread doing stats against the traffic threads doing i/o, which = thread is waiting on the others. There is no attempt to assess the cpu load = itself. So it's designed with a singular purpose of making sure i/o threads only = block on syscalls of write and read.

I probably = should revisit this both in design and implementation. Thanks for bringing it up and = all input is truly appreciated.

Bob

On Jan 12, 2023, at 12:14 AM, Sebastian Moeller <moeller0@gmx.de> = wrote:

Hi Bob,




 On Jan 11, 2023, at 21:09, rjmcmahon =
<rjmcmahon@rjmcmahon.com> wrote:

 

 Iperf 2 is designed to measure network i/o. Note: It doesn't have =
to move large amounts of data. It can support data profiles that don't =
drive TCP's CCA as an example.

 

 Two things I've been asked for and avoided:

 

 1) Integrate clock sync into iperf's test =
traffic


 [SM] This I understand, measurement conditions can be unsuited for =
tight time synchronization...




 2) Measure and output CPU =
usages


 [SM] This one puzzles me, as far as I understand the only way to =
properly diagnose network issues is to rule out other things like CPU =
overload that can have symptoms similar to network issues. As an =
example, the cake qdisc will if CPU cycles become tight first increases =
its internal queueing and jitter (not consciously, it is just an =
observation that once cake does not get access to the CPU as timely as =
it wants, queuing latency and variability increases) and then later also =
shows reduced throughput, so similar things that can happen along an e2e =
network path for completely different reasons, e.g. lower level =
retransmissions or a variable rate link. So i would think that checking =
the CPU load at least coarse would be within the scope of network =
testing tools, no?



Regards

 Sebastian








 I think both of these are outside the =
scope of a tool designed to test network i/o over sockets, rather these =
should be developed & validated independently of a network i/o =
tool.

 

 Clock error really isn't about amount/frequency of traffic but =
rather getting a periodic high-quality reference. I tend to use GPS =
pulse per second to lock the local system oscillator to. As David says, =
most every modern handheld computer has the GPS chips to do this =
already. So to me it seems more of a policy choice between data center =
operators and device mfgs and less of a technical issue.

 

 Bob
 Hello,

  Yall can call me crazy if you want.. but... see below =
[RWG]
 Hi =
Bib,
 On Jan =
9, 2023, at 20:13, rjmcmahon via Starlink =
<starlink@lists.bufferbloat.net> wrote:



 My biggest barrier is the lack of clock sync by the devices, i.e. =
very limited support for PTP in data centers and in end devices. This =
limits the ability to measure one way delays (OWD) and most assume that =
OWD is 1/2 and RTT which typically is a mistake. We know this =
intuitively with airplane flight times or even car commute times where =
the one way time is not 1/2 a round trip time. Google maps & =
directions provide a time estimate for the one way link. It doesn't =
compute a round trip and divide by two.



 For those that can get clock sync working, the iperf 2 =
--trip-times options is useful.
  =
[SM] +1; and yet even with unsynchronized clocks one can try to measure =
how latency changes under load and that can be done per direction. Sure =
this is far inferior to real reliably measured OWDs, but if life/the =
internet deals you lemons....
 [RWG] =
iperf2/iperf3, etc are already moving large amounts of data

 back and forth, for that matter any rate test, why not abuse some =
of

 that data and add the fundemental NTP clock sync data and

 bidirectionally pass each others concept of "current =
time".  IIRC (its

 been 25 years since I worked on NTP at this level) you *should* =
be

 able to get a fairly accurate clock delta between each end, and =
then

 use that info and time stamps in the data stream to compute =
OWD's.

 You need to put 4 time stamps in the packet, and with that you =
can

 compute "offset".


 --trip-times

  enable the measurement of end to end write to read latencies =
(client and server clocks must be =
synchronized)
 [RWG] --clock-skew

  enable the measurement of the wall clock difference between =
sender and receiver
  =
[SM] Sweet!

 Regards

  Sebastian


 Bob
 I have many kvetches about the new latency =
under load tests being

 designed and distributed over the past year. I am delighted! that =
they

 are happening, but most really need third party evaluation, =
and

 calibration, and a solid explanation of what network pathologies =
they

 do and don't cover. Also a RED team attitude towards them, as well =
as

 thinking hard about what you are not measuring (operations =
research).

 I actually rather love the new cloudflare speedtest, because it =
tests

 a single TCP connection, rather than dozens, and at the same time =
folk

 are complaining that it doesn't find the actual =
"speed!". yet... the

 test itself more closely emulates a user experience than speedtest.net

 does. I am personally pretty convinced that the fewer numbers of =
flows

 that a web page opens improves the likelihood of a good user

 experience, but lack data on it.

 To try to tackle the evaluation and calibration part, I've reached =
out

 to all the new test designers in the hope that we could get =
together

 and produce a report of what each new test is actually doing. =
I've

 tweeted, linked in, emailed, and spammed every measurement list I =
know

 of, and only to some response, please reach out to other test =
designer

 folks and have them join the rpm email list?

 My principal kvetches in the new tests so far are:

 0) None of the tests last long enough.

 Ideally there should be a mode where they at least run to =
"time of

 first loss", or periodically, just run longer than the

 industry-stupid^H^H^H^H^H^Hstandard 20 seconds. There be =
dragons

 there! It's really bad science to optimize the internet for 20

 seconds. It's like optimizing a car, to handle well, for just =
20

 seconds.

 1) Not testing up + down + ping at the same time

 None of the new tests actually test the same thing that the =
infamous

 rrul test does - all the others still test up, then down, and =
ping. It

 was/remains my hope that the simpler parts of the flent test suite =
-

 such as the tcp_up_squarewave tests, the rrul test, and the =
rtt_fair

 tests would provide calibration to the test designers.

 we've got zillions of flent results in the archive published =
here:

 https://blog.cerowr=
t.org/post/found_in_flent/

 ps. Misinformation about iperf 2 impacts my ability to do =
this.
 
 The new =
tests have all added up + ping and down + ping, but not up +

 down + ping. Why??

 The behaviors of what happens in that case are really =
non-intuitive, I

 know, but... it's just one more phase to add to any one of those =
new

 tests. I'd be deliriously happy if someone(s) new to the field

 started doing that, even optionally, and boggled at how it =
defeated

 their assumptions.

 Among other things that would show...

 It's the home router industry's dirty secret than darn few =
"gigabit"

 home routers can actually forward in both directions at a gigabit. =
I'd

 like to smash that perception thoroughly, but given our starting =
point

 is a gigabit router was a "gigabit switch" - and =
historically been

 something that couldn't even forward at 200Mbit - we have a long =
way

 to go there.

 Only in the past year have non-x86 home routers appeared that =
could

 actually do a gbit in both directions.

 2) Few are actually testing within-stream latency

 Apple's rpm project is making a stab in that direction. It =
looks

 highly likely, that with a little more work, crusader and

 go-responsiveness can finally start sampling the tcp RTT, loss =
and

 markings, more directly. As for the rest... sampling TCP_INFO =
on

 windows, and Linux, at least, always appeared simple to me, but =
I'm

 discovering how hard it is by delving deep into the rust =
behind

 crusader.

 the goresponsiveness thing is also IMHO running WAY too many =
streams

 at the same time, I guess motivated by an attempt to have the =
test

 complete quickly?

 B) To try and tackle the validation problem:ps. Misinformation =
about iperf 2 impacts my ability to do =
this.
 
 In the =
libreqos.io project we've established a =
testbed where tests can

 be plunked through various ISP plan network emulations. It's =
here:

 https://payne.taht.net (run =
bandwidth test for what's currently hooked

 up)

 We could rather use an AS number and at least a ipv4/24 and =
ipv6/48 to

 leverage with that, so I don't have to nat the various =
emulations.

 (and funding, anyone got funding?) Or, as the code is GPLv2 =
licensed,

 to see more test designers setup a testbed like this to =
calibrate

 their own stuff.

 Presently we're able to test:

 flent

 netperf

 iperf2

 iperf3

 speedtest-cli

 crusader

 the broadband forum udp based test:

 https://github.com/Bro=
adbandForum/obudpst

 trexx

 There's also a virtual machine setup that we can remotely drive a =
web

 browser from (but I didn't want to nat the results to the world) =
to

 test other web services.








 Rpm mailing list

 Rpm@lists.bufferbloat.net

 https://lists.bufferb=
loat.net/listinfo/rpm








 Starlink mailing list

 Starlink@lists.bufferbloat.net

 https://lists.bu=
fferbloat.net/listinfo/starlink








 Starlink mailing list

 Starlink@lists.bufferbloat.net

 https://lists.bu=
fferbloat.net/listinfo/starlink
 
------=_NextPart_000_0643_01D9268D.D6AC7190--