Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[Dave Taht]

Thank you very much for the steer to RFC9097. I'd completely missed that.

On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
>
> (astute readers may have guessed that I pressed "send" too soon on previous message...)
>
> I also conducted upstream tests this time, here are the results:
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
>
> Net Qual                           UDPST (RFC9097)              Ookla
> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap    Ping(no load)
> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22       8
> 22         281 m 214ms  8ms        27 (52)  25       5-248      22       8
> 22         290 m 207ms  8ms        27 (55)  28       0-253      22       9
> 21         330 m 182ms  11ms       23 (44)  28       0-255      22       7
> 22         334 m 180ms  9ms        33 (56)  25       0-255      22       9
>
> The Upstream capacity measurements reflect an interesting feature that we can reliably and repeatably measure with UDPST. The first ~3 seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this behavior in its 1 second sub-interval measurements and has a bimodal reporting option that divides the complete measurement interval in two time intervals to report an initial (turbo) max capacity and a steady-state max capacity for the later intervals. The UDPST capacity results present both measurements: steady-state first.

Certainly we can expect bi-model distributions from many ISPs, as, for
one thing, the "speedboost" concept remains popular, except that it's
misnamed, as it should be called speed-subtract or speed-lose. Worse,
it is often configured "sneakily", in that it doesn't kick in for the
typical observed duration of the test, for some, they cut the
available bandwidth about 20s in, others, 1 or 5 minutes.

One of my biggest issues with the rpm spec so far is that it should,
at least, sometimes, run randomly longer than the overly short
interval it runs for and the tools also allow for manual override of length.

we caught a lot of tomfoolery with flent's rrul test running by default for 1m.

Also, AQMs on the path can take a while to find the optimal drop or mark rate.

>
> The capacity processing in networkQuality and Ookla appear to report the steady-state result.

Ookla used to basically report the last result. Also it's not a good
indicator of web traffic behavior at all, watching the curve
go up much more slowly in their test on say, fiber 2ms, vs starlink, (40ms)....

So adding another mode - how quickly is peak bandwidth actually
reached, would be nice.

I haven't poked into the current iteration of the goresponsiveness
test at all: https://github.com/network-quality/goresponsiveness it
would be good to try collecting more statistics and histograms and
methods of analyzing the data in that libre-source version.

How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?

>I watched the upstream capacity measurements on the Ookla app, and could easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of the test which determined the final result.

I tend to get upset when I see ookla's new test flash a peak result in
the seconds and then settle on some lower number somehow.
So far as I know they are only sampling the latency every 250ms.

>
> The working latency is about 200ms in networkQuality and about 280ms as measured by UDPST (RFC9097). Note that the networkQuality minimum delay is ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference in working latency.  Also, we used the very dynamic Type C load adjustment/search algorithm in UDPST during all of this testing, which could explain the higher working latency to some degree.
>
> So, it's worth noting that the measurements needed for assessing working latency/responsiveness are available in the UDPST utility, and that the UDPST measurements are conducted on UDP transport (used by a growing fraction of Internet traffic).

Thx, didn't know of this work til now!

have you tried irtt?

>
> comments welcome of course,
> Al
>
> > -----Original Message-----
> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > Sent: Sunday, October 30, 2022 8:09 PM
> > To: ippm@ietf.org
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> >
> > Hi again RPM friends and IPPM'ers,
> >
> > As promised, I repeated the tests shared last week, this time using both the
> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > followed Sebastian's calculations as well.
> >
> > Working Latency & Capacity Summary
> >
> > Net Qual                           UDPST                        Ookla
> > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> > Ping(no load)
> > 885       916 m  66ms   8ms        970      28       0-20       940      8
> > 888      1355 h  44ms   8ms        966      28       0-23       940      8
> > 891      1109 h  54ms   8ms        968      27       0-19       940      9
> > 887      1141 h  53ms   11ms       966      27       0-18       937      7
> > 884      1151 h  52ms   9ms        968      28       0-20       937      9
> >
> > With the sequential test option, I noticed that networkQuality achieved nearly
> > the maximum capacity reported almost immediately at the start of a test.
> > However, the reported capacities are low by about 60Mbps, especially when
> > compared to the Ookla TCP measurements.
> >
> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the
> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality
> > RPM measurements were categorized as "High". There doesn't seem to be much
> > buffering in the downstream direction.
> >
> >
> >
> > > -----Original Message-----
> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > Sent: Monday, October 24, 2022 6:36 PM
> > > To: ippm@ietf.org
> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > metrics
> > >
> > >
> > > Hi RPM friends and IPPM'ers,
> > >
> > > I was wondering what a comparison of some of the "working latency" metrics
> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the
> > > downlink provisioned for 1Gbps.
> > >
> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > Speedtest
> > > with as similar connectivity as possible (but we know that the traffic will
> > > diverge to different servers and we can't change that aspect).
> > >
> > > Here's a quick summary of yesterday's results:
> > >
> > > Working Latency & Capacity Summary
> > >
> > > Net Qual                UDPST                        Ookla
> > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no load)
> > > 878       62            970      28       0-19       941      6
> > > 891       92            970      27       0-20       940      7
> > > 891       120           966      28       0-22       937      9
> > > 890       112           970      28       0-21       940      8
> > > 903       70            970      28       0-16       935      9
> > >
> > > Note: all RPM values were categorized as Low.
> > >
> > > networkQuality downstream capacities are always on the low side compared to
> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla
> > > achieved. I think that a longer test duration might be needed to achieve the
> > > actual 1Gbps capacity with networkQuality; intermediate values observed were
> > > certainly headed in the right direction. (I recently upgraded to Monterey
> > 12.6
> > > on my MacBook, so should have the latest version.)
> > >
> > > Also, as Sebastian Moeller's message to the list reminded me, I should have
> > > run the tests with the -v option to help with comparisons. I'll repeat this
> > > test when I can make time.
> > >
> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and
> > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access
> > and
> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were
> > run
> > > on an Ubuntu VM in my MacBook.
> > >
> > > The big disappointment was that the Ookla desktop app I updated over the
> > > weekend did not include the new responsiveness metric! I included the ping
> > > results anyway, and it was clearly using a server in the nearby area.
> > >
> > > So, I have some more work to do, but I hope this is interesting-enough to
> > > start some comparison discussions, and bring-out some suggestions.
> > >
> > > happy testing all,
> > > Al
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >
> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[rjmcmahon]

Would it be possible to get some iperf 2 bounceback test results too?

https://sourceforge.net/projects/iperf2/

Also, for the hunt algo, maybe use TCP first to get a starting point and 
then hunt? Just a thought.

Thanks,
Bob
> Thank you very much for the steer to RFC9097. I'd completely missed 
> that.
> 
> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> 
> wrote:
>> 
>> (astute readers may have guessed that I pressed "send" too soon on 
>> previous message...)
>> 
>> I also conducted upstream tests this time, here are the results:
>> (capacity in Mbps, delays in ms, h and m are RPM categories, High and 
>> Medium)
>> 
>> Net Qual                           UDPST (RFC9097)              Ookla
>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap  
>>   Ping(no load)
>> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22     
>>   8
>> 22         281 m 214ms  8ms        27 (52)  25       5-248      22     
>>   8
>> 22         290 m 207ms  8ms        27 (55)  28       0-253      22     
>>   9
>> 21         330 m 182ms  11ms       23 (44)  28       0-255      22     
>>   7
>> 22         334 m 180ms  9ms        33 (56)  25       0-255      22     
>>   9
>> 
>> The Upstream capacity measurements reflect an interesting feature that 
>> we can reliably and repeatably measure with UDPST. The first ~3 
>> seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST 
>> displays this behavior in its 1 second sub-interval measurements and 
>> has a bimodal reporting option that divides the complete measurement 
>> interval in two time intervals to report an initial (turbo) max 
>> capacity and a steady-state max capacity for the later intervals. The 
>> UDPST capacity results present both measurements: steady-state first.
> 
> Certainly we can expect bi-model distributions from many ISPs, as, for
> one thing, the "speedboost" concept remains popular, except that it's
> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> it is often configured "sneakily", in that it doesn't kick in for the
> typical observed duration of the test, for some, they cut the
> available bandwidth about 20s in, others, 1 or 5 minutes.
> 
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short
> interval it runs for and the tools also allow for manual override of 
> length.
> 
> we caught a lot of tomfoolery with flent's rrul test running by default 
> for 1m.
> 
> Also, AQMs on the path can take a while to find the optimal drop or 
> mark rate.
> 
>> 
>> The capacity processing in networkQuality and Ookla appear to report 
>> the steady-state result.
> 
> Ookla used to basically report the last result. Also it's not a good
> indicator of web traffic behavior at all, watching the curve
> go up much more slowly in their test on say, fiber 2ms, vs starlink, 
> (40ms)....
> 
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
> 
> I haven't poked into the current iteration of the goresponsiveness
> test at all: https://github.com/network-quality/goresponsiveness it
> would be good to try collecting more statistics and histograms and
> methods of analyzing the data in that libre-source version.
> 
> How does networkQuality compare vs a vs your tool vs a vs 
> goresponsiveness?
> 
>> I watched the upstream capacity measurements on the Ookla app, and 
>> could easily see the initial rise to 40-50Mbps, then the drop to 
>> ~22Mbps for most of the test which determined the final result.
> 
> I tend to get upset when I see ookla's new test flash a peak result in
> the seconds and then settle on some lower number somehow.
> So far as I know they are only sampling the latency every 250ms.
> 
>> 
>> The working latency is about 200ms in networkQuality and about 280ms 
>> as measured by UDPST (RFC9097). Note that the networkQuality minimum 
>> delay is ~20ms lower than the UDPST RTTmin, so this accounts for some 
>> of the difference in working latency.  Also, we used the very dynamic 
>> Type C load adjustment/search algorithm in UDPST during all of this 
>> testing, which could explain the higher working latency to some 
>> degree.
>> 
>> So, it's worth noting that the measurements needed for assessing 
>> working latency/responsiveness are available in the UDPST utility, and 
>> that the UDPST measurements are conducted on UDP transport (used by a 
>> growing fraction of Internet traffic).
> 
> Thx, didn't know of this work til now!
> 
> have you tried irtt?
> 
>> 
>> comments welcome of course,
>> Al
>> 
>> > -----Original Message-----
>> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > Sent: Sunday, October 30, 2022 8:09 PM
>> > To: ippm@ietf.org
>> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> > metrics
>> >
>> >
>> > Hi again RPM friends and IPPM'ers,
>> >
>> > As promised, I repeated the tests shared last week, this time using both the
>> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
>> > followed Sebastian's calculations as well.
>> >
>> > Working Latency & Capacity Summary
>> >
>> > Net Qual                           UDPST                        Ookla
>> > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
>> > Ping(no load)
>> > 885       916 m  66ms   8ms        970      28       0-20       940      8
>> > 888      1355 h  44ms   8ms        966      28       0-23       940      8
>> > 891      1109 h  54ms   8ms        968      27       0-19       940      9
>> > 887      1141 h  53ms   11ms       966      27       0-18       937      7
>> > 884      1151 h  52ms   9ms        968      28       0-20       937      9
>> >
>> > With the sequential test option, I noticed that networkQuality achieved nearly
>> > the maximum capacity reported almost immediately at the start of a test.
>> > However, the reported capacities are low by about 60Mbps, especially when
>> > compared to the Ookla TCP measurements.
>> >
>> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the
>> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality
>> > RPM measurements were categorized as "High". There doesn't seem to be much
>> > buffering in the downstream direction.
>> >
>> >
>> >
>> > > -----Original Message-----
>> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > Sent: Monday, October 24, 2022 6:36 PM
>> > > To: ippm@ietf.org
>> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
>> > > metrics
>> > >
>> > >
>> > > Hi RPM friends and IPPM'ers,
>> > >
>> > > I was wondering what a comparison of some of the "working latency" metrics
>> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the
>> > > downlink provisioned for 1Gbps.
>> > >
>> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
>> > Speedtest
>> > > with as similar connectivity as possible (but we know that the traffic will
>> > > diverge to different servers and we can't change that aspect).
>> > >
>> > > Here's a quick summary of yesterday's results:
>> > >
>> > > Working Latency & Capacity Summary
>> > >
>> > > Net Qual                UDPST                        Ookla
>> > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no load)
>> > > 878       62            970      28       0-19       941      6
>> > > 891       92            970      27       0-20       940      7
>> > > 891       120           966      28       0-22       937      9
>> > > 890       112           970      28       0-21       940      8
>> > > 903       70            970      28       0-16       935      9
>> > >
>> > > Note: all RPM values were categorized as Low.
>> > >
>> > > networkQuality downstream capacities are always on the low side compared to
>> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla
>> > > achieved. I think that a longer test duration might be needed to achieve the
>> > > actual 1Gbps capacity with networkQuality; intermediate values observed were
>> > > certainly headed in the right direction. (I recently upgraded to Monterey
>> > 12.6
>> > > on my MacBook, so should have the latest version.)
>> > >
>> > > Also, as Sebastian Moeller's message to the list reminded me, I should have
>> > > run the tests with the -v option to help with comparisons. I'll repeat this
>> > > test when I can make time.
>> > >
>> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and
>> > > the range of variation above the minimum (RTTVarRnge) add-up to very
>> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access
>> > and
>> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
>> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were
>> > run
>> > > on an Ubuntu VM in my MacBook.
>> > >
>> > > The big disappointment was that the Ookla desktop app I updated over the
>> > > weekend did not include the new responsiveness metric! I included the ping
>> > > results anyway, and it was clearly using a server in the nearby area.
>> > >
>> > > So, I have some more work to do, but I hope this is interesting-enough to
>> > > start some comparison discussions, and bring-out some suggestions.
>> > >
>> > > happy testing all,
>> > > Al
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > ippm mailing list
>> > > ippm@ietf.org
>> > >
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > >
>> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>> >
>> > _______________________________________________
>> > ippm mailing list
>> > ippm@ietf.org
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>> 
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

> -----Original Message-----
> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
> Sent: Monday, October 31, 2022 2:53 PM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: MORTON JR., AL <acmorton@att.com>; Rpm <rpm@lists.bufferbloat.net>;
> ippm@ietf.org
> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> Latency" metrics
> 
> Would it be possible to get some iperf 2 bounceback test results too?
> 
> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
> gmYpYN3pBO-aWMfjDRdVRFQ20aHQ5nDHOhEVY1y-
> MkFFyH8YmM4wf8cEtaxzvcwTMaCaJOCNRBtj0tnz9A$
> 
> Also, for the hunt algo, maybe use TCP first to get a starting point and
> then hunt? Just a thought.
> 
> Thanks,
> Bob
> >
<snip>

Thanks for your suggestion, Bob, and it's nice to meet you! 
I was only familiar with the "old" iperf2 at:
https://iperf.fr/iperf-doc.php
before your message and URL arrived (yet another segment of the elephant!).

I didn't quickly find whether your code will run on linux (ubuntu for me). I suppose I use the .tar.gz and compile locally. Let me know, I've got some other tools to check-out first.

You suggest the bounceback test option (I found it on the man page), but there are lots of options and six versions of bounceback!  Based on the results I already reported, can you suggest a specific version of bounceback and a set of options that would be a good first try?
(see the man page at https://iperf2.sourceforge.io/iperf-manpage.html )

Regarding our hunt algorithms (search for max), the new Type C algo (in release 7.5.0) locates the Max very quickly. I showed a comparison of our Type B and Type C search algorithms on a slide at the IPPM meeting in July. See Slide 10:
https://datatracker.ietf.org/meeting/114/materials/slides-114-ippm-sec-dir-review-discussion-test-protocol-for-one-way-ip-capacity-measurement-00

The measured capacity reaches 1 Gbps in about 1 second with the Type C algorithm, without choosing or testing for a starting point (our default starting point rate is ~500kbps, very low to accommodate "any" subscribed rate).

regards, and thanks again,
Al

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[rjmcmahon]

One can download iperf version 2.1.8 tarball or use git to get the 
latest and compile from master. Sourceforge supports both. 
https://sourceforge.net/projects/iperf2/

A simple test is to use the --bounceback and then, with load, add 
--bounceback-congest (see below) or maybe use one way congestion vs the 
default of full-duplex.

Build instructions is in the INSTALL file. Should work on linux, 
windows, BSD, etc.

I think with WiFi the distributions may be more than bi-modal and one 
second decisions may be too short. It doesn't mean that the traffic has 
to be active and at capacity per the full decision interval, rather the 
forwarding/phy state machines & other logic may need more than one 
second to find their local optimums. Also, maybe some sort of online 
mean & min shift detections could help? Or one can use the inP metric 
too. This is calculated per Little's law. Though OWD requires clock sync 
indicated as being done to iperf via the --trip-times option.

I added your delay docs into the iperf2 docs directory. Thanks for 
mentioning them. They're helpful. 
https://sourceforge.net/p/iperf2/code/ci/master/tree/doc/delay_docs/

Bob

[root@ctrl1fc35 ~]# iperf -c 192.168.1.231 --bounceback
------------------------------------------------------------
Client connecting to 192.168.1.231, TCP port 5001 with pid 717413 (1 
flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs & 
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.15%enp2s0 port 43538 connected with 192.168.1.231 
port 5001 (bb w/quickack len/hold=100/0) (sock=3) 
(icwnd/mss/irtt=14/1448/4442) (ct=4.55 ms) on 2022-10-31 15:25:56 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB 
cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.802/2.008/5.765/1.243 ms    0   14K/3958 us    263 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.836/2.179/5.112/0.884 ms    0   14K/3727 us    261 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.941/3.576/4.221/0.159 ms    0   14K/3800 us    254 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=4.042/3.790/4.847/0.295 ms    0   14K/3853 us    247 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.930/3.699/4.162/0.123 ms    0   14K/3834 us    254 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=4.037/3.752/4.612/0.238 ms    0   14K/3858 us    248 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.941/3.827/4.045/0.078 ms    0   14K/3833 us    254 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=4.042/3.805/4.778/0.277 ms    0   14K/3869 us    247 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=3.935/3.749/4.099/0.127 ms    0   14K/3822 us    254 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=4.041/3.783/4.885/0.320 ms    0   14K/3863 us    247 rps
[  1] 0.00-10.02 sec  19.5 KBytes  16.0 Kbits/sec    
100=3.955/2.008/5.765/0.503 ms    0   14K/3872 us    253 rps
[  1] 0.00-10.02 sec BB8(f)-PDF: 
bin(w=100us):cnt(100)=21:1,22:1,24:1,28:1,30:1,32:1,36:2,37:2,38:6,39:20,40:24,41:25,42:4,43:1,44:1,47:1,48:1,49:3,51:1,52:2,58:1 
(5.00/95.00/99.7%=30/49/58,Outliers=0,obl/obu=0/0)

[root@ctrl1fc35 ~]# iperf -c 192.168.1.231 --bounceback 
--bounceback-congest
------------------------------------------------------------
Client connecting to 192.168.1.231, TCP port 5001 with pid 717416 (1 
flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs & 
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  2] local 192.168.1.15%enp2s0 port 43540 connected with 192.168.1.231 
port 5001 (full-duplex) (sock=4) (qack) (icwnd/mss/irtt=14/1448/4000) 
(ct=4.10 ms) on 2022-10-31 15:26:15 (PDT)
[  1] local 192.168.1.15%enp2s0 port 43542 connected with 192.168.1.231 
port 5001 (bb w/quickack len/hold=100/0) (sock=3) 
(icwnd/mss/irtt=14/1448/4003) (ct=4.10 ms) on 2022-10-31 15:26:15 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB 
cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=13.770/1.855/50.097/15.368 ms    0   14K/14387 us    73 rps
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
Cwnd/RTT(var)        NetPwr
[  2] 0.00-1.00 sec  45.2 MBytes   379 Mbits/sec  474213/0          0    
  7611K/23840(768) us  1989
[ ID] Interval        Transfer    Bandwidth       Reads=Dist
[ *2] 0.00-1.00 sec  29.8 MBytes   250 Mbits/sec  
312674=78:10:29:14:57:17:87:14
[ ID] Interval       Transfer     Bandwidth
[FD2] 0.00-1.00 sec  75.0 MBytes   629 Mbits/sec
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=25.213/15.954/48.097/9.889 ms    0   14K/23702 us    40 rps
[ *2] 1.00-2.00 sec  45.6 MBytes   383 Mbits/sec  
478326=4:19:3:10:16:12:12:13
[  2] 1.00-2.00 sec  56.4 MBytes   473 Mbits/sec  591760/0          0    
  7611K/14112(1264) us  4193
[FD2] 1.00-2.00 sec   102 MBytes   856 Mbits/sec
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=25.220/11.400/51.321/14.320 ms    0   14K/25023 us    40 rps
[ *2] 2.00-3.00 sec  39.1 MBytes   328 Mbits/sec  
410533=18:32:13:15:25:27:7:23
[  2] 2.00-3.00 sec  57.2 MBytes   480 Mbits/sec  599942/0          0    
  7611K/14356(920) us  4179
[FD2] 2.00-3.00 sec  96.4 MBytes   808 Mbits/sec
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=16.023/5.588/50.304/12.940 ms    0   14K/16648 us    62 rps
[ *2] 3.00-4.00 sec  35.9 MBytes   301 Mbits/sec  
376791=17:34:11:15:33:37:22:26
[  2] 3.00-4.00 sec  56.6 MBytes   475 Mbits/sec  593663/0          0    
  7611K/45828(2395) us  1295
[FD2] 3.00-4.00 sec  92.5 MBytes   776 Mbits/sec
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=12.445/7.353/19.663/3.601 ms    0   14K/13359 us    80 rps
[ *2] 4.00-5.00 sec  42.4 MBytes   356 Mbits/sec  
444818=29:34:8:15:21:26:15:20
[  2] 4.00-5.00 sec  54.9 MBytes   461 Mbits/sec  575659/0          0    
  7611K/13205(2359) us  4359
[FD2] 4.00-5.00 sec  97.3 MBytes   816 Mbits/sec
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=12.092/6.493/16.031/3.279 ms    0   14K/12345 us    83 rps
[ *2] 5.00-6.00 sec  37.9 MBytes   318 Mbits/sec  
397808=10:22:20:28:24:20:11:10
[  2] 5.00-6.00 sec  56.9 MBytes   477 Mbits/sec  596477/0          0    
  7611K/15607(167) us  3822
[FD2] 5.00-6.00 sec  94.8 MBytes   795 Mbits/sec
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=13.916/9.555/19.109/2.876 ms    0   14K/13690 us    72 rps
[ *2] 6.00-7.00 sec  46.0 MBytes   386 Mbits/sec  
482572=25:46:13:18:34:35:27:32
[  2] 6.00-7.00 sec  57.2 MBytes   480 Mbits/sec  600055/0          0    
  7611K/14175(669) us  4233
[FD2] 6.00-7.00 sec   103 MBytes   866 Mbits/sec
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=13.125/4.997/19.527/4.239 ms    0   14K/13695 us    76 rps
[ *2] 7.00-8.00 sec  40.1 MBytes   336 Mbits/sec  
420304=21:30:11:6:5:17:14:18
[  2] 7.00-8.00 sec  56.0 MBytes   470 Mbits/sec  587082/0          0    
  7611K/24914(3187) us  2356
[FD2] 7.00-8.00 sec  96.1 MBytes   806 Mbits/sec
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=8.221/2.939/17.274/4.419 ms    0   14K/9861 us    122 rps
[ *2] 8.00-9.00 sec  33.6 MBytes   282 Mbits/sec  
352847=17:35:16:19:15:27:17:17
[  2] 8.00-9.00 sec  56.2 MBytes   471 Mbits/sec  589281/0          0    
  7611K/13206(403) us  4462
[FD2] 8.00-9.00 sec  89.8 MBytes   754 Mbits/sec
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=10.896/6.822/14.941/2.788 ms    0   14K/10419 us    92 rps
[ *2] 9.00-10.00 sec  36.4 MBytes   306 Mbits/sec  
382097=18:48:22:25:29:32:28:24
[  2] 9.00-10.00 sec  57.2 MBytes   480 Mbits/sec  599784/0          0   
   7611K/15965(850) us  3757
[FD2] 9.00-10.00 sec  93.6 MBytes   785 Mbits/sec
[  2] 0.00-10.00 sec   554 MBytes   465 Mbits/sec  5807917/0          0  
    7611K/15965(850) us  3637
[ *2] 0.00-10.07 sec   391 MBytes   326 Mbits/sec  
4099553=239:319:147:168:261:251:241:200
[FD2] 0.00-10.07 sec   945 MBytes   787 Mbits/sec
[  1] 0.00-10.07 sec  21.5 KBytes  17.5 Kbits/sec    
110=14.330/1.855/51.321/9.904 ms    0   14K/6896 us    70 rps
[  1] 0.00-10.07 sec BB8(f)-PDF: 
bin(w=100us):cnt(110)=19:1,30:1,33:2,34:1,40:1,45:2,47:1,50:1,51:1,55:1,56:1,58:1,64:1,65:1,66:1,67:1,69:1,70:2,71:1,74:1,77:1,79:2,83:2,84:1,90:1,93:2,96:3,97:2,99:1,100:1,103:2,105:1,111:1,112:1,113:2,114:2,115:1,116:1,118:2,119:1,122:2,123:1,124:1,127:1,129:2,130:3,133:1,143:1,144:1,145:1,147:3,149:4,150:2,156:1,157:2,158:1,160:1,161:1,162:1,163:1,164:1,167:1,173:1,181:1,182:2,186:1,192:1,196:1,197:1,199:1,218:1,221:1,235:1,242:1,246:1,278:1,281:1,282:1,286:1,342:1,481:1,496:1,501:1,504:1,514:1 
(5.00/95.00/99.7%=40/342/514,Outliers=0,obl/obu=0/0)
[ CT] final connect times (min/avg/max/stdev) = 4.104/4.104/4.104/0.000 
ms (tot/err) = 2/0>> -----Original Message-----
>> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
>> Sent: Monday, October 31, 2022 2:53 PM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: MORTON JR., AL <acmorton@att.com>; Rpm 
>> <rpm@lists.bufferbloat.net>;
>> ippm@ietf.org
>> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of 
>> "Working
>> Latency" metrics
>> 
>> Would it be possible to get some iperf 2 bounceback test results too?
>> 
>> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
>> gmYpYN3pBO-aWMfjDRdVRFQ20aHQ5nDHOhEVY1y-
>> MkFFyH8YmM4wf8cEtaxzvcwTMaCaJOCNRBtj0tnz9A$
>> 
>> Also, for the hunt algo, maybe use TCP first to get a starting point 
>> and
>> then hunt? Just a thought.
>> 
>> Thanks,
>> Bob
>> >
> <snip>
> 
> Thanks for your suggestion, Bob, and it's nice to meet you!
> I was only familiar with the "old" iperf2 at:
> https://iperf.fr/iperf-doc.php
> before your message and URL arrived (yet another segment of the 
> elephant!).
> 
> I didn't quickly find whether your code will run on linux (ubuntu for
> me). I suppose I use the .tar.gz and compile locally. Let me know,
> I've got some other tools to check-out first.
> 
> You suggest the bounceback test option (I found it on the man page),
> but there are lots of options and six versions of bounceback!  Based
> on the results I already reported, can you suggest a specific version
> of bounceback and a set of options that would be a good first try?
> (see the man page at https://iperf2.sourceforge.io/iperf-manpage.html )
> 
> Regarding our hunt algorithms (search for max), the new Type C algo
> (in release 7.5.0) locates the Max very quickly. I showed a comparison
> of our Type B and Type C search algorithms on a slide at the IPPM
> meeting in July. See Slide 10:
> https://datatracker.ietf.org/meeting/114/materials/slides-114-ippm-sec-dir-review-discussion-test-protocol-for-one-way-ip-capacity-measurement-00
> 
> The measured capacity reaches 1 Gbps in about 1 second with the Type C
> algorithm, without choosing or testing for a starting point (our
> default starting point rate is ~500kbps, very low to accommodate "any"
> subscribed rate).
> 
> regards, and thanks again,
> Al

[Rpm] lightweight active sensing of bandwidth and buffering

[Dave Taht]

For about 2 years now the cake w-adaptive bandwidth project has been
exploring techniques to lightweightedly sense  bandwidth and buffering
problems. One of my favorites was their discovery that ICMP type 13
got them working OWD from millions of ipv4 devices!

They've also explored leveraging ntp and multiple other methods, and
have scripts available that do a good job of compensating for 5g and
starlink's misbehaviors.

They've also pioneered a whole bunch of new graphing techniques, which
I do wish were used more than single number summaries especially in
analyzing the behaviors of new metrics like rpm, samknows, ookla, and
RFC9097 - to see what is being missed.

There are thousands of posts about this research topic, a new post on
OWD just went by here.

https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793

and of course, I love flent's enormous graphing toolset for simulating
and analyzing complex network behaviors.

Re: [Rpm] lightweight active sensing of bandwidth and buffering

[rjmcmahon]

Bufferbloat shifts the minimum of the latency or OWD CDF. A suggestion 
is to disable x-axis auto-scaling and start from zero.

Bob
> For about 2 years now the cake w-adaptive bandwidth project has been
> exploring techniques to lightweightedly sense  bandwidth and buffering
> problems. One of my favorites was their discovery that ICMP type 13
> got them working OWD from millions of ipv4 devices!
> 
> They've also explored leveraging ntp and multiple other methods, and
> have scripts available that do a good job of compensating for 5g and
> starlink's misbehaviors.
> 
> They've also pioneered a whole bunch of new graphing techniques, which
> I do wish were used more than single number summaries especially in
> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
> RFC9097 - to see what is being missed.
> 
> There are thousands of posts about this research topic, a new post on
> OWD just went by here.
> 
> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
> 
> and of course, I love flent's enormous graphing toolset for simulating
> and analyzing complex network behaviors.

Re: [Rpm] lightweight active sensing of bandwidth and buffering

[Sebastian Moeller]

Hi Bob,

thanks!


> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Bufferbloat shifts the minimum of the latency or OWD CDF.

	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).


> A suggestion is to disable x-axis auto-scaling and start from zero.

	[SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this


*) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.

**) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....


> 
> Bob
>> For about 2 years now the cake w-adaptive bandwidth project has been
>> exploring techniques to lightweightedly sense  bandwidth and buffering
>> problems. One of my favorites was their discovery that ICMP type 13
>> got them working OWD from millions of ipv4 devices!
>> They've also explored leveraging ntp and multiple other methods, and
>> have scripts available that do a good job of compensating for 5g and
>> starlink's misbehaviors.
>> They've also pioneered a whole bunch of new graphing techniques, which
>> I do wish were used more than single number summaries especially in
>> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
>> RFC9097 - to see what is being missed.
>> There are thousands of posts about this research topic, a new post on
>> OWD just went by here.
>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> and of course, I love flent's enormous graphing toolset for simulating
>> and analyzing complex network behaviors.
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Ruediger.Geib]

Bob, Sebastian,

not being active on your topic, just to add what I observed on congestion:
- starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
- buffer fill reaches a "steady state", called bufferbloat on access I think; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct). I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
- a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill. The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
- in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.

Regards,

Ruediger


> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Bufferbloat shifts the minimum of the latency or OWD CDF.

	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).


> A suggestion is to disable x-axis auto-scaling and start from zero.

	[SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this


*) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.

**) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....


> 
> Bob
>> For about 2 years now the cake w-adaptive bandwidth project has been 
>> exploring techniques to lightweightedly sense  bandwidth and 
>> buffering problems. One of my favorites was their discovery that ICMP 
>> type 13 got them working OWD from millions of ipv4 devices!
>> They've also explored leveraging ntp and multiple other methods, and 
>> have scripts available that do a good job of compensating for 5g and 
>> starlink's misbehaviors.
>> They've also pioneered a whole bunch of new graphing techniques, 
>> which I do wish were used more than single number summaries 
>> especially in analyzing the behaviors of new metrics like rpm, 
>> samknows, ookla, and
>> RFC9097 - to see what is being missed.
>> There are thousands of posts about this research topic, a new post on 
>> OWD just went by here.
>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> and of course, I love flent's enormous graphing toolset for 
>> simulating and analyzing complex network behaviors.
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[rjmcmahon]

Most measuring bloat are ignoring queue build up phase and rather start 
taking measurements after the bottleneck queue is in a standing state.

My opinion, the best units for bloat is packets for UDP or bytes for 
TCP. Min delay is a proxy measurement.

Little's law allows one to compute this though does assume the network 
is in a stable state over the measurement interval. In the real world, 
this probably is rarely true. So we, in test & measurement engineering, 
force the standing state with some sort of measurement co-traffic and 
call it "working conditions" or equivalent. ;)

Bob
> Bob, Sebastian,
> 
> not being active on your topic, just to add what I observed on 
> congestion:
> - starts with an increase of jitter, but measured minimum delays still
> remain constant. Technically, a queue builds up some of the time, but
> it isn't present permanently.
> - buffer fill reaches a "steady state", called bufferbloat on access I
> think; technically, OWD increases also for the minimum delays, jitter
> now decreases (what you've described that as "the delay magnitude"
> decreases or "minimum CDF shift" respectively, if I'm correct). I'd
> expect packet loss to occur, once the buffer fill is on steady state,
> but loss might be randomly distributed and could be of a low
> percentage.
> - a sudden rather long load burst may cause a  jump-start to
> "steady-state" buffer fill. The above holds for a slow but steady load
> increase (where the measurement frequency determines the timescale
> qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't
> an easy to handle single metric to identify congestion.
> 
> Regards,
> 
> Ruediger
> 
> 
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm 
>> <rpm@lists.bufferbloat.net> wrote:
>> 
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
> 
> 	[SM] Thank you for spelling this out explicitly, I only worked on a
> vage implicit assumption along those lines. However what I want to
> avoid is using delay magnitude itself as classifier between high and
> low load condition as that seems statistically uncouth to then show
> that the delay differs between the two classes;).
> 	Yet, your comment convinced me that my current load threshold (at
> least for the high load condition) probably is too small, exactly
> because the "base" of the high-load CDFs coincides with the base of
> the low-load CDFs implying that the high-load class contains too many
> samples with decent delay (which after all is one of the goals of the
> whole autorate endeavor).
> 
> 
>> A suggestion is to disable x-axis auto-scaling and start from zero.
> 
> 	[SM] Will reconsider. I started with start at zero, end then switched
> to an x-range that starts with the delay corresponding to 0.01% for
> the reflector/condition with the lowest such value and stops at 97.5%
> for the reflector/condition with the highest delay value. My rationale
> is that the base delay/path delay of each reflector is not all that
> informative* (and it can still be learned from reading the x-axis),
> the long tail > 50% however is where I expect most differences so I
> want to emphasize this and finally I wanted to avoid that the actual
> "curvy" part gets compressed so much that all lines more or less
> coincide. As I said, I will reconsider this
> 
> 
> *) We also maintain individual baselines per reflector, so I could
> just plot the differences from baseline, but that would essentially
> equalize all reflectors, and I think having a plot that easily shows
> reflectors with outlying base delay can be informative when selecting
> reflector candidates. However once we actually switch to OWDs baseline
> correction might be required anyways, as due to colck differences ICMP
> type 13/14 data can have massive offsets that are mostly indicative of
> un synched clocks**.
> 
> **) This is whyI would prefer to use NTP servers as reflectors with
> NTP requests, my expectation is all of these should be reasonably
> synced by default so that offsets should be in the sane range....
> 
> 
>> 
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense  bandwidth and
>>> buffering problems. One of my favorites was their discovery that ICMP
>>> type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques,
>>> which I do wish were used more than single number summaries
>>> especially in analyzing the behaviors of new metrics like rpm,
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Dave Taht]

On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
<rpm@lists.bufferbloat.net> wrote:
>
> Most measuring bloat are ignoring queue build up phase and rather start
> taking measurements after the bottleneck queue is in a standing state.

+10. It's the slow start transient that is holding things back. If we
could, for example
open up the 110+ objects and flows web pages require all at once, and
let 'em rip, instead of 15 at a time, without destroying the network,
web PLT would get much better.

> My opinion, the best units for bloat is packets for UDP or bytes for
> TCP. Min delay is a proxy measurement.

bytes, period. bytes = time. Sure most udp today is small packets but
quic and videconferencing change that.

>
> Little's law allows one to compute this though does assume the network
> is in a stable state over the measurement interval. In the real world,
> this probably is rarely true. So we, in test & measurement engineering,
> force the standing state with some sort of measurement co-traffic and
> call it "working conditions" or equivalent. ;)

There was an extremely long, nuanced debate about little's law and
where it applies, last year, here:

https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html

I don't want to go into it, again.

>
> Bob
> > Bob, Sebastian,
> >
> > not being active on your topic, just to add what I observed on
> > congestion:
> > - starts with an increase of jitter, but measured minimum delays still
> > remain constant. Technically, a queue builds up some of the time, but
> > it isn't present permanently.
> > - buffer fill reaches a "steady state", called bufferbloat on access I
> > think; technically, OWD increases also for the minimum delays, jitter
> > now decreases (what you've described that as "the delay magnitude"
> > decreases or "minimum CDF shift" respectively, if I'm correct). I'd
> > expect packet loss to occur, once the buffer fill is on steady state,
> > but loss might be randomly distributed and could be of a low
> > percentage.
> > - a sudden rather long load burst may cause a  jump-start to
> > "steady-state" buffer fill. The above holds for a slow but steady load
> > increase (where the measurement frequency determines the timescale
> > qualifying "slow").
> > - in the end, max-min delay or delay distribution/jitter likely isn't
> > an easy to handle single metric to identify congestion.
> >
> > Regards,
> >
> > Ruediger
> >
> >
> >> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
> >> <rpm@lists.bufferbloat.net> wrote:
> >>
> >> Bufferbloat shifts the minimum of the latency or OWD CDF.
> >
> >       [SM] Thank you for spelling this out explicitly, I only worked on a
> > vage implicit assumption along those lines. However what I want to
> > avoid is using delay magnitude itself as classifier between high and
> > low load condition as that seems statistically uncouth to then show
> > that the delay differs between the two classes;).
> >       Yet, your comment convinced me that my current load threshold (at
> > least for the high load condition) probably is too small, exactly
> > because the "base" of the high-load CDFs coincides with the base of
> > the low-load CDFs implying that the high-load class contains too many
> > samples with decent delay (which after all is one of the goals of the
> > whole autorate endeavor).
> >
> >
> >> A suggestion is to disable x-axis auto-scaling and start from zero.
> >
> >       [SM] Will reconsider. I started with start at zero, end then switched
> > to an x-range that starts with the delay corresponding to 0.01% for
> > the reflector/condition with the lowest such value and stops at 97.5%
> > for the reflector/condition with the highest delay value. My rationale
> > is that the base delay/path delay of each reflector is not all that
> > informative* (and it can still be learned from reading the x-axis),
> > the long tail > 50% however is where I expect most differences so I
> > want to emphasize this and finally I wanted to avoid that the actual
> > "curvy" part gets compressed so much that all lines more or less
> > coincide. As I said, I will reconsider this
> >
> >
> > *) We also maintain individual baselines per reflector, so I could
> > just plot the differences from baseline, but that would essentially
> > equalize all reflectors, and I think having a plot that easily shows
> > reflectors with outlying base delay can be informative when selecting
> > reflector candidates. However once we actually switch to OWDs baseline
> > correction might be required anyways, as due to colck differences ICMP
> > type 13/14 data can have massive offsets that are mostly indicative of
> > un synched clocks**.
> >
> > **) This is whyI would prefer to use NTP servers as reflectors with
> > NTP requests, my expectation is all of these should be reasonably
> > synced by default so that offsets should be in the sane range....
> >
> >
> >>
> >> Bob
> >>> For about 2 years now the cake w-adaptive bandwidth project has been
> >>> exploring techniques to lightweightedly sense  bandwidth and
> >>> buffering problems. One of my favorites was their discovery that ICMP
> >>> type 13 got them working OWD from millions of ipv4 devices!
> >>> They've also explored leveraging ntp and multiple other methods, and
> >>> have scripts available that do a good job of compensating for 5g and
> >>> starlink's misbehaviors.
> >>> They've also pioneered a whole bunch of new graphing techniques,
> >>> which I do wish were used more than single number summaries
> >>> especially in analyzing the behaviors of new metrics like rpm,
> >>> samknows, ookla, and
> >>> RFC9097 - to see what is being missed.
> >>> There are thousands of posts about this research topic, a new post on
> >>> OWD just went by here.
> >>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
> >>> and of course, I love flent's enormous graphing toolset for
> >>> simulating and analyzing complex network behaviors.
> >> _______________________________________________
> >> Rpm mailing list
> >> Rpm@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> > https://www.ietf.org/mailman/listinfo/ippm
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[rjmcmahon]

I used iperf 2's Little's law calculation to find the buffer sizes 
designed in by our hardware team(s). They were surprised that the 
numbers exactly matched their designs - Little applied - and I never saw 
either the hardware nor its design spec.

It seems reasonable to use something if & when it works and is useful. 
The challenge seems to be knowing the limits of any claims (or 
simulations.) I think engineers do this much when we assume linearity 
over some small interval as an example in finite element analysis and 
structures: 
https://control.com/technical-articles/the-difference-between-linear-and-nonlinear-finite-element-analysis-fea/

Bob
> On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
> <rpm@lists.bufferbloat.net> wrote:
>> 
>> Most measuring bloat are ignoring queue build up phase and rather 
>> start
>> taking measurements after the bottleneck queue is in a standing state.
> 
> +10. It's the slow start transient that is holding things back. If we
> could, for example
> open up the 110+ objects and flows web pages require all at once, and
> let 'em rip, instead of 15 at a time, without destroying the network,
> web PLT would get much better.
> 
>> My opinion, the best units for bloat is packets for UDP or bytes for
>> TCP. Min delay is a proxy measurement.
> 
> bytes, period. bytes = time. Sure most udp today is small packets but
> quic and videconferencing change that.
> 
>> 
>> Little's law allows one to compute this though does assume the network
>> is in a stable state over the measurement interval. In the real world,
>> this probably is rarely true. So we, in test & measurement 
>> engineering,
>> force the standing state with some sort of measurement co-traffic and
>> call it "working conditions" or equivalent. ;)
> 
> There was an extremely long, nuanced debate about little's law and
> where it applies, last year, here:
> 
> https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html
> 
> I don't want to go into it, again.
> 
>> 
>> Bob
>> > Bob, Sebastian,
>> >
>> > not being active on your topic, just to add what I observed on
>> > congestion:
>> > - starts with an increase of jitter, but measured minimum delays still
>> > remain constant. Technically, a queue builds up some of the time, but
>> > it isn't present permanently.
>> > - buffer fill reaches a "steady state", called bufferbloat on access I
>> > think; technically, OWD increases also for the minimum delays, jitter
>> > now decreases (what you've described that as "the delay magnitude"
>> > decreases or "minimum CDF shift" respectively, if I'm correct). I'd
>> > expect packet loss to occur, once the buffer fill is on steady state,
>> > but loss might be randomly distributed and could be of a low
>> > percentage.
>> > - a sudden rather long load burst may cause a  jump-start to
>> > "steady-state" buffer fill. The above holds for a slow but steady load
>> > increase (where the measurement frequency determines the timescale
>> > qualifying "slow").
>> > - in the end, max-min delay or delay distribution/jitter likely isn't
>> > an easy to handle single metric to identify congestion.
>> >
>> > Regards,
>> >
>> > Ruediger
>> >
>> >
>> >> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>> >> <rpm@lists.bufferbloat.net> wrote:
>> >>
>> >> Bufferbloat shifts the minimum of the latency or OWD CDF.
>> >
>> >       [SM] Thank you for spelling this out explicitly, I only worked on a
>> > vage implicit assumption along those lines. However what I want to
>> > avoid is using delay magnitude itself as classifier between high and
>> > low load condition as that seems statistically uncouth to then show
>> > that the delay differs between the two classes;).
>> >       Yet, your comment convinced me that my current load threshold (at
>> > least for the high load condition) probably is too small, exactly
>> > because the "base" of the high-load CDFs coincides with the base of
>> > the low-load CDFs implying that the high-load class contains too many
>> > samples with decent delay (which after all is one of the goals of the
>> > whole autorate endeavor).
>> >
>> >
>> >> A suggestion is to disable x-axis auto-scaling and start from zero.
>> >
>> >       [SM] Will reconsider. I started with start at zero, end then switched
>> > to an x-range that starts with the delay corresponding to 0.01% for
>> > the reflector/condition with the lowest such value and stops at 97.5%
>> > for the reflector/condition with the highest delay value. My rationale
>> > is that the base delay/path delay of each reflector is not all that
>> > informative* (and it can still be learned from reading the x-axis),
>> > the long tail > 50% however is where I expect most differences so I
>> > want to emphasize this and finally I wanted to avoid that the actual
>> > "curvy" part gets compressed so much that all lines more or less
>> > coincide. As I said, I will reconsider this
>> >
>> >
>> > *) We also maintain individual baselines per reflector, so I could
>> > just plot the differences from baseline, but that would essentially
>> > equalize all reflectors, and I think having a plot that easily shows
>> > reflectors with outlying base delay can be informative when selecting
>> > reflector candidates. However once we actually switch to OWDs baseline
>> > correction might be required anyways, as due to colck differences ICMP
>> > type 13/14 data can have massive offsets that are mostly indicative of
>> > un synched clocks**.
>> >
>> > **) This is whyI would prefer to use NTP servers as reflectors with
>> > NTP requests, my expectation is all of these should be reasonably
>> > synced by default so that offsets should be in the sane range....
>> >
>> >
>> >>
>> >> Bob
>> >>> For about 2 years now the cake w-adaptive bandwidth project has been
>> >>> exploring techniques to lightweightedly sense  bandwidth and
>> >>> buffering problems. One of my favorites was their discovery that ICMP
>> >>> type 13 got them working OWD from millions of ipv4 devices!
>> >>> They've also explored leveraging ntp and multiple other methods, and
>> >>> have scripts available that do a good job of compensating for 5g and
>> >>> starlink's misbehaviors.
>> >>> They've also pioneered a whole bunch of new graphing techniques,
>> >>> which I do wish were used more than single number summaries
>> >>> especially in analyzing the behaviors of new metrics like rpm,
>> >>> samknows, ookla, and
>> >>> RFC9097 - to see what is being missed.
>> >>> There are thousands of posts about this research topic, a new post on
>> >>> OWD just went by here.
>> >>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> >>> and of course, I love flent's enormous graphing toolset for
>> >>> simulating and analyzing complex network behaviors.
>> >> _______________________________________________
>> >> Rpm mailing list
>> >> Rpm@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/rpm
>> >
>> > _______________________________________________
>> > ippm mailing list
>> > ippm@ietf.org
>> > https://www.ietf.org/mailman/listinfo/ippm
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Sebastian Moeller]

> On Nov 2, 2022, at 20:44, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
> <rpm@lists.bufferbloat.net> wrote:
>> 
>> Most measuring bloat are ignoring queue build up phase and rather start
>> taking measurements after the bottleneck queue is in a standing state.
> 
> +10. It's the slow start transient that is holding things back.

	[SM] From my naive perspective slow start has a few facets:
a) it is absolutely the right approach conceptually with no reliable prior knowledge of the capacity the best we can do is to probe it by increasing the sending rate over time (and the current exponential growth is already plenty aggressive*)
b) Since this needs feedback from the remote endpoint at best we can figure out 1 RTT later whether we sent at acceptable rate
c) we want to go as fast as reasonable
d) but not any faster ;)
e) the trickiest part is really deciding when to leave slow start's aggressive rate increase per RTT regime so that the >= 1 RTT "blind" spot does not lead to too big a transient queue spike.
f) The slow start phase ca be relatively short, so the less averaging we need to do to decide when to drop out of slow start the better as averaging costs time.

IMHO the best way forward would be to switch from bit-banging the queue state (as in L4S' design) over multiple packets to sending a multi-bit queue occupancy signal per-packet, then the sender should be able to figure out the rate of queueing change as a function of sending rate change and predict a reasonable time to switch to congestion avoidance without having to wait for a drop (assuming the remote end signals these queue occupancy signals back to the sender in a timely fashion)...
(This is really just transfering the ideas from Arslan, Serhat, and Nick McKeown. ‘Switches Know the Exact Amount of Congestion’. In Proceedings of the 2019 Workshop on Buffer Sizing, 1–6, 2019. to slow-start).


*) Two factors to play with is size of the starting batch (aka initial window) and the actual factor of increase per RTT, people statred playing with the first but so far seem reasonable enough not to touch the second ;)


> If we
> could, for example
> open up the 110+ objects and flows web pages require all at once, and
> let 'em rip, instead of 15 at a time, without destroying the network,
> web PLT would get much better.
> 
>> My opinion, the best units for bloat is packets for UDP or bytes for
>> TCP. Min delay is a proxy measurement.
> 
> bytes, period. bytes = time. Sure most udp today is small packets but
> quic and videconferencing change that.
> 
>> 
>> Little's law allows one to compute this though does assume the network
>> is in a stable state over the measurement interval. In the real world,
>> this probably is rarely true. So we, in test & measurement engineering,
>> force the standing state with some sort of measurement co-traffic and
>> call it "working conditions" or equivalent. ;)
> 
> There was an extremely long, nuanced debate about little's law and
> where it applies, last year, here:
> 
> https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html
> 
> I don't want to go into it, again.
> 
>> 
>> Bob
>>> Bob, Sebastian,
>>> 
>>> not being active on your topic, just to add what I observed on
>>> congestion:
>>> - starts with an increase of jitter, but measured minimum delays still
>>> remain constant. Technically, a queue builds up some of the time, but
>>> it isn't present permanently.
>>> - buffer fill reaches a "steady state", called bufferbloat on access I
>>> think; technically, OWD increases also for the minimum delays, jitter
>>> now decreases (what you've described that as "the delay magnitude"
>>> decreases or "minimum CDF shift" respectively, if I'm correct). I'd
>>> expect packet loss to occur, once the buffer fill is on steady state,
>>> but loss might be randomly distributed and could be of a low
>>> percentage.
>>> - a sudden rather long load burst may cause a  jump-start to
>>> "steady-state" buffer fill. The above holds for a slow but steady load
>>> increase (where the measurement frequency determines the timescale
>>> qualifying "slow").
>>> - in the end, max-min delay or delay distribution/jitter likely isn't
>>> an easy to handle single metric to identify congestion.
>>> 
>>> Regards,
>>> 
>>> Ruediger
>>> 
>>> 
>>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>>>> <rpm@lists.bufferbloat.net> wrote:
>>>> 
>>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>> 
>>>      [SM] Thank you for spelling this out explicitly, I only worked on a
>>> vage implicit assumption along those lines. However what I want to
>>> avoid is using delay magnitude itself as classifier between high and
>>> low load condition as that seems statistically uncouth to then show
>>> that the delay differs between the two classes;).
>>>      Yet, your comment convinced me that my current load threshold (at
>>> least for the high load condition) probably is too small, exactly
>>> because the "base" of the high-load CDFs coincides with the base of
>>> the low-load CDFs implying that the high-load class contains too many
>>> samples with decent delay (which after all is one of the goals of the
>>> whole autorate endeavor).
>>> 
>>> 
>>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>> 
>>>      [SM] Will reconsider. I started with start at zero, end then switched
>>> to an x-range that starts with the delay corresponding to 0.01% for
>>> the reflector/condition with the lowest such value and stops at 97.5%
>>> for the reflector/condition with the highest delay value. My rationale
>>> is that the base delay/path delay of each reflector is not all that
>>> informative* (and it can still be learned from reading the x-axis),
>>> the long tail > 50% however is where I expect most differences so I
>>> want to emphasize this and finally I wanted to avoid that the actual
>>> "curvy" part gets compressed so much that all lines more or less
>>> coincide. As I said, I will reconsider this
>>> 
>>> 
>>> *) We also maintain individual baselines per reflector, so I could
>>> just plot the differences from baseline, but that would essentially
>>> equalize all reflectors, and I think having a plot that easily shows
>>> reflectors with outlying base delay can be informative when selecting
>>> reflector candidates. However once we actually switch to OWDs baseline
>>> correction might be required anyways, as due to colck differences ICMP
>>> type 13/14 data can have massive offsets that are mostly indicative of
>>> un synched clocks**.
>>> 
>>> **) This is whyI would prefer to use NTP servers as reflectors with
>>> NTP requests, my expectation is all of these should be reasonably
>>> synced by default so that offsets should be in the sane range....
>>> 
>>> 
>>>> 
>>>> Bob
>>>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>>>> exploring techniques to lightweightedly sense  bandwidth and
>>>>> buffering problems. One of my favorites was their discovery that ICMP
>>>>> type 13 got them working OWD from millions of ipv4 devices!
>>>>> They've also explored leveraging ntp and multiple other methods, and
>>>>> have scripts available that do a good job of compensating for 5g and
>>>>> starlink's misbehaviors.
>>>>> They've also pioneered a whole bunch of new graphing techniques,
>>>>> which I do wish were used more than single number summaries
>>>>> especially in analyzing the behaviors of new metrics like rpm,
>>>>> samknows, ookla, and
>>>>> RFC9097 - to see what is being missed.
>>>>> There are thousands of posts about this research topic, a new post on
>>>>> OWD just went by here.
>>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>>> and of course, I love flent's enormous graphing toolset for
>>>>> simulating and analyzing complex network behaviors.
>>>> _______________________________________________
>>>> Rpm mailing list
>>>> Rpm@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/rpm
>>> 
>>> _______________________________________________
>>> ippm mailing list
>>> ippm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/ippm
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
> 
> 
> 
> -- 
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Sebastian Moeller]

Dear Ruediger,

thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow. 
The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).

*) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
**) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).


> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
> 
> Bob, Sebastian,
> 
> not being active on your topic, just to add what I observed on congestion:

	[SM] I will try to explain how/if we could exploit your observations for our controller

> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.

	[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.

> - buffer fill reaches a "steady state", called bufferbloat on access I think

	[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?

> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).

	[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).

> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.

	[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.

> - a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill.

	[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.


> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.

	[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).

Many Thanks & Kind Regards
	Sebastian


> 
> Regards,
> 
> Ruediger
> 
> 
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>> 
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
> 
> 	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
> 	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
> 
> 
>> A suggestion is to disable x-axis auto-scaling and start from zero.
> 
> 	[SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this
> 
> 
> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
> 
> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
> 
> 
>> 
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been 
>>> exploring techniques to lightweightedly sense  bandwidth and 
>>> buffering problems. One of my favorites was their discovery that ICMP 
>>> type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and 
>>> have scripts available that do a good job of compensating for 5g and 
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques, 
>>> which I do wish were used more than single number summaries 
>>> especially in analyzing the behaviors of new metrics like rpm, 
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on 
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for 
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Ruediger.Geib]

Hi Sebastian,

[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.

RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.

Regards,

Ruediger



-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <moeller0@gmx.de> 
Gesendet: Mittwoch, 2. November 2022 22:41
An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm <rpm@lists.bufferbloat.net>; ippm@ietf.org
Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and buffering

Dear Ruediger,

thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow. 
The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).

*) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
**) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).


> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
> 
> Bob, Sebastian,
> 
> not being active on your topic, just to add what I observed on congestion:

	[SM] I will try to explain how/if we could exploit your observations for our controller

> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.

	[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.

> - buffer fill reaches a "steady state", called bufferbloat on access I 
> think

	[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?

> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).

	[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).

> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.

	[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.

> - a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill.

	[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.


> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.

	[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).

Many Thanks & Kind Regards
	Sebastian


> 
> Regards,
> 
> Ruediger
> 
> 
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>> 
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
> 
> 	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
> 	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
> 
> 
>> A suggestion is to disable x-axis auto-scaling and start from zero.
> 
> 	[SM] Will reconsider. I started with start at zero, end then switched 
> to an x-range that starts with the delay corresponding to 0.01% for 
> the reflector/condition with the lowest such value and stops at 97.5% 
> for the reflector/condition with the highest delay value. My rationale 
> is that the base delay/path delay of each reflector is not all that 
> informative* (and it can still be learned from reading the x-axis), 
> the long tail > 50% however is where I expect most differences so I 
> want to emphasize this and finally I wanted to avoid that the actual 
> "curvy" part gets compressed so much that all lines more or less 
> coincide. As I said, I will reconsider this
> 
> 
> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
> 
> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
> 
> 
>> 
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been 
>>> exploring techniques to lightweightedly sense  bandwidth and 
>>> buffering problems. One of my favorites was their discovery that 
>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and 
>>> have scripts available that do a good job of compensating for 5g and 
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques, 
>>> which I do wish were used more than single number summaries 
>>> especially in analyzing the behaviors of new metrics like rpm, 
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post 
>>> on OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for 
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Sebastian Moeller]

Hi Ruediger,

> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
> 
> Hi Sebastian,
> 
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
> 
> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.

	[SM] Exactly! We do try to account for some of these factors to some degree:
a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
	A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
	B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).

And yes, the goal is not to model congestion veridically, but really to offer something that is "good enough" to help while not being excessively complicated (though admittedly as often happens with heuristics we added more and more code once testing showed our previous approach as too simplistic). Also this effort over on the OpenWrt list is actually a set of multiple parallel efforts  with cake-autorate* only one ~4 alternatives (implemented in perl, lua, awk, and bash) that cross-pollinated each other and IMHO all approaches improved from open discussion and friendly competition.


*) And cake-autorate is neither the first nor the most innovative of the bunch (that would IMHO the perl implementation that had existed first and had single-handedly demonstrated the suitability of ICMP type 13/14 timestamps for directional "congestion" measurements) but the one with the most dedicated and open-minded main developer** (CCd) integrating all ideas (and coming up with new ones), without whom neither of the approaches would have become public or even happened I think. It is also the approach that appears to have the most active development momentum and testing community.

**) A developer that also uses this code on a day-to-day basis, so "dog-fooding" at its finest, that really helps I think to stay pragmatic ;)


> 
> Regards,
> 
> Ruediger
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Sebastian Moeller <moeller0@gmx.de> 
> Gesendet: Mittwoch, 2. November 2022 22:41
> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm <rpm@lists.bufferbloat.net>; ippm@ietf.org
> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and buffering
> 
> Dear Ruediger,
> 
> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow. 
> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
> 
> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
> 
> 
>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>> 
>> Bob, Sebastian,
>> 
>> not being active on your topic, just to add what I observed on congestion:
> 
> 	[SM] I will try to explain how/if we could exploit your observations for our controller
> 
>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
> 
> 	[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
> 
>> - buffer fill reaches a "steady state", called bufferbloat on access I 
>> think
> 
> 	[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
> 
>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
> 
> 	[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
> 
>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
> 
> 	[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
> 
>> - a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill.
> 
> 	[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
> 
> 
>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
> 
> 	[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
> 
> Many Thanks & Kind Regards
> 	Sebastian
> 
> 
>> 
>> Regards,
>> 
>> Ruediger
>> 
>> 
>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>> 
>> 	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
>> 	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>> 
>> 
>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>> 
>> 	[SM] Will reconsider. I started with start at zero, end then switched 
>> to an x-range that starts with the delay corresponding to 0.01% for 
>> the reflector/condition with the lowest such value and stops at 97.5% 
>> for the reflector/condition with the highest delay value. My rationale 
>> is that the base delay/path delay of each reflector is not all that 
>> informative* (and it can still be learned from reading the x-axis), 
>> the long tail > 50% however is where I expect most differences so I 
>> want to emphasize this and finally I wanted to avoid that the actual 
>> "curvy" part gets compressed so much that all lines more or less 
>> coincide. As I said, I will reconsider this
>> 
>> 
>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>> 
>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>> 
>> 
>>> 
>>> Bob
>>>> For about 2 years now the cake w-adaptive bandwidth project has been 
>>>> exploring techniques to lightweightedly sense  bandwidth and 
>>>> buffering problems. One of my favorites was their discovery that 
>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>> They've also explored leveraging ntp and multiple other methods, and 
>>>> have scripts available that do a good job of compensating for 5g and 
>>>> starlink's misbehaviors.
>>>> They've also pioneered a whole bunch of new graphing techniques, 
>>>> which I do wish were used more than single number summaries 
>>>> especially in analyzing the behaviors of new metrics like rpm, 
>>>> samknows, ookla, and
>>>> RFC9097 - to see what is being missed.
>>>> There are thousands of posts about this research topic, a new post 
>>>> on OWD just went by here.
>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>> and of course, I love flent's enormous graphing toolset for 
>>>> simulating and analyzing complex network behaviors.
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>> 
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
> 

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Ruediger.Geib]

Hi Sebastian,

Quick feedback [RG2]

-----------------------------------------
Hi Ruediger,

> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
> 
> Hi Sebastian,
> 
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
> 
> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.

	[SM] Exactly! We do try to account for some of these factors to some degree:
a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
[RG2] Allright, that approach is reasonable.

b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
	A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
	B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).

[RG2] That's a tricky and absolutely required part, differentiating a path change from congestion. A comment on B) - please keep in mind, that the baseline path may have a smaller delay, but immediately face congestion. That may not be an issue, if the congestion (buffer-fill) to be detected reliably adds latency in a range significantly above the monitored physical delay.   

<snip> 

> Regards,
> 
> Ruediger
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Sebastian Moeller <moeller0@gmx.de>
> Gesendet: Mittwoch, 2. November 2022 22:41
> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm 
> <rpm@lists.bufferbloat.net>; ippm@ietf.org
> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and 
> buffering
> 
> Dear Ruediger,
> 
> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow. 
> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
> 
> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
> 
> 
>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>> 
>> Bob, Sebastian,
>> 
>> not being active on your topic, just to add what I observed on congestion:
> 
> 	[SM] I will try to explain how/if we could exploit your observations 
> for our controller
> 
>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
> 
> 	[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
> 
>> - buffer fill reaches a "steady state", called bufferbloat on access 
>> I think
> 
> 	[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
> 
>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
> 
> 	[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
> 
>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
> 
> 	[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
> 
>> - a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill.
> 
> 	[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
> 
> 
>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
> 
> 	[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
> 
> Many Thanks & Kind Regards
> 	Sebastian
> 
> 
>> 
>> Regards,
>> 
>> Ruediger
>> 
>> 
>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>> 
>> 	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
>> 	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>> 
>> 
>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>> 
>> 	[SM] Will reconsider. I started with start at zero, end then 
>> switched to an x-range that starts with the delay corresponding to 
>> 0.01% for the reflector/condition with the lowest such value and 
>> stops at 97.5% for the reflector/condition with the highest delay 
>> value. My rationale is that the base delay/path delay of each 
>> reflector is not all that
>> informative* (and it can still be learned from reading the x-axis), 
>> the long tail > 50% however is where I expect most differences so I 
>> want to emphasize this and finally I wanted to avoid that the actual 
>> "curvy" part gets compressed so much that all lines more or less 
>> coincide. As I said, I will reconsider this
>> 
>> 
>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>> 
>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>> 
>> 
>>> 
>>> Bob
>>>> For about 2 years now the cake w-adaptive bandwidth project has 
>>>> been exploring techniques to lightweightedly sense  bandwidth and 
>>>> buffering problems. One of my favorites was their discovery that 
>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>> They've also explored leveraging ntp and multiple other methods, 
>>>> and have scripts available that do a good job of compensating for 
>>>> 5g and starlink's misbehaviors.
>>>> They've also pioneered a whole bunch of new graphing techniques, 
>>>> which I do wish were used more than single number summaries 
>>>> especially in analyzing the behaviors of new metrics like rpm, 
>>>> samknows, ookla, and
>>>> RFC9097 - to see what is being missed.
>>>> There are thousands of posts about this research topic, a new post 
>>>> on OWD just went by here.
>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>> and of course, I love flent's enormous graphing toolset for 
>>>> simulating and analyzing complex network behaviors.
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>> 
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
> 

Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering

[Sebastian Moeller]

Hi Ruediger,

many thanks for the interesting discussion.


> On Nov 3, 2022, at 12:25, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
> 
> Hi Sebastian,
> 
> Quick feedback [RG2]
> 
> -----------------------------------------
> Hi Ruediger,
> 
>> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>> 
>> Hi Sebastian,
>> 
>> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
>> 
>> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.
> 
> 	[SM] Exactly! We do try to account for some of these factors to some degree:
> a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
> [RG2] Allright, that approach is reasonable.
> 
> b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
> 	A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
> 	B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).
> 
> [RG2] That's a tricky and absolutely required part, differentiating a path change from congestion. A comment on B) - please keep in mind, that the baseline path may have a smaller delay, but immediately face congestion. That may not be an issue, if the congestion (buffer-fill) to be detected reliably adds latency in a range significantly above the monitored physical delay.

	[SM2] Yes! We have to deal with a set of known unknowns here and hence we need to employ a few heuristics. About the min-delay being inflated already by persistent congestion, yes we are blind to that no ifs not buts. Rule B) tries to keep such errors as short as possible in that we reduce our baseline estimate immediately after we have evidence that we over-estimated it. That does as you observed help little if congestion stays persistently high. Personally I am lucky enough that I rarelely encountered something like this on my access link, but that is just that luck.
	Our main goal is to control a known bottleneck (like an access link) and not the full internet path to a specific target location so most relevant traffic should be self generated so we have some rough estimate on the relative load of our link (but the available capacity share might still depend on other users in a shared segment so our load estimate is only approximate otherwise our job would be done already, if we know what fraction of our available capacity we currentky use and our current traffic rate we could directly calculate the shaper limit to keep responsiveness high ;) ).

Regards
	Sebastian


>   
> 
> <snip> 
> 
>> Regards,
>> 
>> Ruediger
>> 
>> 
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Sebastian Moeller <moeller0@gmx.de>
>> Gesendet: Mittwoch, 2. November 2022 22:41
>> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
>> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm 
>> <rpm@lists.bufferbloat.net>; ippm@ietf.org
>> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and 
>> buffering
>> 
>> Dear Ruediger,
>> 
>> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow. 
>> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
>> 
>> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
>> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
>> 
>> 
>>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>>> 
>>> Bob, Sebastian,
>>> 
>>> not being active on your topic, just to add what I observed on congestion:
>> 
>> 	[SM] I will try to explain how/if we could exploit your observations 
>> for our controller
>> 
>>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
>> 
>> 	[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
>> 
>>> - buffer fill reaches a "steady state", called bufferbloat on access 
>>> I think
>> 
>> 	[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
>> 
>>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
>> 
>> 	[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
>> 
>>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
>> 
>> 	[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
>> 
>>> - a sudden rather long load burst may cause a  jump-start to "steady-state" buffer fill.
>> 
>> 	[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
>> 
>> 
>>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
>> 
>> 	[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
>> 
>> Many Thanks & Kind Regards
>> 	Sebastian
>> 
>> 
>>> 
>>> Regards,
>>> 
>>> Ruediger
>>> 
>>> 
>>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>>> 
>>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>> 
>>> 	[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;). 
>>> 	Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>>> 
>>> 
>>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>> 
>>> 	[SM] Will reconsider. I started with start at zero, end then 
>>> switched to an x-range that starts with the delay corresponding to 
>>> 0.01% for the reflector/condition with the lowest such value and 
>>> stops at 97.5% for the reflector/condition with the highest delay 
>>> value. My rationale is that the base delay/path delay of each 
>>> reflector is not all that
>>> informative* (and it can still be learned from reading the x-axis), 
>>> the long tail > 50% however is where I expect most differences so I 
>>> want to emphasize this and finally I wanted to avoid that the actual 
>>> "curvy" part gets compressed so much that all lines more or less 
>>> coincide. As I said, I will reconsider this
>>> 
>>> 
>>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>>> 
>>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>>> 
>>> 
>>>> 
>>>> Bob
>>>>> For about 2 years now the cake w-adaptive bandwidth project has 
>>>>> been exploring techniques to lightweightedly sense  bandwidth and 
>>>>> buffering problems. One of my favorites was their discovery that 
>>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>>> They've also explored leveraging ntp and multiple other methods, 
>>>>> and have scripts available that do a good job of compensating for 
>>>>> 5g and starlink's misbehaviors.
>>>>> They've also pioneered a whole bunch of new graphing techniques, 
>>>>> which I do wish were used more than single number summaries 
>>>>> especially in analyzing the behaviors of new metrics like rpm, 
>>>>> samknows, ookla, and
>>>>> RFC9097 - to see what is being missed.
>>>>> There are thousands of posts about this research topic, a new post 
>>>>> on OWD just went by here.
>>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>>> and of course, I love flent's enormous graphing toolset for 
>>>>> simulating and analyzing complex network behaviors.
>>>> _______________________________________________
>>>> Rpm mailing list
>>>> Rpm@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/rpm
>>> 
>>> _______________________________________________
>>> ippm mailing list
>>> ippm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/ippm

Re: [Rpm] lightweight active sensing of bandwidth and buffering

[rjmcmahon]

Bloat is really a unit of memory so the inP metric in iperf 2 may be 
preferred. It's calculated using Little's law. The bloated UDP run over 
a WiFi link is ~2470 packets and min e2e latency of ~31 ms. The non 
bloat steady state is ~330 packets and min latency of ~4ms.  The latter 
is likely showing that the WiFi transmits being delayed by ~4ms NAV 
though I'd have to dig more to verify. (Note: I'll probably add inP to 
burst isoch traffic too. Not there yet as there hasn't been wide 
adoption of --isochronous though some use it extensively)

[root@fedora ~]# iperf -s -u -i 1 -e -B 192.168.1.231%eth1
------------------------------------------------------------
Server listening on UDP port 5001 with pid 611171
Binding to local address 192.168.1.231 and iface eth1
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.231%eth1 port 5001 connected with 192.168.1.15 
port 50870 (trip-times) (sock=3) (peer 2.1.9-master) on 2022-11-02 
12:00:36 (PDT)
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  
Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-1.00 sec   118 MBytes   986 Mbits/sec   0.016 ms 4545/88410 
(5.1%) 29.484/5.175/42.872/6.106 ms 83979 pps 83865/2476(513) pkts 4181
[  1] 0.00-1.00 sec  47 datagrams received out-of-order
[  1] 1.00-2.00 sec   123 MBytes  1.03 Gbits/sec   0.017 ms 3937/91326 
(4.3%) 31.576/29.353/39.704/1.169 ms 87396 pps 87389/2760(102) pkts 4068
[  1] 2.00-3.00 sec   123 MBytes  1.03 Gbits/sec   0.008 ms 3311/91243 
(3.6%) 31.602/29.218/39.096/1.259 ms 87899 pps 87932/2778(111) pkts 4090
[  1] 3.00-4.00 sec   123 MBytes  1.03 Gbits/sec   0.010 ms 3474/91385 
(3.8%) 31.570/29.032/42.499/1.856 ms 87943 pps 87911/2776(163) pkts 4093
[  1] 4.00-5.00 sec   122 MBytes  1.02 Gbits/sec   0.015 ms 4482/91331 
(4.9%) 31.349/29.163/40.234/1.291 ms 86813 pps 86849/2721(112) pkts 4073
[  1] 5.00-6.00 sec   122 MBytes  1.02 Gbits/sec   0.023 ms 4249/91102 
(4.7%) 31.376/29.336/41.963/1.318 ms 86890 pps 86853/2726(115) pkts 4069
[  1] 6.00-7.00 sec   122 MBytes  1.03 Gbits/sec   0.007 ms 4052/91410 
(4.4%) 31.424/29.178/38.683/1.037 ms 87344 pps 87358/2745(91) pkts 4087
[  1] 7.00-8.00 sec   123 MBytes  1.03 Gbits/sec   0.013 ms 3464/91297 
(3.8%) 31.414/29.189/38.161/1.061 ms 87839 pps 87833/2759(93) pkts 4110
[  1] 8.00-9.00 sec   123 MBytes  1.03 Gbits/sec   0.010 ms 3253/90930 
(3.6%) 31.686/29.335/39.989/1.478 ms 87662 pps 87677/2778(130) pkts 4068
[  1] 9.00-10.00 sec   122 MBytes  1.03 Gbits/sec   0.010 ms 3971/91331 
(4.3%) 31.486/29.197/39.750/1.407 ms 87378 pps 87360/2751(123) pkts 4079
[  1] 0.00-10.03 sec  1.20 GBytes  1.02 Gbits/sec   0.021 ms 
39124/913050 (4.3%) 31.314/5.175/42.872/2.365 ms 87095 pps 
873926/2727(206) pkts 4089
[  1] 0.00-10.03 sec  47 datagrams received out-of-order
[  3] WARNING: ack of last datagram failed.
[  2] local 192.168.1.231%eth1 port 5001 connected with 192.168.1.15 
port 46481 (trip-times) (sock=5) (peer 2.1.9-master) on 2022-11-02 
12:01:24 (PDT)
[  2] 0.00-1.00 sec   106 MBytes   886 Mbits/sec   0.012 ms 799/76100 
(1%) 4.377/2.780/9.694/0.547 ms 75391 pps 75301/330(41) pkts 25287
[  2] 0.00-1.00 sec  46 datagrams received out-of-order
[  2] 1.00-2.00 sec   106 MBytes   892 Mbits/sec   0.018 ms 746/76624 
(0.97%) 4.453/3.148/9.444/0.693 ms 75884 pps 75878/338(53) pkts 25048
[  2] 2.00-3.00 sec   106 MBytes   892 Mbits/sec   0.010 ms 640/76528 
(0.84%) 4.388/3.022/9.848/0.529 ms 75888 pps 75888/333(40) pkts 25425
[  2] 3.00-4.00 sec   106 MBytes   891 Mbits/sec   0.014 ms 707/76436 
(0.92%) 4.412/3.065/7.490/0.514 ms 75727 pps 75729/334(39) pkts 25231
[  2] 4.00-5.00 sec   106 MBytes   892 Mbits/sec   0.012 ms 764/76623 
(1%) 4.408/3.081/8.836/0.571 ms 75859 pps 75859/334(43) pkts 25300
[  2] 5.00-6.00 sec   106 MBytes   888 Mbits/sec   0.016 ms 941/76483 
(1.2%) 4.330/3.097/7.841/0.490 ms 75548 pps 75542/327(37) pkts 25648
[  2] 6.00-7.00 sec   106 MBytes   893 Mbits/sec   0.015 ms 549/76500 
(0.72%) 4.405/3.021/9.493/0.554 ms 75945 pps 75951/335(42) pkts 25346
[  2] 7.00-8.00 sec   106 MBytes   892 Mbits/sec   0.021 ms 683/76572 
(0.89%) 4.362/3.042/8.770/0.488 ms 75892 pps 75889/331(37) pkts 25576
[  2] 8.00-9.00 sec   106 MBytes   892 Mbits/sec   0.019 ms 688/76520 
(0.9%) 4.355/2.964/10.486/0.553 ms 75906 pps 75832/331(42) pkts 25599
[  2] 9.00-10.00 sec   106 MBytes   892 Mbits/sec   0.015 ms 644/76496 
(0.84%) 4.387/3.043/10.469/0.563 ms 75775 pps 75852/332(43) pkts 25419
[  2] 0.00-10.01 sec  1.04 GBytes   891 Mbits/sec   0.024 ms 7161/765314 
(0.94%) 4.388/2.780/10.486/0.554 ms 75775 pps 758153/333(42) pkts 25385
[  2] 0.00-10.01 sec  46 datagrams received out-of-order


[root@ctrl1fc35 ~]# iperf -c 192.168.1.231%enp2s0 -u --trip-times -b 1G
------------------------------------------------------------
Client connecting to 192.168.1.231, UDP port 5001 with pid 724001 via 
enp2s0 (1 flows)
Sending 1470 byte datagrams, IPG target: 10.95 us (kalman adjust)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.15 port 50870 connected with 192.168.1.231 port 
5001 (trip-times)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.00 sec  1.25 GBytes  1.07 Gbits/sec
[  1] Sent 913051 datagrams
[  1] Server Report:
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  
Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-10.03 sec  1.20 GBytes  1.02 Gbits/sec   0.020 ms 
39124/913050 (4.3%) 31.314/5.175/42.872/0.021 ms 90994 pps 90994/2849(2) 
pkts 4089
[  1] 0.00-10.03 sec  47 datagrams received out-of-order
[root@ctrl1fc35 ~]# iperf -c 192.168.1.231%enp2s0 -u --trip-times -b 
900m
------------------------------------------------------------
Client connecting to 192.168.1.231, UDP port 5001 with pid 724015 via 
enp2s0 (1 flows)
Sending 1470 byte datagrams, IPG target: 13.07 us (kalman adjust)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.15 port 46481 connected with 192.168.1.231 port 
5001 (trip-times)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.00 sec  1.05 GBytes   900 Mbits/sec
[  1] Sent 765315 datagrams
[  1] Server Report:
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  
Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-10.01 sec  1.04 GBytes   891 Mbits/sec   0.024 ms 7161/765314 
(0.94%) 4.388/2.780/10.486/0.032 ms 76490 pps 76490/336(2) pkts 25385
[  1] 0.00-10.01 sec  46 datagrams received out-of-order

Bob
> Hi Bob,
> 
> thanks!
> 
> 
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm 
>> <rpm@lists.bufferbloat.net> wrote:
>> 
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
> 
> 	[SM] Thank you for spelling this out explicitly, I only worked on a
> vage implicit assumption along those lines. However what I want to
> avoid is using delay magnitude itself as classifier between high and
> low load condition as that seems statistically uncouth to then show
> that the delay differs between the two classes;).
> 	Yet, your comment convinced me that my current load threshold (at
> least for the high load condition) probably is too small, exactly
> because the "base" of the high-load CDFs coincides with the base of
> the low-load CDFs implying that the high-load class contains too many
> samples with decent delay (which after all is one of the goals of the
> whole autorate endeavor).
> 
> 
>> A suggestion is to disable x-axis auto-scaling and start from zero.
> 
> 	[SM] Will reconsider. I started with start at zero, end then switched
> to an x-range that starts with the delay corresponding to 0.01% for
> the reflector/condition with the lowest such value and stops at 97.5%
> for the reflector/condition with the highest delay value. My rationale
> is that the base delay/path delay of each reflector is not all that
> informative* (and it can still be learned from reading the x-axis),
> the long tail > 50% however is where I expect most differences so I
> want to emphasize this and finally I wanted to avoid that the actual
> "curvy" part gets compressed so much that all lines more or less
> coincide. As I said, I will reconsider this
> 
> 
> *) We also maintain individual baselines per reflector, so I could
> just plot the differences from baseline, but that would essentially
> equalize all reflectors, and I think having a plot that easily shows
> reflectors with outlying base delay can be informative when selecting
> reflector candidates. However once we actually switch to OWDs baseline
> correction might be required anyways, as due to colck differences ICMP
> type 13/14 data can have massive offsets that are mostly indicative of
> un synched clocks**.
> 
> **) This is whyI would prefer to use NTP servers as reflectors with
> NTP requests, my expectation is all of these should be reasonably
> synced by default so that offsets should be in the sane range....
> 
> 
>> 
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense  bandwidth and 
>>> buffering
>>> problems. One of my favorites was their discovery that ICMP type 13
>>> got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques, 
>>> which
>>> I do wish were used more than single number summaries especially in
>>> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for 
>>> simulating
>>> and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

Thanks for your read/reply/suggestions, Dave!

Allow me to pull a few of your comments forward to try for a more cogent reply.

Dave wrote:
> Thank you very much for the steer to RFC9097. I'd completely missed that.
You're quite welcome! We all have a hand on the elephant with eyes closed, and only learn the whole story when we talk to each other...
The repo for UDPST is here: https://github.com/BroadbandForum/obudpst
We are also working to standardize the protocol the UDPST uses to measure RFC 9097:
https://datatracker.ietf.org/doc/html/draft-ietf-ippm-capacity-protocol-03
and potentially many aspects of network and application performance.

Dave wrote:
> Certainly we can expect bi-modal distributions ... should be called ... speed-lose.
Agreed. I'm glad we added the bimodal analysis feature in UDPST. It works with our max test duration (60s, we didn't want people to run capacity tests ad nauseum), but we won't be able to detect speed-loose beyond that.

Dave wrote:
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short interval <now> ...
We don't have adaptable duration either. Another perspective on duration comes from folks who test paths with mobile access: they prefer 5-7 second duration, and the Type C algo helps.

Dave wrote:
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
I think we report this in our prolific JSON-formatted output (#sub-interval with the Max Cap).

Dave wrote:
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
I'll try to install goresponsiveness later this week, so that we have a view of this.

Dave wrote:
> have you tried irtt?   (https://github.com/heistp/irtt )
I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481

irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...

thanks again, Dave.
Al

> -----Original Message-----
> From: Dave Taht <dave.taht@gmail.com>
> Sent: Monday, October 31, 2022 12:52 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
> 
> Thank you very much for the steer to RFC9097. I'd completely missed that.
> 
> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> >
> > (astute readers may have guessed that I pressed "send" too soon on previous
> message...)
> >
> > I also conducted upstream tests this time, here are the results:
> > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
> >
> > Net Qual                           UDPST (RFC9097)              Ookla
> > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> Ping(no load)
> > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22       8
> > 22         281 m 214ms  8ms        27 (52)  25       5-248      22       8
> > 22         290 m 207ms  8ms        27 (55)  28       0-253      22       9
> > 21         330 m 182ms  11ms       23 (44)  28       0-255      22       7
> > 22         334 m 180ms  9ms        33 (56)  25       0-255      22       9
> >
> > The Upstream capacity measurements reflect an interesting feature that we
> can reliably and repeatably measure with UDPST. The first ~3 seconds of
> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> behavior in its 1 second sub-interval measurements and has a bimodal reporting
> option that divides the complete measurement interval in two time intervals to
> report an initial (turbo) max capacity and a steady-state max capacity for the
> later intervals. The UDPST capacity results present both measurements: steady-
> state first.
> 
> Certainly we can expect bi-model distributions from many ISPs, as, for
> one thing, the "speedboost" concept remains popular, except that it's
> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> it is often configured "sneakily", in that it doesn't kick in for the
> typical observed duration of the test, for some, they cut the
> available bandwidth about 20s in, others, 1 or 5 minutes.
> 
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short
> interval it runs for and the tools also allow for manual override of length.
> 
> we caught a lot of tomfoolery with flent's rrul test running by default for
> 1m.
> 
> Also, AQMs on the path can take a while to find the optimal drop or mark rate.
> 
> >
> > The capacity processing in networkQuality and Ookla appear to report the
> steady-state result.
> 
> Ookla used to basically report the last result. Also it's not a good
> indicator of web traffic behavior at all, watching the curve
> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> (40ms)....
> 
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
> 
> I haven't poked into the current iteration of the goresponsiveness
> test at all: https://urldefense.com/v3/__https://github.com/network-
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> would be good to try collecting more statistics and histograms and
> methods of analyzing the data in that libre-source version.
> 
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> 
> >I watched the upstream capacity measurements on the Ookla app, and could
> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> the test which determined the final result.
> 
> I tend to get upset when I see ookla's new test flash a peak result in
> the seconds and then settle on some lower number somehow.
> So far as I know they are only sampling the latency every 250ms.
> 
> >
> > The working latency is about 200ms in networkQuality and about 280ms as
> measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> in working latency.  Also, we used the very dynamic Type C load
> adjustment/search algorithm in UDPST during all of this testing, which could
> explain the higher working latency to some degree.
> >
> > So, it's worth noting that the measurements needed for assessing working
> latency/responsiveness are available in the UDPST utility, and that the UDPST
> measurements are conducted on UDP transport (used by a growing fraction of
> Internet traffic).
> 
> Thx, didn't know of this work til now!
> 
> have you tried irtt?
> 
> >
> > comments welcome of course,
> > Al
> >
> > > -----Original Message-----
> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > Sent: Sunday, October 30, 2022 8:09 PM
> > > To: ippm@ietf.org
> > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > metrics
> > >
> > >
> > > Hi again RPM friends and IPPM'ers,
> > >
> > > As promised, I repeated the tests shared last week, this time using both
> the
> > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > followed Sebastian's calculations as well.
> > >
> > > Working Latency & Capacity Summary
> > >
> > > Net Qual                           UDPST                        Ookla
> > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> > > Ping(no load)
> > > 885       916 m  66ms   8ms        970      28       0-20       940      8
> > > 888      1355 h  44ms   8ms        966      28       0-23       940      8
> > > 891      1109 h  54ms   8ms        968      27       0-19       940      9
> > > 887      1141 h  53ms   11ms       966      27       0-18       937      7
> > > 884      1151 h  52ms   9ms        968      28       0-20       937      9
> > >
> > > With the sequential test option, I noticed that networkQuality achieved
> nearly
> > > the maximum capacity reported almost immediately at the start of a test.
> > > However, the reported capacities are low by about 60Mbps, especially when
> > > compared to the Ookla TCP measurements.
> > >
> > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> the
> > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> networkQuality
> > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > buffering in the downstream direction.
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > To: ippm@ietf.org
> > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > metrics
> > > >
> > > >
> > > > Hi RPM friends and IPPM'ers,
> > > >
> > > > I was wondering what a comparison of some of the "working latency"
> metrics
> > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> the
> > > > downlink provisioned for 1Gbps.
> > > >
> > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > Speedtest
> > > > with as similar connectivity as possible (but we know that the traffic
> will
> > > > diverge to different servers and we can't change that aspect).
> > > >
> > > > Here's a quick summary of yesterday's results:
> > > >
> > > > Working Latency & Capacity Summary
> > > >
> > > > Net Qual                UDPST                        Ookla
> > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no
> load)
> > > > 878       62            970      28       0-19       941      6
> > > > 891       92            970      27       0-20       940      7
> > > > 891       120           966      28       0-22       937      9
> > > > 890       112           970      28       0-21       940      8
> > > > 903       70            970      28       0-16       935      9
> > > >
> > > > Note: all RPM values were categorized as Low.
> > > >
> > > > networkQuality downstream capacities are always on the low side compared
> to
> > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> Ookla
> > > > achieved. I think that a longer test duration might be needed to achieve
> the
> > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> were
> > > > certainly headed in the right direction. (I recently upgraded to
> Monterey
> > > 12.6
> > > > on my MacBook, so should have the latest version.)
> > > >
> > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> have
> > > > run the tests with the -v option to help with comparisons. I'll repeat
> this
> > > > test when I can make time.
> > > >
> > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> and
> > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> access
> > > and
> > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> were
> > > run
> > > > on an Ubuntu VM in my MacBook.
> > > >
> > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > weekend did not include the new responsiveness metric! I included the
> ping
> > > > results anyway, and it was clearly using a server in the nearby area.
> > > >
> > > > So, I have some more work to do, but I hope this is interesting-enough
> to
> > > > start some comparison discussions, and bring-out some suggestions.
> > > >
> > > > happy testing all,
> > > > Al
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >
> > >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > T!g-
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> KXU$
> 
> 
> 
> --
> This song goes out to all the folk that thought Stadia would work:
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> song-activity-6981366665607352320-
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> c48iQFub34zz4iE$
> Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[Dave Taht]

On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:

> > have you tried irtt?   (https://github.com/heistp/irtt )
> I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481

irtt was meant to simulate high speed voip and one day
videoconferencing. Please inspect the json output
for other metrics. Due to OS limits it is typically only accurate to a
3ms interval. One thing it does admirably is begin to expose the
sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
technologies, as well as request/grant systems like cable and gpon,
especially when otherwise idle.

Here is a highres plot of starlink's behaviors from last year:
https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848/3238

clearly showing them "optimizing for bandwidth" and changing next sat
hop, and about a 40ms interval of buffering between these switches.
I'd published elsewhere, if anyone cares, a preliminary study of what
starlink's default behaviors did to cubic and BBR...

>
> irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...

The more the merrier! Someday the "right" metrics will arrive.

As a side note, this paper focuses on RAN uplink latency
https://dl.ifip.org/db/conf/itc/itc2021/1570740615.pdf which I think
is a major barrier to most forms of 5G actually achieving good
performance in a FPS game, if it is true for more RANs. I'd like more
to be testing uplink latencies idle and with load, on all
technologies.

>
> thanks again, Dave.
> Al
>
> > -----Original Message-----
> > From: Dave Taht <dave.taht@gmail.com>
> > Sent: Monday, October 31, 2022 12:52 PM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> > Thank you very much for the steer to RFC9097. I'd completely missed that.
> >
> > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > >
> > > (astute readers may have guessed that I pressed "send" too soon on previous
> > message...)
> > >
> > > I also conducted upstream tests this time, here are the results:
> > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > Medium)
> > >
> > > Net Qual                           UDPST (RFC9097)              Ookla
> > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> > Ping(no load)
> > > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22       8
> > > 22         281 m 214ms  8ms        27 (52)  25       5-248      22       8
> > > 22         290 m 207ms  8ms        27 (55)  28       0-253      22       9
> > > 21         330 m 182ms  11ms       23 (44)  28       0-255      22       7
> > > 22         334 m 180ms  9ms        33 (56)  25       0-255      22       9
> > >
> > > The Upstream capacity measurements reflect an interesting feature that we
> > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > behavior in its 1 second sub-interval measurements and has a bimodal reporting
> > option that divides the complete measurement interval in two time intervals to
> > report an initial (turbo) max capacity and a steady-state max capacity for the
> > later intervals. The UDPST capacity results present both measurements: steady-
> > state first.
> >
> > Certainly we can expect bi-model distributions from many ISPs, as, for
> > one thing, the "speedboost" concept remains popular, except that it's
> > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > it is often configured "sneakily", in that it doesn't kick in for the
> > typical observed duration of the test, for some, they cut the
> > available bandwidth about 20s in, others, 1 or 5 minutes.
> >
> > One of my biggest issues with the rpm spec so far is that it should,
> > at least, sometimes, run randomly longer than the overly short
> > interval it runs for and the tools also allow for manual override of length.
> >
> > we caught a lot of tomfoolery with flent's rrul test running by default for
> > 1m.
> >
> > Also, AQMs on the path can take a while to find the optimal drop or mark rate.
> >
> > >
> > > The capacity processing in networkQuality and Ookla appear to report the
> > steady-state result.
> >
> > Ookla used to basically report the last result. Also it's not a good
> > indicator of web traffic behavior at all, watching the curve
> > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > (40ms)....
> >
> > So adding another mode - how quickly is peak bandwidth actually
> > reached, would be nice.
> >
> > I haven't poked into the current iteration of the goresponsiveness
> > test at all: https://urldefense.com/v3/__https://github.com/network-
> > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > would be good to try collecting more statistics and histograms and
> > methods of analyzing the data in that libre-source version.
> >
> > How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> >
> > >I watched the upstream capacity measurements on the Ookla app, and could
> > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> > the test which determined the final result.
> >
> > I tend to get upset when I see ookla's new test flash a peak result in
> > the seconds and then settle on some lower number somehow.
> > So far as I know they are only sampling the latency every 250ms.
> >
> > >
> > > The working latency is about 200ms in networkQuality and about 280ms as
> > measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> > ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> > in working latency.  Also, we used the very dynamic Type C load
> > adjustment/search algorithm in UDPST during all of this testing, which could
> > explain the higher working latency to some degree.
> > >
> > > So, it's worth noting that the measurements needed for assessing working
> > latency/responsiveness are available in the UDPST utility, and that the UDPST
> > measurements are conducted on UDP transport (used by a growing fraction of
> > Internet traffic).
> >
> > Thx, didn't know of this work til now!
> >
> > have you tried irtt?
> >
> > >
> > > comments welcome of course,
> > > Al
> > >
> > > > -----Original Message-----
> > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > To: ippm@ietf.org
> > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > metrics
> > > >
> > > >
> > > > Hi again RPM friends and IPPM'ers,
> > > >
> > > > As promised, I repeated the tests shared last week, this time using both
> > the
> > > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > > followed Sebastian's calculations as well.
> > > >
> > > > Working Latency & Capacity Summary
> > > >
> > > > Net Qual                           UDPST                        Ookla
> > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> > > > Ping(no load)
> > > > 885       916 m  66ms   8ms        970      28       0-20       940      8
> > > > 888      1355 h  44ms   8ms        966      28       0-23       940      8
> > > > 891      1109 h  54ms   8ms        968      27       0-19       940      9
> > > > 887      1141 h  53ms   11ms       966      27       0-18       937      7
> > > > 884      1151 h  52ms   9ms        968      28       0-20       937      9
> > > >
> > > > With the sequential test option, I noticed that networkQuality achieved
> > nearly
> > > > the maximum capacity reported almost immediately at the start of a test.
> > > > However, the reported capacities are low by about 60Mbps, especially when
> > > > compared to the Ookla TCP measurements.
> > > >
> > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> > the
> > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > networkQuality
> > > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > > buffering in the downstream direction.
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > To: ippm@ietf.org
> > > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > > metrics
> > > > >
> > > > >
> > > > > Hi RPM friends and IPPM'ers,
> > > > >
> > > > > I was wondering what a comparison of some of the "working latency"
> > metrics
> > > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> > the
> > > > > downlink provisioned for 1Gbps.
> > > > >
> > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > > Speedtest
> > > > > with as similar connectivity as possible (but we know that the traffic
> > will
> > > > > diverge to different servers and we can't change that aspect).
> > > > >
> > > > > Here's a quick summary of yesterday's results:
> > > > >
> > > > > Working Latency & Capacity Summary
> > > > >
> > > > > Net Qual                UDPST                        Ookla
> > > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no
> > load)
> > > > > 878       62            970      28       0-19       941      6
> > > > > 891       92            970      27       0-20       940      7
> > > > > 891       120           966      28       0-22       937      9
> > > > > 890       112           970      28       0-21       940      8
> > > > > 903       70            970      28       0-16       935      9
> > > > >
> > > > > Note: all RPM values were categorized as Low.
> > > > >
> > > > > networkQuality downstream capacities are always on the low side compared
> > to
> > > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> > Ookla
> > > > > achieved. I think that a longer test duration might be needed to achieve
> > the
> > > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> > were
> > > > > certainly headed in the right direction. (I recently upgraded to
> > Monterey
> > > > 12.6
> > > > > on my MacBook, so should have the latest version.)
> > > > >
> > > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> > have
> > > > > run the tests with the -v option to help with comparisons. I'll repeat
> > this
> > > > > test when I can make time.
> > > > >
> > > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> > and
> > > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> > access
> > > > and
> > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> > were
> > > > run
> > > > > on an Ubuntu VM in my MacBook.
> > > > >
> > > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > > weekend did not include the new responsiveness metric! I included the
> > ping
> > > > > results anyway, and it was clearly using a server in the nearby area.
> > > > >
> > > > > So, I have some more work to do, but I hope this is interesting-enough
> > to
> > > > > start some comparison discussions, and bring-out some suggestions.
> > > > >
> > > > > happy testing all,
> > > > > Al
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > >
> > > >
> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > T!g-
> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > KXU$
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> > song-activity-6981366665607352320-
> > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > c48iQFub34zz4iE$
> > Dave Täht CEO, TekLibre, LLC



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[Dave Taht]

Dear Al:

OK, I took your udpst tool for a spin.

NICE! 120k binary (I STILL work on machines with only 4MB of flash),
good feature set, VERY fast, and in very brief testing, seemed
to be accurate in the starlink case, though it's hard to tell with
them as the rate changes every 15s.

I filed a couple bug reports on trivial stuff:
https://github.com/BroadbandForum/obudpst/issues/8

(Adding diffserv and ecn washing or marking detection would be a nice
feature to have)

Aside from the sheer joy coming from the "it compiles! and runs!"
phase I haven't looked much further.

I left a copy running on one of my starlink testbeds -
fremont.starlink.taht.net - if anyone wants to try it. It's
instrumented with netperf, flent, irtt, iperf2 (not quite the latest
version from bob, but close), and now udpst, and good to about a gbit.

nice tool!

Has anyone here played with crusader? ( https://github.com/Zoxc/crusader )

On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>
> > > have you tried irtt?   (https://github.com/heistp/irtt )
> > I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481
>
> irtt was meant to simulate high speed voip and one day
> videoconferencing. Please inspect the json output
> for other metrics. Due to OS limits it is typically only accurate to a
> 3ms interval. One thing it does admirably is begin to expose the
> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> technologies, as well as request/grant systems like cable and gpon,
> especially when otherwise idle.
>
> Here is a highres plot of starlink's behaviors from last year:
> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848/3238
>
> clearly showing them "optimizing for bandwidth" and changing next sat
> hop, and about a 40ms interval of buffering between these switches.
> I'd published elsewhere, if anyone cares, a preliminary study of what
> starlink's default behaviors did to cubic and BBR...
>
> >
> > irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...
>
> The more the merrier! Someday the "right" metrics will arrive.
>
> As a side note, this paper focuses on RAN uplink latency
> https://dl.ifip.org/db/conf/itc/itc2021/1570740615.pdf which I think
> is a major barrier to most forms of 5G actually achieving good
> performance in a FPS game, if it is true for more RANs. I'd like more
> to be testing uplink latencies idle and with load, on all
> technologies.
>
> >
> > thanks again, Dave.
> > Al
> >
> > > -----Original Message-----
> > > From: Dave Taht <dave.taht@gmail.com>
> > > Sent: Monday, October 31, 2022 12:52 PM
> > > To: MORTON JR., AL <acmorton@att.com>
> > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > > metrics
> > >
> > > Thank you very much for the steer to RFC9097. I'd completely missed that.
> > >
> > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > > >
> > > > (astute readers may have guessed that I pressed "send" too soon on previous
> > > message...)
> > > >
> > > > I also conducted upstream tests this time, here are the results:
> > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > Medium)
> > > >
> > > > Net Qual                           UDPST (RFC9097)              Ookla
> > > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> > > Ping(no load)
> > > > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22       8
> > > > 22         281 m 214ms  8ms        27 (52)  25       5-248      22       8
> > > > 22         290 m 207ms  8ms        27 (55)  28       0-253      22       9
> > > > 21         330 m 182ms  11ms       23 (44)  28       0-255      22       7
> > > > 22         334 m 180ms  9ms        33 (56)  25       0-255      22       9
> > > >
> > > > The Upstream capacity measurements reflect an interesting feature that we
> > > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > > behavior in its 1 second sub-interval measurements and has a bimodal reporting
> > > option that divides the complete measurement interval in two time intervals to
> > > report an initial (turbo) max capacity and a steady-state max capacity for the
> > > later intervals. The UDPST capacity results present both measurements: steady-
> > > state first.
> > >
> > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > one thing, the "speedboost" concept remains popular, except that it's
> > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > it is often configured "sneakily", in that it doesn't kick in for the
> > > typical observed duration of the test, for some, they cut the
> > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > >
> > > One of my biggest issues with the rpm spec so far is that it should,
> > > at least, sometimes, run randomly longer than the overly short
> > > interval it runs for and the tools also allow for manual override of length.
> > >
> > > we caught a lot of tomfoolery with flent's rrul test running by default for
> > > 1m.
> > >
> > > Also, AQMs on the path can take a while to find the optimal drop or mark rate.
> > >
> > > >
> > > > The capacity processing in networkQuality and Ookla appear to report the
> > > steady-state result.
> > >
> > > Ookla used to basically report the last result. Also it's not a good
> > > indicator of web traffic behavior at all, watching the curve
> > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > (40ms)....
> > >
> > > So adding another mode - how quickly is peak bandwidth actually
> > > reached, would be nice.
> > >
> > > I haven't poked into the current iteration of the goresponsiveness
> > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > > would be good to try collecting more statistics and histograms and
> > > methods of analyzing the data in that libre-source version.
> > >
> > > How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> > >
> > > >I watched the upstream capacity measurements on the Ookla app, and could
> > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> > > the test which determined the final result.
> > >
> > > I tend to get upset when I see ookla's new test flash a peak result in
> > > the seconds and then settle on some lower number somehow.
> > > So far as I know they are only sampling the latency every 250ms.
> > >
> > > >
> > > > The working latency is about 200ms in networkQuality and about 280ms as
> > > measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> > > in working latency.  Also, we used the very dynamic Type C load
> > > adjustment/search algorithm in UDPST during all of this testing, which could
> > > explain the higher working latency to some degree.
> > > >
> > > > So, it's worth noting that the measurements needed for assessing working
> > > latency/responsiveness are available in the UDPST utility, and that the UDPST
> > > measurements are conducted on UDP transport (used by a growing fraction of
> > > Internet traffic).
> > >
> > > Thx, didn't know of this work til now!
> > >
> > > have you tried irtt?
> > >
> > > >
> > > > comments welcome of course,
> > > > Al
> > > >
> > > > > -----Original Message-----
> > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > To: ippm@ietf.org
> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > Latency"
> > > > > metrics
> > > > >
> > > > >
> > > > > Hi again RPM friends and IPPM'ers,
> > > > >
> > > > > As promised, I repeated the tests shared last week, this time using both
> > > the
> > > > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > > > followed Sebastian's calculations as well.
> > > > >
> > > > > Working Latency & Capacity Summary
> > > > >
> > > > > Net Qual                           UDPST                        Ookla
> > > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> > > > > Ping(no load)
> > > > > 885       916 m  66ms   8ms        970      28       0-20       940      8
> > > > > 888      1355 h  44ms   8ms        966      28       0-23       940      8
> > > > > 891      1109 h  54ms   8ms        968      27       0-19       940      9
> > > > > 887      1141 h  53ms   11ms       966      27       0-18       937      7
> > > > > 884      1151 h  52ms   9ms        968      28       0-20       937      9
> > > > >
> > > > > With the sequential test option, I noticed that networkQuality achieved
> > > nearly
> > > > > the maximum capacity reported almost immediately at the start of a test.
> > > > > However, the reported capacities are low by about 60Mbps, especially when
> > > > > compared to the Ookla TCP measurements.
> > > > >
> > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> > > the
> > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > networkQuality
> > > > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > > > buffering in the downstream direction.
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > To: ippm@ietf.org
> > > > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > > > metrics
> > > > > >
> > > > > >
> > > > > > Hi RPM friends and IPPM'ers,
> > > > > >
> > > > > > I was wondering what a comparison of some of the "working latency"
> > > metrics
> > > > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> > > the
> > > > > > downlink provisioned for 1Gbps.
> > > > > >
> > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > > > Speedtest
> > > > > > with as similar connectivity as possible (but we know that the traffic
> > > will
> > > > > > diverge to different servers and we can't change that aspect).
> > > > > >
> > > > > > Here's a quick summary of yesterday's results:
> > > > > >
> > > > > > Working Latency & Capacity Summary
> > > > > >
> > > > > > Net Qual                UDPST                        Ookla
> > > > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no
> > > load)
> > > > > > 878       62            970      28       0-19       941      6
> > > > > > 891       92            970      27       0-20       940      7
> > > > > > 891       120           966      28       0-22       937      9
> > > > > > 890       112           970      28       0-21       940      8
> > > > > > 903       70            970      28       0-16       935      9
> > > > > >
> > > > > > Note: all RPM values were categorized as Low.
> > > > > >
> > > > > > networkQuality downstream capacities are always on the low side compared
> > > to
> > > > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> > > Ookla
> > > > > > achieved. I think that a longer test duration might be needed to achieve
> > > the
> > > > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> > > were
> > > > > > certainly headed in the right direction. (I recently upgraded to
> > > Monterey
> > > > > 12.6
> > > > > > on my MacBook, so should have the latest version.)
> > > > > >
> > > > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> > > have
> > > > > > run the tests with the -v option to help with comparisons. I'll repeat
> > > this
> > > > > > test when I can make time.
> > > > > >
> > > > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> > > and
> > > > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> > > access
> > > > > and
> > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> > > were
> > > > > run
> > > > > > on an Ubuntu VM in my MacBook.
> > > > > >
> > > > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > > > weekend did not include the new responsiveness metric! I included the
> > > ping
> > > > > > results anyway, and it was clearly using a server in the nearby area.
> > > > > >
> > > > > > So, I have some more work to do, but I hope this is interesting-enough
> > > to
> > > > > > start some comparison discussions, and bring-out some suggestions.
> > > > > >
> > > > > > happy testing all,
> > > > > > Al
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > >
> > > > >
> > > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > T!g-
> > > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > KXU$
> > >
> > >
> > >
> > > --
> > > This song goes out to all the folk that thought Stadia would work:
> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> > > song-activity-6981366665607352320-
> > > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > c48iQFub34zz4iE$
> > > Dave Täht CEO, TekLibre, LLC
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

Hi Dave, 
Thanks for trying UDPST (RFC 9097)! 

Something you might try with starlink:
use the -X option and UDPST will generate random payloads.

The measurements with -X will reflect the uncompressed that are possible.
I tried this on a ship-board Internet access: uncompressed rate was 100kbps.

A few more quick replies below,
Al

> -----Original Message-----
> From: Dave Taht <dave.taht@gmail.com>
> Sent: Tuesday, November 1, 2022 12:22 AM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
> 
> Dear Al:
> 
> OK, I took your udpst tool for a spin.
> 
> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> good feature set, VERY fast, 
[acm] 
Len Ciavattone (my partner in crime on several measurement projects) is the lead coder: he has implemented many measurement tools extremely efficiently, this one in C-lang.

> and in very brief testing, seemed
> to be accurate in the starlink case, though it's hard to tell with
> them as the rate changes every 15s.
[acm] 
Great! Our guiding principle developing UDPST has been to test the accuracy of measurements against a ground-truth. It pays-off.

> 
> I filed a couple bug reports on trivial stuff:
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> SswH_opYmEw$
[acm] 
much appreciated...  We have an OB-UDPST project meeting this Friday, can discuss then.

> 
> (Adding diffserv and ecn washing or marking detection would be a nice
> feature to have)
> 
> Aside from the sheer joy coming from the "it compiles! and runs!"
> phase I haven't looked much further.
> 
> I left a copy running on one of my starlink testbeds -
> fremont.starlink.taht.net - if anyone wants to try it. It's
> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> version from bob, but close), and now udpst, and good to about a gbit.
> 
> nice tool!
[acm] 
Thanks again!

> 
> Has anyone here played with crusader? (
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$   )
> 
> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> >
> > > > have you tried irtt?
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$   )
> > > I have not. Seems like a reasonable tool for UDP testing. The feature I
> didn't like in my scan of the documentation is the use of Inter-packet delay
> variation (IPDV) instead of packet delay variation (PDV): variation from the
> minimum (or reference) delay. The morbidly curious can find my analysis in RFC
> 5481:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> T7QSlc$
> >
> > irtt was meant to simulate high speed voip and one day
> > videoconferencing. Please inspect the json output
> > for other metrics. Due to OS limits it is typically only accurate to a
> > 3ms interval. One thing it does admirably is begin to expose the
> > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > technologies, as well as request/grant systems like cable and gpon,
> > especially when otherwise idle.
> >
> > Here is a highres plot of starlink's behaviors from last year:
> > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> bandwidth-
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> >
> > clearly showing them "optimizing for bandwidth" and changing next sat
> > hop, and about a 40ms interval of buffering between these switches.
> > I'd published elsewhere, if anyone cares, a preliminary study of what
> > starlink's default behaviors did to cubic and BBR...
> >
> > >
> > > irtt's use of IPDV means that the results won’t compare with UDPST, and
> possibly networkQuality. But I may give it a try anyway...
> >
> > The more the merrier! Someday the "right" metrics will arrive.
> >
> > As a side note, this paper focuses on RAN uplink latency
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHgvqerjg$   which I think
> > is a major barrier to most forms of 5G actually achieving good
> > performance in a FPS game, if it is true for more RANs. I'd like more
> > to be testing uplink latencies idle and with load, on all
> > technologies.
> >
> > >
> > > thanks again, Dave.
> > > Al
> > >
> > > > -----Original Message-----
> > > > From: Dave Taht <dave.taht@gmail.com>
> > > > Sent: Monday, October 31, 2022 12:52 PM
> > > > To: MORTON JR., AL <acmorton@att.com>
> > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > > metrics
> > > >
> > > > Thank you very much for the steer to RFC9097. I'd completely missed
> that.
> > > >
> > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > > > >
> > > > > (astute readers may have guessed that I pressed "send" too soon on
> previous
> > > > message...)
> > > > >
> > > > > I also conducted upstream tests this time, here are the results:
> > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > > Medium)
> > > > >
> > > > > Net Qual                           UDPST (RFC9097)              Ookla
> > > > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> > > > Ping(no load)
> > > > > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
> 8
> > > > > 22         281 m 214ms  8ms        27 (52)  25       5-248      22
> 8
> > > > > 22         290 m 207ms  8ms        27 (55)  28       0-253      22
> 9
> > > > > 21         330 m 182ms  11ms       23 (44)  28       0-255      22
> 7
> > > > > 22         334 m 180ms  9ms        33 (56)  25       0-255      22
> 9
> > > > >
> > > > > The Upstream capacity measurements reflect an interesting feature that
> we
> > > > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > > > behavior in its 1 second sub-interval measurements and has a bimodal
> reporting
> > > > option that divides the complete measurement interval in two time
> intervals to
> > > > report an initial (turbo) max capacity and a steady-state max capacity
> for the
> > > > later intervals. The UDPST capacity results present both measurements:
> steady-
> > > > state first.
> > > >
> > > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > > one thing, the "speedboost" concept remains popular, except that it's
> > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > > it is often configured "sneakily", in that it doesn't kick in for the
> > > > typical observed duration of the test, for some, they cut the
> > > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > > >
> > > > One of my biggest issues with the rpm spec so far is that it should,
> > > > at least, sometimes, run randomly longer than the overly short
> > > > interval it runs for and the tools also allow for manual override of
> length.
> > > >
> > > > we caught a lot of tomfoolery with flent's rrul test running by default
> for
> > > > 1m.
> > > >
> > > > Also, AQMs on the path can take a while to find the optimal drop or mark
> rate.
> > > >
> > > > >
> > > > > The capacity processing in networkQuality and Ookla appear to report
> the
> > > > steady-state result.
> > > >
> > > > Ookla used to basically report the last result. Also it's not a good
> > > > indicator of web traffic behavior at all, watching the curve
> > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > > (40ms)....
> > > >
> > > > So adding another mode - how quickly is peak bandwidth actually
> > > > reached, would be nice.
> > > >
> > > > I haven't poked into the current iteration of the goresponsiveness
> > > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > > > would be good to try collecting more statistics and histograms and
> > > > methods of analyzing the data in that libre-source version.
> > > >
> > > > How does networkQuality compare vs a vs your tool vs a vs
> goresponsiveness?
> > > >
> > > > >I watched the upstream capacity measurements on the Ookla app, and
> could
> > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> most of
> > > > the test which determined the final result.
> > > >
> > > > I tend to get upset when I see ookla's new test flash a peak result in
> > > > the seconds and then settle on some lower number somehow.
> > > > So far as I know they are only sampling the latency every 250ms.
> > > >
> > > > >
> > > > > The working latency is about 200ms in networkQuality and about 280ms
> as
> > > > measured by UDPST (RFC9097). Note that the networkQuality minimum delay
> is
> > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> difference
> > > > in working latency.  Also, we used the very dynamic Type C load
> > > > adjustment/search algorithm in UDPST during all of this testing, which
> could
> > > > explain the higher working latency to some degree.
> > > > >
> > > > > So, it's worth noting that the measurements needed for assessing
> working
> > > > latency/responsiveness are available in the UDPST utility, and that the
> UDPST
> > > > measurements are conducted on UDP transport (used by a growing fraction
> of
> > > > Internet traffic).
> > > >
> > > > Thx, didn't know of this work til now!
> > > >
> > > > have you tried irtt?
> > > >
> > > > >
> > > > > comments welcome of course,
> > > > > Al
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > > To: ippm@ietf.org
> > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > > Latency"
> > > > > > metrics
> > > > > >
> > > > > >
> > > > > > Hi again RPM friends and IPPM'ers,
> > > > > >
> > > > > > As promised, I repeated the tests shared last week, this time using
> both
> > > > the
> > > > > > verbose (-v) and sequential (-s) dwn/up test options of
> networkQuality. I
> > > > > > followed Sebastian's calculations as well.
> > > > > >
> > > > > > Working Latency & Capacity Summary
> > > > > >
> > > > > > Net Qual                           UDPST
> Ookla
> > > > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
> DnCap
> > > > > > Ping(no load)
> > > > > > 885       916 m  66ms   8ms        970      28       0-20       940
> 8
> > > > > > 888      1355 h  44ms   8ms        966      28       0-23       940
> 8
> > > > > > 891      1109 h  54ms   8ms        968      27       0-19       940
> 9
> > > > > > 887      1141 h  53ms   11ms       966      27       0-18       937
> 7
> > > > > > 884      1151 h  52ms   9ms        968      28       0-20       937
> 9
> > > > > >
> > > > > > With the sequential test option, I noticed that networkQuality
> achieved
> > > > nearly
> > > > > > the maximum capacity reported almost immediately at the start of a
> test.
> > > > > > However, the reported capacities are low by about 60Mbps, especially
> when
> > > > > > compared to the Ookla TCP measurements.
> > > > > >
> > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high
> end of
> > > > the
> > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > > networkQuality
> > > > > > RPM measurements were categorized as "High". There doesn't seem to
> be much
> > > > > > buffering in the downstream direction.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > > To: ippm@ietf.org
> > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > > > > > metrics
> > > > > > >
> > > > > > >
> > > > > > > Hi RPM friends and IPPM'ers,
> > > > > > >
> > > > > > > I was wondering what a comparison of some of the "working latency"
> > > > metrics
> > > > > > > would look like, so I ran some tests using a service on DOCSIS
> 3.1, with
> > > > the
> > > > > > > downlink provisioned for 1Gbps.
> > > > > > >
> > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
> Ookla
> > > > > > Speedtest
> > > > > > > with as similar connectivity as possible (but we know that the
> traffic
> > > > will
> > > > > > > diverge to different servers and we can't change that aspect).
> > > > > > >
> > > > > > > Here's a quick summary of yesterday's results:
> > > > > > >
> > > > > > > Working Latency & Capacity Summary
> > > > > > >
> > > > > > > Net Qual                UDPST                        Ookla
> > > > > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
> Ping(no
> > > > load)
> > > > > > > 878       62            970      28       0-19       941      6
> > > > > > > 891       92            970      27       0-20       940      7
> > > > > > > 891       120           966      28       0-22       937      9
> > > > > > > 890       112           970      28       0-21       940      8
> > > > > > > 903       70            970      28       0-16       935      9
> > > > > > >
> > > > > > > Note: all RPM values were categorized as Low.
> > > > > > >
> > > > > > > networkQuality downstream capacities are always on the low side
> compared
> > > > to
> > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
> what
> > > > Ookla
> > > > > > > achieved. I think that a longer test duration might be needed to
> achieve
> > > > the
> > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
> observed
> > > > were
> > > > > > > certainly headed in the right direction. (I recently upgraded to
> > > > Monterey
> > > > > > 12.6
> > > > > > > on my MacBook, so should have the latest version.)
> > > > > > >
> > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
> should
> > > > have
> > > > > > > run the tests with the -v option to help with comparisons. I'll
> repeat
> > > > this
> > > > > > > test when I can make time.
> > > > > > >
> > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during the
> test)
> > > > and
> > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
> very
> > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
> this
> > > > access
> > > > > > and
> > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
> Chicago
> > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
> measurements
> > > > were
> > > > > > run
> > > > > > > on an Ubuntu VM in my MacBook.
> > > > > > >
> > > > > > > The big disappointment was that the Ookla desktop app I updated
> over the
> > > > > > > weekend did not include the new responsiveness metric! I included
> the
> > > > ping
> > > > > > > results anyway, and it was clearly using a server in the nearby
> area.
> > > > > > >
> > > > > > > So, I have some more work to do, but I hope this is interesting-
> enough
> > > > to
> > > > > > > start some comparison discussions, and bring-out some suggestions.
> > > > > > >
> > > > > > > happy testing all,
> > > > > > > Al
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ippm mailing list
> > > > > > > ippm@ietf.org
> > > > > > >
> > > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > >
> > > > > >
> > > >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > T!g-
> > > >
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > > KXU$
> > > >
> > > >
> > > >
> > > > --
> > > > This song goes out to all the folk that thought Stadia would work:
> > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-
> > > > song-activity-6981366665607352320-
> > > >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > > c48iQFub34zz4iE$
> > > > Dave Täht CEO, TekLibre, LLC
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-song-activity-6981366665607352320-
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHLHDpSWs$
> > Dave Täht CEO, TekLibre, LLC
> 
> 
> 
> --
> This song goes out to all the folk that thought Stadia would work:
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> song-activity-6981366665607352320-
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHLHDpSWs$
> Dave Täht CEO, TekLibre, LLC

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

Hi all,

I have been working through the threads on misery metrics and lightweight sensing of bw and buffering, and with those insights and gold-nuggets in mind, I'm iterating through testing the combinations of tools that Dave That and Bob McMahon suggested.

earlier this week, Dave wrote:
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?

goresponsiveness installed flawlessly - very nice instructions and getting-started info.

Comparison of NetQual (networkQuality -vs), UDPST -A c, gores/goresponsiveness

Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-stream service
(capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)

NetQual                            UDPST (RFC 9097)             gores
DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap    RPM    DelLD
882       788 m  76ms   8ms        967      28       0-16       127      (1382) 43ms 
892      1036 h  58ms   8          966      27       0-18       128      (2124) 28ms
887      1304 h  46ms   6          969      27       0-18       130      (1478) 41ms
885      1008 h  60ms   8          967      28       0-22       127      (1490) 40ms
894      1383 h  43ms   11         967      28       0-15       133      (2731) 22ms

NetQual                            UDPST (RFC 9097)             gores
UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap    RPM    DelLD
21        327 m  183ms  8ms        22 (51)  28       0-253      12       
21        413 m  145ms  8          22 (43)  28       0-255      15       
22        273 m  220ms  6          23 (53)  28       0-259      10       
21        377 m  159ms  8          23 (51)  28       0-250      10       
22        281 m  214ms  11         23 (52)  28       0-250       6       

These tests were conducted in a round-robin fashion to minimize the possibility of network variations between measurements:
NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat 

NetQual indicates the same reduced capacity in Downstream when compared to UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer capacity, dep. on VLAN tag).
Upstream capacities are not very different (a factor that made TCP methods more viable many years ago when most consumer access speeds were limited to 10's of Megabits).

gores reports significantly lower capacity in both downstream and upstream measurements, a factor of 7 less than NetQual for downstream. Interestingly, the reduced capacity (taken as the working load) results in higher responsiveness: RPM meas are higher and loaded delays are lower for downstream.

The comparison of networkQuality and goresponsiveness is somewhat confounded by the need to use the Apple server infrastructure for both these methods (the documentation provides this option - thanks!). I don't have admin access to our server at the moment. But the measured differences are large despite the confounding factor. 

goresponsiveness has its own, very different output format than networkQuality. There isn't a comparable "-v" option other than -debug (which is extremely detailed). gores only reports RPM for the downstream.

I hope that these results will prompt more of the principle evangelists and coders to weigh-in.

It's also worth noting that the RTTrange reported by UDPST is the range above the minimum RTT, and represents the entire 10 second test. The consistency of the maximum of the range (~255ms) seems to indicate that UDPST has characterized the length of the upstream buffer during measurements.

I'm sure there is more to observe that is prompted by these measurements; comments welcome!

Al


> -----Original Message-----
> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> Sent: Tuesday, November 1, 2022 10:51 AM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> Hi Dave,
> Thanks for trying UDPST (RFC 9097)!
> 
> Something you might try with starlink:
> use the -X option and UDPST will generate random payloads.
> 
> The measurements with -X will reflect the uncompressed that are possible.
> I tried this on a ship-board Internet access: uncompressed rate was 100kbps.
> 
> A few more quick replies below,
> Al
> 
> > -----Original Message-----
> > From: Dave Taht <dave.taht@gmail.com>
> > Sent: Tuesday, November 1, 2022 12:22 AM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> > Dear Al:
> >
> > OK, I took your udpst tool for a spin.
> >
> > NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> > good feature set, VERY fast,
> [acm]
> Len Ciavattone (my partner in crime on several measurement projects) is the
> lead coder: he has implemented many measurement tools extremely efficiently,
> this one in C-lang.
> 
> > and in very brief testing, seemed
> > to be accurate in the starlink case, though it's hard to tell with
> > them as the rate changes every 15s.
> [acm]
> Great! Our guiding principle developing UDPST has been to test the accuracy of
> measurements against a ground-truth. It pays-off.
> 
> >
> > I filed a couple bug reports on trivial stuff:
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > SswH_opYmEw$
> [acm]
> much appreciated...  We have an OB-UDPST project meeting this Friday, can
> discuss then.
> 
> >
> > (Adding diffserv and ecn washing or marking detection would be a nice
> > feature to have)
> >
> > Aside from the sheer joy coming from the "it compiles! and runs!"
> > phase I haven't looked much further.
> >
> > I left a copy running on one of my starlink testbeds -
> > fremont.starlink.taht.net - if anyone wants to try it. It's
> > instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> > version from bob, but close), and now udpst, and good to about a gbit.
> >
> > nice tool!
> [acm]
> Thanks again!
> 
> >
> > Has anyone here played with crusader? (
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$   )
> >
> > On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> > >
> > > > > have you tried irtt?
> >
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$   )
> > > > I have not. Seems like a reasonable tool for UDP testing. The feature I
> > didn't like in my scan of the documentation is the use of Inter-packet delay
> > variation (IPDV) instead of packet delay variation (PDV): variation from the
> > minimum (or reference) delay. The morbidly curious can find my analysis in
> RFC
> > 5481:
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > T7QSlc$
> > >
> > > irtt was meant to simulate high speed voip and one day
> > > videoconferencing. Please inspect the json output
> > > for other metrics. Due to OS limits it is typically only accurate to a
> > > 3ms interval. One thing it does admirably is begin to expose the
> > > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > > technologies, as well as request/grant systems like cable and gpon,
> > > especially when otherwise idle.
> > >
> > > Here is a highres plot of starlink's behaviors from last year:
> > > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> > bandwidth-
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > >
> > > clearly showing them "optimizing for bandwidth" and changing next sat
> > > hop, and about a 40ms interval of buffering between these switches.
> > > I'd published elsewhere, if anyone cares, a preliminary study of what
> > > starlink's default behaviors did to cubic and BBR...
> > >
> > > >
> > > > irtt's use of IPDV means that the results won’t compare with UDPST, and
> > possibly networkQuality. But I may give it a try anyway...
> > >
> > > The more the merrier! Someday the "right" metrics will arrive.
> > >
> > > As a side note, this paper focuses on RAN uplink latency
> > >
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHgvqerjg$   which I think
> > > is a major barrier to most forms of 5G actually achieving good
> > > performance in a FPS game, if it is true for more RANs. I'd like more
> > > to be testing uplink latencies idle and with load, on all
> > > technologies.
> > >
> > > >
> > > > thanks again, Dave.
> > > > Al
> > > >
> > > > > -----Original Message-----
> > > > > From: Dave Taht <dave.taht@gmail.com>
> > > > > Sent: Monday, October 31, 2022 12:52 PM
> > > > > To: MORTON JR., AL <acmorton@att.com>
> > > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > > metrics
> > > > >
> > > > > Thank you very much for the steer to RFC9097. I'd completely missed
> > that.
> > > > >
> > > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> wrote:
> > > > > >
> > > > > > (astute readers may have guessed that I pressed "send" too soon on
> > previous
> > > > > message...)
> > > > > >
> > > > > > I also conducted upstream tests this time, here are the results:
> > > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High
> and
> > > > > Medium)
> > > > > >
> > > > > > Net Qual                           UDPST (RFC9097)
> Ookla
> > > > > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
> UpCap
> > > > > Ping(no load)
> > > > > > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
> > 8
> > > > > > 22         281 m 214ms  8ms        27 (52)  25       5-248      22
> > 8
> > > > > > 22         290 m 207ms  8ms        27 (55)  28       0-253      22
> > 9
> > > > > > 21         330 m 182ms  11ms       23 (44)  28       0-255      22
> > 7
> > > > > > 22         334 m 180ms  9ms        33 (56)  25       0-255      22
> > 9
> > > > > >
> > > > > > The Upstream capacity measurements reflect an interesting feature
> that
> > we
> > > > > can reliably and repeatably measure with UDPST. The first ~3 seconds
> of
> > > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> this
> > > > > behavior in its 1 second sub-interval measurements and has a bimodal
> > reporting
> > > > > option that divides the complete measurement interval in two time
> > intervals to
> > > > > report an initial (turbo) max capacity and a steady-state max capacity
> > for the
> > > > > later intervals. The UDPST capacity results present both measurements:
> > steady-
> > > > > state first.
> > > > >
> > > > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > > > one thing, the "speedboost" concept remains popular, except that it's
> > > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > > > it is often configured "sneakily", in that it doesn't kick in for the
> > > > > typical observed duration of the test, for some, they cut the
> > > > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > > > >
> > > > > One of my biggest issues with the rpm spec so far is that it should,
> > > > > at least, sometimes, run randomly longer than the overly short
> > > > > interval it runs for and the tools also allow for manual override of
> > length.
> > > > >
> > > > > we caught a lot of tomfoolery with flent's rrul test running by
> default
> > for
> > > > > 1m.
> > > > >
> > > > > Also, AQMs on the path can take a while to find the optimal drop or
> mark
> > rate.
> > > > >
> > > > > >
> > > > > > The capacity processing in networkQuality and Ookla appear to report
> > the
> > > > > steady-state result.
> > > > >
> > > > > Ookla used to basically report the last result. Also it's not a good
> > > > > indicator of web traffic behavior at all, watching the curve
> > > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > > > (40ms)....
> > > > >
> > > > > So adding another mode - how quickly is peak bandwidth actually
> > > > > reached, would be nice.
> > > > >
> > > > > I haven't poked into the current iteration of the goresponsiveness
> > > > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > > >
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > > > > would be good to try collecting more statistics and histograms and
> > > > > methods of analyzing the data in that libre-source version.
> > > > >
> > > > > How does networkQuality compare vs a vs your tool vs a vs
> > goresponsiveness?
> > > > >
> > > > > >I watched the upstream capacity measurements on the Ookla app, and
> > could
> > > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> > most of
> > > > > the test which determined the final result.
> > > > >
> > > > > I tend to get upset when I see ookla's new test flash a peak result in
> > > > > the seconds and then settle on some lower number somehow.
> > > > > So far as I know they are only sampling the latency every 250ms.
> > > > >
> > > > > >
> > > > > > The working latency is about 200ms in networkQuality and about 280ms
> > as
> > > > > measured by UDPST (RFC9097). Note that the networkQuality minimum
> delay
> > is
> > > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> > difference
> > > > > in working latency.  Also, we used the very dynamic Type C load
> > > > > adjustment/search algorithm in UDPST during all of this testing, which
> > could
> > > > > explain the higher working latency to some degree.
> > > > > >
> > > > > > So, it's worth noting that the measurements needed for assessing
> > working
> > > > > latency/responsiveness are available in the UDPST utility, and that
> the
> > UDPST
> > > > > measurements are conducted on UDP transport (used by a growing
> fraction
> > of
> > > > > Internet traffic).
> > > > >
> > > > > Thx, didn't know of this work til now!
> > > > >
> > > > > have you tried irtt?
> > > > >
> > > > > >
> > > > > > comments welcome of course,
> > > > > > Al
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > > > To: ippm@ietf.org
> > > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > > > Latency"
> > > > > > > metrics
> > > > > > >
> > > > > > >
> > > > > > > Hi again RPM friends and IPPM'ers,
> > > > > > >
> > > > > > > As promised, I repeated the tests shared last week, this time
> using
> > both
> > > > > the
> > > > > > > verbose (-v) and sequential (-s) dwn/up test options of
> > networkQuality. I
> > > > > > > followed Sebastian's calculations as well.
> > > > > > >
> > > > > > > Working Latency & Capacity Summary
> > > > > > >
> > > > > > > Net Qual                           UDPST
> > Ookla
> > > > > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
> > DnCap
> > > > > > > Ping(no load)
> > > > > > > 885       916 m  66ms   8ms        970      28       0-20
> 940
> > 8
> > > > > > > 888      1355 h  44ms   8ms        966      28       0-23
> 940
> > 8
> > > > > > > 891      1109 h  54ms   8ms        968      27       0-19
> 940
> > 9
> > > > > > > 887      1141 h  53ms   11ms       966      27       0-18
> 937
> > 7
> > > > > > > 884      1151 h  52ms   9ms        968      28       0-20
> 937
> > 9
> > > > > > >
> > > > > > > With the sequential test option, I noticed that networkQuality
> > achieved
> > > > > nearly
> > > > > > > the maximum capacity reported almost immediately at the start of a
> > test.
> > > > > > > However, the reported capacities are low by about 60Mbps,
> especially
> > when
> > > > > > > compared to the Ookla TCP measurements.
> > > > > > >
> > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> high
> > end of
> > > > > the
> > > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > > > networkQuality
> > > > > > > RPM measurements were categorized as "High". There doesn't seem to
> > be much
> > > > > > > buffering in the downstream direction.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > > > To: ippm@ietf.org
> > > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > > > > > metrics
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi RPM friends and IPPM'ers,
> > > > > > > >
> > > > > > > > I was wondering what a comparison of some of the "working
> latency"
> > > > > metrics
> > > > > > > > would look like, so I ran some tests using a service on DOCSIS
> > 3.1, with
> > > > > the
> > > > > > > > downlink provisioned for 1Gbps.
> > > > > > > >
> > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
> > Ookla
> > > > > > > Speedtest
> > > > > > > > with as similar connectivity as possible (but we know that the
> > traffic
> > > > > will
> > > > > > > > diverge to different servers and we can't change that aspect).
> > > > > > > >
> > > > > > > > Here's a quick summary of yesterday's results:
> > > > > > > >
> > > > > > > > Working Latency & Capacity Summary
> > > > > > > >
> > > > > > > > Net Qual                UDPST                        Ookla
> > > > > > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
> > Ping(no
> > > > > load)
> > > > > > > > 878       62            970      28       0-19       941      6
> > > > > > > > 891       92            970      27       0-20       940      7
> > > > > > > > 891       120           966      28       0-22       937      9
> > > > > > > > 890       112           970      28       0-21       940      8
> > > > > > > > 903       70            970      28       0-16       935      9
> > > > > > > >
> > > > > > > > Note: all RPM values were categorized as Low.
> > > > > > > >
> > > > > > > > networkQuality downstream capacities are always on the low side
> > compared
> > > > > to
> > > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
> > what
> > > > > Ookla
> > > > > > > > achieved. I think that a longer test duration might be needed to
> > achieve
> > > > > the
> > > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
> > observed
> > > > > were
> > > > > > > > certainly headed in the right direction. (I recently upgraded to
> > > > > Monterey
> > > > > > > 12.6
> > > > > > > > on my MacBook, so should have the latest version.)
> > > > > > > >
> > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
> > should
> > > > > have
> > > > > > > > run the tests with the -v option to help with comparisons. I'll
> > repeat
> > > > > this
> > > > > > > > test when I can make time.
> > > > > > > >
> > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during
> the
> > test)
> > > > > and
> > > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
> > very
> > > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
> > this
> > > > > access
> > > > > > > and
> > > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
> > Chicago
> > > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
> > measurements
> > > > > were
> > > > > > > run
> > > > > > > > on an Ubuntu VM in my MacBook.
> > > > > > > >
> > > > > > > > The big disappointment was that the Ookla desktop app I updated
> > over the
> > > > > > > > weekend did not include the new responsiveness metric! I
> included
> > the
> > > > > ping
> > > > > > > > results anyway, and it was clearly using a server in the nearby
> > area.
> > > > > > > >
> > > > > > > > So, I have some more work to do, but I hope this is interesting-
> > enough
> > > > > to
> > > > > > > > start some comparison discussions, and bring-out some
> suggestions.
> > > > > > > >
> > > > > > > > happy testing all,
> > > > > > > > Al
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > ippm mailing list
> > > > > > > > ippm@ietf.org
> > > > > > > >
> > > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > > >
> > > > > > >
> > > > >
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ippm mailing list
> > > > > > > ippm@ietf.org
> > > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > > T!g-
> > > > >
> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > >
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > > > KXU$
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > This song goes out to all the folk that thought Stadia would work:
> > > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > mushroom-
> > > > > song-activity-6981366665607352320-
> > > > >
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > > > c48iQFub34zz4iE$
> > > > > Dave Täht CEO, TekLibre, LLC
> > >
> > >
> > >
> > > --
> > > This song goes out to all the folk that thought Stadia would work:
> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > mushroom-song-activity-6981366665607352320-
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHLHDpSWs$
> > > Dave Täht CEO, TekLibre, LLC
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-
> > song-activity-6981366665607352320-
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHLHDpSWs$
> > Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[rjmcmahon]

Any iperf 2 data? https://sourceforge.net/projects/iperf2/

Bob
> Hi all,
> 
> I have been working through the threads on misery metrics and
> lightweight sensing of bw and buffering, and with those insights and
> gold-nuggets in mind, I'm iterating through testing the combinations
> of tools that Dave That and Bob McMahon suggested.
> 
> earlier this week, Dave wrote:
>> How does networkQuality compare vs a vs your tool vs a vs 
>> goresponsiveness?
> 
> goresponsiveness installed flawlessly - very nice instructions and
> getting-started info.
> 
> Comparison of NetQual (networkQuality -vs), UDPST -A c, 
> gores/goresponsiveness
> 
> Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps
> down-stream service
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and 
> Medium)
> 
> NetQual                            UDPST (RFC 9097)             gores
> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
>   RPM    DelLD
> 882       788 m  76ms   8ms        967      28       0-16       127
>   (1382) 43ms
> 892      1036 h  58ms   8          966      27       0-18       128
>   (2124) 28ms
> 887      1304 h  46ms   6          969      27       0-18       130
>   (1478) 41ms
> 885      1008 h  60ms   8          967      28       0-22       127
>   (1490) 40ms
> 894      1383 h  43ms   11         967      28       0-15       133
>   (2731) 22ms
> 
> NetQual                            UDPST (RFC 9097)             gores
> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
>   RPM    DelLD
> 21        327 m  183ms  8ms        22 (51)  28       0-253      12
> 21        413 m  145ms  8          22 (43)  28       0-255      15
> 22        273 m  220ms  6          23 (53)  28       0-259      10
> 21        377 m  159ms  8          23 (51)  28       0-250      10
> 22        281 m  214ms  11         23 (52)  28       0-250       6
> 
> These tests were conducted in a round-robin fashion to minimize the
> possibility of network variations between measurements:
> NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - 
> repeat
> 
> NetQual indicates the same reduced capacity in Downstream when
> compared to UDPST (940Mbps is the max for TCP payloads, while 967-970
> is max for IP-layer capacity, dep. on VLAN tag).
> Upstream capacities are not very different (a factor that made TCP
> methods more viable many years ago when most consumer access speeds
> were limited to 10's of Megabits).
> 
> gores reports significantly lower capacity in both downstream and
> upstream measurements, a factor of 7 less than NetQual for downstream.
> Interestingly, the reduced capacity (taken as the working load)
> results in higher responsiveness: RPM meas are higher and loaded
> delays are lower for downstream.
> 
> The comparison of networkQuality and goresponsiveness is somewhat
> confounded by the need to use the Apple server infrastructure for both
> these methods (the documentation provides this option - thanks!). I
> don't have admin access to our server at the moment. But the measured
> differences are large despite the confounding factor.
> 
> goresponsiveness has its own, very different output format than
> networkQuality. There isn't a comparable "-v" option other than -debug
> (which is extremely detailed). gores only reports RPM for the
> downstream.
> 
> I hope that these results will prompt more of the principle
> evangelists and coders to weigh-in.
> 
> It's also worth noting that the RTTrange reported by UDPST is the
> range above the minimum RTT, and represents the entire 10 second test.
> The consistency of the maximum of the range (~255ms) seems to indicate
> that UDPST has characterized the length of the upstream buffer during
> measurements.
> 
> I'm sure there is more to observe that is prompted by these
> measurements; comments welcome!
> 
> Al
> 
> 
>> -----Original Message-----
>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> Sent: Tuesday, November 1, 2022 10:51 AM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> Subject: Re: [ippm] Preliminary measurement comparison of "Working 
>> Latency"
>> metrics
>> 
>> *** Security Advisory: This Message Originated Outside of AT&T ***.
>> Reference http://cso.att.com/EmailSecurity/IDSP.html for more 
>> information.
>> 
>> Hi Dave,
>> Thanks for trying UDPST (RFC 9097)!
>> 
>> Something you might try with starlink:
>> use the -X option and UDPST will generate random payloads.
>> 
>> The measurements with -X will reflect the uncompressed that are 
>> possible.
>> I tried this on a ship-board Internet access: uncompressed rate was 
>> 100kbps.
>> 
>> A few more quick replies below,
>> Al
>> 
>> > -----Original Message-----
>> > From: Dave Taht <dave.taht@gmail.com>
>> > Sent: Tuesday, November 1, 2022 12:22 AM
>> > To: MORTON JR., AL <acmorton@att.com>
>> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> > metrics
>> >
>> > Dear Al:
>> >
>> > OK, I took your udpst tool for a spin.
>> >
>> > NICE! 120k binary (I STILL work on machines with only 4MB of flash),
>> > good feature set, VERY fast,
>> [acm]
>> Len Ciavattone (my partner in crime on several measurement projects) 
>> is the
>> lead coder: he has implemented many measurement tools extremely 
>> efficiently,
>> this one in C-lang.
>> 
>> > and in very brief testing, seemed
>> > to be accurate in the starlink case, though it's hard to tell with
>> > them as the rate changes every 15s.
>> [acm]
>> Great! Our guiding principle developing UDPST has been to test the 
>> accuracy of
>> measurements against a ground-truth. It pays-off.
>> 
>> >
>> > I filed a couple bug reports on trivial stuff:
>> >
>> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
>> >
>> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
>> > SswH_opYmEw$
>> [acm]
>> much appreciated...  We have an OB-UDPST project meeting this Friday, 
>> can
>> discuss then.
>> 
>> >
>> > (Adding diffserv and ecn washing or marking detection would be a nice
>> > feature to have)
>> >
>> > Aside from the sheer joy coming from the "it compiles! and runs!"
>> > phase I haven't looked much further.
>> >
>> > I left a copy running on one of my starlink testbeds -
>> > fremont.starlink.taht.net - if anyone wants to try it. It's
>> > instrumented with netperf, flent, irtt, iperf2 (not quite the latest
>> > version from bob, but close), and now udpst, and good to about a gbit.
>> >
>> > nice tool!
>> [acm]
>> Thanks again!
>> 
>> >
>> > Has anyone here played with crusader? (
>> >
>> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
>> > oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$   )
>> >
>> > On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>> > >
>> > > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>> > >
>> > > > > have you tried irtt?
>> >
>> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
>> > H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$   )
>> > > > I have not. Seems like a reasonable tool for UDP testing. The feature I
>> > didn't like in my scan of the documentation is the use of Inter-packet delay
>> > variation (IPDV) instead of packet delay variation (PDV): variation from the
>> > minimum (or reference) delay. The morbidly curious can find my analysis in
>> RFC
>> > 5481:
>> >
>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
>> >
>> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
>> > T7QSlc$
>> > >
>> > > irtt was meant to simulate high speed voip and one day
>> > > videoconferencing. Please inspect the json output
>> > > for other metrics. Due to OS limits it is typically only accurate to a
>> > > 3ms interval. One thing it does admirably is begin to expose the
>> > > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
>> > > technologies, as well as request/grant systems like cable and gpon,
>> > > especially when otherwise idle.
>> > >
>> > > Here is a highres plot of starlink's behaviors from last year:
>> > > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
>> > bandwidth-
>> >
>> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
>> > KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
>> > >
>> > > clearly showing them "optimizing for bandwidth" and changing next sat
>> > > hop, and about a 40ms interval of buffering between these switches.
>> > > I'd published elsewhere, if anyone cares, a preliminary study of what
>> > > starlink's default behaviors did to cubic and BBR...
>> > >
>> > > >
>> > > > irtt's use of IPDV means that the results won’t compare with UDPST, and
>> > possibly networkQuality. But I may give it a try anyway...
>> > >
>> > > The more the merrier! Someday the "right" metrics will arrive.
>> > >
>> > > As a side note, this paper focuses on RAN uplink latency
>> > >
>> >
>> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
>> >
>> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHgvqerjg$   which I think
>> > > is a major barrier to most forms of 5G actually achieving good
>> > > performance in a FPS game, if it is true for more RANs. I'd like more
>> > > to be testing uplink latencies idle and with load, on all
>> > > technologies.
>> > >
>> > > >
>> > > > thanks again, Dave.
>> > > > Al
>> > > >
>> > > > > -----Original Message-----
>> > > > > From: Dave Taht <dave.taht@gmail.com>
>> > > > > Sent: Monday, October 31, 2022 12:52 PM
>> > > > > To: MORTON JR., AL <acmorton@att.com>
>> > > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
>> > Latency"
>> > > > > metrics
>> > > > >
>> > > > > Thank you very much for the steer to RFC9097. I'd completely missed
>> > that.
>> > > > >
>> > > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
>> wrote:
>> > > > > >
>> > > > > > (astute readers may have guessed that I pressed "send" too soon on
>> > previous
>> > > > > message...)
>> > > > > >
>> > > > > > I also conducted upstream tests this time, here are the results:
>> > > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High
>> and
>> > > > > Medium)
>> > > > > >
>> > > > > > Net Qual                           UDPST (RFC9097)
>> Ookla
>> > > > > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
>> UpCap
>> > > > > Ping(no load)
>> > > > > > 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
>> > 8
>> > > > > > 22         281 m 214ms  8ms        27 (52)  25       5-248      22
>> > 8
>> > > > > > 22         290 m 207ms  8ms        27 (55)  28       0-253      22
>> > 9
>> > > > > > 21         330 m 182ms  11ms       23 (44)  28       0-255      22
>> > 7
>> > > > > > 22         334 m 180ms  9ms        33 (56)  25       0-255      22
>> > 9
>> > > > > >
>> > > > > > The Upstream capacity measurements reflect an interesting feature
>> that
>> > we
>> > > > > can reliably and repeatably measure with UDPST. The first ~3 seconds
>> of
>> > > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
>> this
>> > > > > behavior in its 1 second sub-interval measurements and has a bimodal
>> > reporting
>> > > > > option that divides the complete measurement interval in two time
>> > intervals to
>> > > > > report an initial (turbo) max capacity and a steady-state max capacity
>> > for the
>> > > > > later intervals. The UDPST capacity results present both measurements:
>> > steady-
>> > > > > state first.
>> > > > >
>> > > > > Certainly we can expect bi-model distributions from many ISPs, as, for
>> > > > > one thing, the "speedboost" concept remains popular, except that it's
>> > > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
>> > > > > it is often configured "sneakily", in that it doesn't kick in for the
>> > > > > typical observed duration of the test, for some, they cut the
>> > > > > available bandwidth about 20s in, others, 1 or 5 minutes.
>> > > > >
>> > > > > One of my biggest issues with the rpm spec so far is that it should,
>> > > > > at least, sometimes, run randomly longer than the overly short
>> > > > > interval it runs for and the tools also allow for manual override of
>> > length.
>> > > > >
>> > > > > we caught a lot of tomfoolery with flent's rrul test running by
>> default
>> > for
>> > > > > 1m.
>> > > > >
>> > > > > Also, AQMs on the path can take a while to find the optimal drop or
>> mark
>> > rate.
>> > > > >
>> > > > > >
>> > > > > > The capacity processing in networkQuality and Ookla appear to report
>> > the
>> > > > > steady-state result.
>> > > > >
>> > > > > Ookla used to basically report the last result. Also it's not a good
>> > > > > indicator of web traffic behavior at all, watching the curve
>> > > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
>> > > > > (40ms)....
>> > > > >
>> > > > > So adding another mode - how quickly is peak bandwidth actually
>> > > > > reached, would be nice.
>> > > > >
>> > > > > I haven't poked into the current iteration of the goresponsiveness
>> > > > > test at all: https://urldefense.com/v3/__https://github.com/network-
>> > > > >
>> >
>> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
>> > > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
>> > > > > would be good to try collecting more statistics and histograms and
>> > > > > methods of analyzing the data in that libre-source version.
>> > > > >
>> > > > > How does networkQuality compare vs a vs your tool vs a vs
>> > goresponsiveness?
>> > > > >
>> > > > > >I watched the upstream capacity measurements on the Ookla app, and
>> > could
>> > > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
>> > most of
>> > > > > the test which determined the final result.
>> > > > >
>> > > > > I tend to get upset when I see ookla's new test flash a peak result in
>> > > > > the seconds and then settle on some lower number somehow.
>> > > > > So far as I know they are only sampling the latency every 250ms.
>> > > > >
>> > > > > >
>> > > > > > The working latency is about 200ms in networkQuality and about 280ms
>> > as
>> > > > > measured by UDPST (RFC9097). Note that the networkQuality minimum
>> delay
>> > is
>> > > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
>> > difference
>> > > > > in working latency.  Also, we used the very dynamic Type C load
>> > > > > adjustment/search algorithm in UDPST during all of this testing, which
>> > could
>> > > > > explain the higher working latency to some degree.
>> > > > > >
>> > > > > > So, it's worth noting that the measurements needed for assessing
>> > working
>> > > > > latency/responsiveness are available in the UDPST utility, and that
>> the
>> > UDPST
>> > > > > measurements are conducted on UDP transport (used by a growing
>> fraction
>> > of
>> > > > > Internet traffic).
>> > > > >
>> > > > > Thx, didn't know of this work til now!
>> > > > >
>> > > > > have you tried irtt?
>> > > > >
>> > > > > >
>> > > > > > comments welcome of course,
>> > > > > > Al
>> > > > > >
>> > > > > > > -----Original Message-----
>> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > > > > > Sent: Sunday, October 30, 2022 8:09 PM
>> > > > > > > To: ippm@ietf.org
>> > > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
>> > > > > Latency"
>> > > > > > > metrics
>> > > > > > >
>> > > > > > >
>> > > > > > > Hi again RPM friends and IPPM'ers,
>> > > > > > >
>> > > > > > > As promised, I repeated the tests shared last week, this time
>> using
>> > both
>> > > > > the
>> > > > > > > verbose (-v) and sequential (-s) dwn/up test options of
>> > networkQuality. I
>> > > > > > > followed Sebastian's calculations as well.
>> > > > > > >
>> > > > > > > Working Latency & Capacity Summary
>> > > > > > >
>> > > > > > > Net Qual                           UDPST
>> > Ookla
>> > > > > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
>> > DnCap
>> > > > > > > Ping(no load)
>> > > > > > > 885       916 m  66ms   8ms        970      28       0-20
>> 940
>> > 8
>> > > > > > > 888      1355 h  44ms   8ms        966      28       0-23
>> 940
>> > 8
>> > > > > > > 891      1109 h  54ms   8ms        968      27       0-19
>> 940
>> > 9
>> > > > > > > 887      1141 h  53ms   11ms       966      27       0-18
>> 937
>> > 7
>> > > > > > > 884      1151 h  52ms   9ms        968      28       0-20
>> 937
>> > 9
>> > > > > > >
>> > > > > > > With the sequential test option, I noticed that networkQuality
>> > achieved
>> > > > > nearly
>> > > > > > > the maximum capacity reported almost immediately at the start of a
>> > test.
>> > > > > > > However, the reported capacities are low by about 60Mbps,
>> especially
>> > when
>> > > > > > > compared to the Ookla TCP measurements.
>> > > > > > >
>> > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
>> high
>> > end of
>> > > > > the
>> > > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
>> > > > > networkQuality
>> > > > > > > RPM measurements were categorized as "High". There doesn't seem to
>> > be much
>> > > > > > > buffering in the downstream direction.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > -----Original Message-----
>> > > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > > > > > > Sent: Monday, October 24, 2022 6:36 PM
>> > > > > > > > To: ippm@ietf.org
>> > > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
>> > Latency"
>> > > > > > > > metrics
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Hi RPM friends and IPPM'ers,
>> > > > > > > >
>> > > > > > > > I was wondering what a comparison of some of the "working
>> latency"
>> > > > > metrics
>> > > > > > > > would look like, so I ran some tests using a service on DOCSIS
>> > 3.1, with
>> > > > > the
>> > > > > > > > downlink provisioned for 1Gbps.
>> > > > > > > >
>> > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
>> > Ookla
>> > > > > > > Speedtest
>> > > > > > > > with as similar connectivity as possible (but we know that the
>> > traffic
>> > > > > will
>> > > > > > > > diverge to different servers and we can't change that aspect).
>> > > > > > > >
>> > > > > > > > Here's a quick summary of yesterday's results:
>> > > > > > > >
>> > > > > > > > Working Latency & Capacity Summary
>> > > > > > > >
>> > > > > > > > Net Qual                UDPST                        Ookla
>> > > > > > > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
>> > Ping(no
>> > > > > load)
>> > > > > > > > 878       62            970      28       0-19       941      6
>> > > > > > > > 891       92            970      27       0-20       940      7
>> > > > > > > > 891       120           966      28       0-22       937      9
>> > > > > > > > 890       112           970      28       0-21       940      8
>> > > > > > > > 903       70            970      28       0-16       935      9
>> > > > > > > >
>> > > > > > > > Note: all RPM values were categorized as Low.
>> > > > > > > >
>> > > > > > > > networkQuality downstream capacities are always on the low side
>> > compared
>> > > > > to
>> > > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
>> > what
>> > > > > Ookla
>> > > > > > > > achieved. I think that a longer test duration might be needed to
>> > achieve
>> > > > > the
>> > > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
>> > observed
>> > > > > were
>> > > > > > > > certainly headed in the right direction. (I recently upgraded to
>> > > > > Monterey
>> > > > > > > 12.6
>> > > > > > > > on my MacBook, so should have the latest version.)
>> > > > > > > >
>> > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
>> > should
>> > > > > have
>> > > > > > > > run the tests with the -v option to help with comparisons. I'll
>> > repeat
>> > > > > this
>> > > > > > > > test when I can make time.
>> > > > > > > >
>> > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during
>> the
>> > test)
>> > > > > and
>> > > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
>> > very
>> > > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
>> > this
>> > > > > access
>> > > > > > > and
>> > > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
>> > Chicago
>> > > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
>> > measurements
>> > > > > were
>> > > > > > > run
>> > > > > > > > on an Ubuntu VM in my MacBook.
>> > > > > > > >
>> > > > > > > > The big disappointment was that the Ookla desktop app I updated
>> > over the
>> > > > > > > > weekend did not include the new responsiveness metric! I
>> included
>> > the
>> > > > > ping
>> > > > > > > > results anyway, and it was clearly using a server in the nearby
>> > area.
>> > > > > > > >
>> > > > > > > > So, I have some more work to do, but I hope this is interesting-
>> > enough
>> > > > > to
>> > > > > > > > start some comparison discussions, and bring-out some
>> suggestions.
>> > > > > > > >
>> > > > > > > > happy testing all,
>> > > > > > > > Al
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > _______________________________________________
>> > > > > > > > ippm mailing list
>> > > > > > > > ippm@ietf.org
>> > > > > > > >
>> > > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > > > > >
>> > > > > > >
>> > > > >
>> >
>> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>> > > > > > >
>> > > > > > > _______________________________________________
>> > > > > > > ippm mailing list
>> > > > > > > ippm@ietf.org
>> > > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > > > > T!g-
>> > > > >
>> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>> > > > > >
>> > > > > > _______________________________________________
>> > > > > > ippm mailing list
>> > > > > > ippm@ietf.org
>> > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > >
>> >
>> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
>> > > > > KXU$
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > This song goes out to all the folk that thought Stadia would work:
>> > > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> > mushroom-
>> > > > > song-activity-6981366665607352320-
>> > > > >
>> >
>> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
>> > > > > c48iQFub34zz4iE$
>> > > > > Dave Täht CEO, TekLibre, LLC
>> > >
>> > >
>> > >
>> > > --
>> > > This song goes out to all the folk that thought Stadia would work:
>> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> > mushroom-song-activity-6981366665607352320-
>> >
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHLHDpSWs$
>> > > Dave Täht CEO, TekLibre, LLC
>> >
>> >
>> >
>> > --
>> > This song goes out to all the folk that thought Stadia would work:
>> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> mushroom-
>> > song-activity-6981366665607352320-
>> >
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHLHDpSWs$
>> > Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

> -----Original Message-----
> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
> Sent: Friday, November 4, 2022 2:12 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: Dave Taht <dave.taht@gmail.com>; ippm@ietf.org; Rpm
> <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
> 
> Any iperf 2 data?
> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
> jm0H0ICmGbrmtAj1pqCwuDPP8s6l40wS3hZvt7i5KLoRmNgT9UXRigC0rIfnQk6NjodSBgtUaBP7yL
> coVw$
> 
[acm] 
Not yet, I hope to make some progress this weekend with the other tools...
Al
 

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[Sebastian Moeller]

Hi Al,


> On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Hi all,
> 
> I have been working through the threads on misery metrics and lightweight sensing of bw and buffering, and with those insights and gold-nuggets in mind, I'm iterating through testing the combinations of tools that Dave That and Bob McMahon suggested.
> 
> earlier this week, Dave wrote:
>> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> 
> goresponsiveness installed flawlessly - very nice instructions and getting-started info.
> 
> Comparison of NetQual (networkQuality -vs), UDPST -A c, gores/goresponsiveness
> 
> Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-stream service
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
> 
> NetQual                            UDPST (RFC 9097)             gores
> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap    RPM    DelLD
> 882       788 m  76ms   8ms        967      28       0-16       127      (1382) 43ms 
> 892      1036 h  58ms   8          966      27       0-18       128      (2124) 28ms
> 887      1304 h  46ms   6          969      27       0-18       130      (1478) 41ms
> 885      1008 h  60ms   8          967      28       0-22       127      (1490) 40ms
> 894      1383 h  43ms   11         967      28       0-15       133      (2731) 22ms
> 
> NetQual                            UDPST (RFC 9097)             gores
> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap    RPM    DelLD
> 21        327 m  183ms  8ms        22 (51)  28       0-253      12       
> 21        413 m  145ms  8          22 (43)  28       0-255      15       
> 22        273 m  220ms  6          23 (53)  28       0-259      10       
> 21        377 m  159ms  8          23 (51)  28       0-250      10       
> 22        281 m  214ms  11         23 (52)  28       0-250       6       
> 
> These tests were conducted in a round-robin fashion to minimize the possibility of network variations between measurements:
> NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat 
> 
> NetQual indicates the same reduced capacity in Downstream when compared to UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer capacity, dep. on VLAN tag).

	[SM] DOCSIS ISPs traditionally provision higher peak rates than they advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the 1050 Mbps range. But typically gigabit ethernet limits the practically achievable throughput somewhat:

Ethernet payload rate: 			1000 * ((1500)/(1500 + 38)) = 			 975.29 Mbps
Ethernet payload rate +VLAN: 		1000 * ((1500)/(1500 + 38 + 4)) = 		 972.76 Mbps
IPv4 payload (ethernet+VLAN): 		1000 * ((1500 - 20)/(1500 + 38 + 4)) = 		 959.79 Mbps
IPv6 payload (ethernet+VLAN): 		1000 * ((1500 - 40)/(1500 + 38 + 4)) = 		 946.82 Mbps
IPv4/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 20)/(1500 + 38 + 4)) = 	 946.82 Mbps
IPv6/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 40)/(1500 + 38 + 4)) = 	 933.85 Mbps
IPv4/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 20)/(1500 + 38 + 4)) = 939.04 Mbps
IPv6/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 40)/(1500 + 38 + 4)) = 926.07 Mbps


Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS, but on-line speedtest almost never return the simple average rate over the measurement duration, but play "clever" tricks to exclude the start-up phase and to aggregate over multiple flows that invariably ending with results that tend to exceed the hard limits shown above...


> Upstream capacities are not very different (a factor that made TCP methods more viable many years ago when most consumer access speeds were limited to 10's of Megabits).

	[SM] My take on this is that this is partly due to the goal of ramping up very quickly and get away with a short measurement duration. That causes imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I often ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and for timecourses to look at and reason about).

> gores reports significantly lower capacity in both downstream and upstream measurements, a factor of 7 less than NetQual for downstream. Interestingly, the reduced capacity (taken as the working load) results in higher responsiveness: RPM meas are higher and loaded delays are lower for downstream.

	[SM] Yepp, if you only fill-up the queue partially you will harvest less queueing delay and hence retain more responsiveness, albeit this really just seems to be better interpreted as go-responsiveness failing to achieve "working-conditions". 

I tend to run gores like this:
time ./networkQuality --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config  --sattimeout 60 --extended-stats --logger-filename go_networkQuality_$(date +%Y%m%d_%H%M%S)

--sattimeout 60 extends the time out for the saturation measurement somewhat (before I saw it simply failing on a 1 Gbps access link, it did give some diagnostic message though).


> 
> The comparison of networkQuality and goresponsiveness is somewhat confounded by the need to use the Apple server infrastructure for both these methods (the documentation provides this option - thanks!). I don't have admin access to our server at the moment. But the measured differences are large despite the confounding factor. 

	[SM] Puzzled, I thought when comparing the two ntworkQuality variants having the same back-end sort of helps be reducing the differences to the clients? But comparison to UDPST suffers somewhat.

> 
> goresponsiveness has its own, very different output format than networkQuality. There isn't a comparable "-v" option other than -debug (which is extremely detailed). gores only reports RPM for the downstream.

	[SM] I agree it would be nice if gores would grow a sequential mode as well.


> I hope that these results will prompt more of the principle evangelists and coders to weigh-in.

	[SM] No luck ;) but at least "amateur hour" (aka me) is ready to discuss.

> 
> It's also worth noting that the RTTrange reported by UDPST is the range above the minimum RTT, and represents the entire 10 second test. The consistency of the maximum of the range (~255ms) seems to indicate that UDPST has characterized the length of the upstream buffer during measurements.

	[SM] thanks for explaining that.

Regards
	Sebastian

> 
> I'm sure there is more to observe that is prompted by these measurements; comments welcome!
> 
> Al
> 
> 
>> -----Original Message-----
>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> Sent: Tuesday, November 1, 2022 10:51 AM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> metrics
>> 
>> *** Security Advisory: This Message Originated Outside of AT&T ***.
>> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
>> 
>> Hi Dave,
>> Thanks for trying UDPST (RFC 9097)!
>> 
>> Something you might try with starlink:
>> use the -X option and UDPST will generate random payloads.
>> 
>> The measurements with -X will reflect the uncompressed that are possible.
>> I tried this on a ship-board Internet access: uncompressed rate was 100kbps.
>> 
>> A few more quick replies below,
>> Al
>> 
>>> -----Original Message-----
>>> From: Dave Taht <dave.taht@gmail.com>
>>> Sent: Tuesday, November 1, 2022 12:22 AM
>>> To: MORTON JR., AL <acmorton@att.com>
>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>>> metrics
>>> 
>>> Dear Al:
>>> 
>>> OK, I took your udpst tool for a spin.
>>> 
>>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
>>> good feature set, VERY fast,
>> [acm]
>> Len Ciavattone (my partner in crime on several measurement projects) is the
>> lead coder: he has implemented many measurement tools extremely efficiently,
>> this one in C-lang.
>> 
>>> and in very brief testing, seemed
>>> to be accurate in the starlink case, though it's hard to tell with
>>> them as the rate changes every 15s.
>> [acm]
>> Great! Our guiding principle developing UDPST has been to test the accuracy of
>> measurements against a ground-truth. It pays-off.
>> 
>>> 
>>> I filed a couple bug reports on trivial stuff:
>>> 
>> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
>>> 
>> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
>>> SswH_opYmEw$
>> [acm]
>> much appreciated...  We have an OB-UDPST project meeting this Friday, can
>> discuss then.
>> 
>>> 
>>> (Adding diffserv and ecn washing or marking detection would be a nice
>>> feature to have)
>>> 
>>> Aside from the sheer joy coming from the "it compiles! and runs!"
>>> phase I haven't looked much further.
>>> 
>>> I left a copy running on one of my starlink testbeds -
>>> fremont.starlink.taht.net - if anyone wants to try it. It's
>>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
>>> version from bob, but close), and now udpst, and good to about a gbit.
>>> 
>>> nice tool!
>> [acm]
>> Thanks again!
>> 
>>> 
>>> Has anyone here played with crusader? (
>>> 
>> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
>>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$   )
>>> 
>>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>>>> 
>>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>>>> 
>>>>>> have you tried irtt?
>>> 
>> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
>>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$   )
>>>>> I have not. Seems like a reasonable tool for UDP testing. The feature I
>>> didn't like in my scan of the documentation is the use of Inter-packet delay
>>> variation (IPDV) instead of packet delay variation (PDV): variation from the
>>> minimum (or reference) delay. The morbidly curious can find my analysis in
>> RFC
>>> 5481:
>>> 
>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
>>> 
>> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
>>> T7QSlc$
>>>> 
>>>> irtt was meant to simulate high speed voip and one day
>>>> videoconferencing. Please inspect the json output
>>>> for other metrics. Due to OS limits it is typically only accurate to a
>>>> 3ms interval. One thing it does admirably is begin to expose the
>>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
>>>> technologies, as well as request/grant systems like cable and gpon,
>>>> especially when otherwise idle.
>>>> 
>>>> Here is a highres plot of starlink's behaviors from last year:
>>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
>>> bandwidth-
>>> 
>> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
>>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
>>>> 
>>>> clearly showing them "optimizing for bandwidth" and changing next sat
>>>> hop, and about a 40ms interval of buffering between these switches.
>>>> I'd published elsewhere, if anyone cares, a preliminary study of what
>>>> starlink's default behaviors did to cubic and BBR...
>>>> 
>>>>> 
>>>>> irtt's use of IPDV means that the results won’t compare with UDPST, and
>>> possibly networkQuality. But I may give it a try anyway...
>>>> 
>>>> The more the merrier! Someday the "right" metrics will arrive.
>>>> 
>>>> As a side note, this paper focuses on RAN uplink latency
>>>> 
>>> 
>> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
>>> 
>> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHgvqerjg$   which I think
>>>> is a major barrier to most forms of 5G actually achieving good
>>>> performance in a FPS game, if it is true for more RANs. I'd like more
>>>> to be testing uplink latencies idle and with load, on all
>>>> technologies.
>>>> 
>>>>> 
>>>>> thanks again, Dave.
>>>>> Al
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Dave Taht <dave.taht@gmail.com>
>>>>>> Sent: Monday, October 31, 2022 12:52 PM
>>>>>> To: MORTON JR., AL <acmorton@att.com>
>>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
>>> Latency"
>>>>>> metrics
>>>>>> 
>>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
>>> that.
>>>>>> 
>>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
>> wrote:
>>>>>>> 
>>>>>>> (astute readers may have guessed that I pressed "send" too soon on
>>> previous
>>>>>> message...)
>>>>>>> 
>>>>>>> I also conducted upstream tests this time, here are the results:
>>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
>> and
>>>>>> Medium)
>>>>>>> 
>>>>>>> Net Qual                           UDPST (RFC9097)
>> Ookla
>>>>>>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
>> UpCap
>>>>>> Ping(no load)
>>>>>>> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
>>> 8
>>>>>>> 22         281 m 214ms  8ms        27 (52)  25       5-248      22
>>> 8
>>>>>>> 22         290 m 207ms  8ms        27 (55)  28       0-253      22
>>> 9
>>>>>>> 21         330 m 182ms  11ms       23 (44)  28       0-255      22
>>> 7
>>>>>>> 22         334 m 180ms  9ms        33 (56)  25       0-255      22
>>> 9
>>>>>>> 
>>>>>>> The Upstream capacity measurements reflect an interesting feature
>> that
>>> we
>>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
>> of
>>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
>> this
>>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
>>> reporting
>>>>>> option that divides the complete measurement interval in two time
>>> intervals to
>>>>>> report an initial (turbo) max capacity and a steady-state max capacity
>>> for the
>>>>>> later intervals. The UDPST capacity results present both measurements:
>>> steady-
>>>>>> state first.
>>>>>> 
>>>>>> Certainly we can expect bi-model distributions from many ISPs, as, for
>>>>>> one thing, the "speedboost" concept remains popular, except that it's
>>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
>>>>>> it is often configured "sneakily", in that it doesn't kick in for the
>>>>>> typical observed duration of the test, for some, they cut the
>>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
>>>>>> 
>>>>>> One of my biggest issues with the rpm spec so far is that it should,
>>>>>> at least, sometimes, run randomly longer than the overly short
>>>>>> interval it runs for and the tools also allow for manual override of
>>> length.
>>>>>> 
>>>>>> we caught a lot of tomfoolery with flent's rrul test running by
>> default
>>> for
>>>>>> 1m.
>>>>>> 
>>>>>> Also, AQMs on the path can take a while to find the optimal drop or
>> mark
>>> rate.
>>>>>> 
>>>>>>> 
>>>>>>> The capacity processing in networkQuality and Ookla appear to report
>>> the
>>>>>> steady-state result.
>>>>>> 
>>>>>> Ookla used to basically report the last result. Also it's not a good
>>>>>> indicator of web traffic behavior at all, watching the curve
>>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
>>>>>> (40ms)....
>>>>>> 
>>>>>> So adding another mode - how quickly is peak bandwidth actually
>>>>>> reached, would be nice.
>>>>>> 
>>>>>> I haven't poked into the current iteration of the goresponsiveness
>>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
>>>>>> 
>>> 
>> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
>>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
>>>>>> would be good to try collecting more statistics and histograms and
>>>>>> methods of analyzing the data in that libre-source version.
>>>>>> 
>>>>>> How does networkQuality compare vs a vs your tool vs a vs
>>> goresponsiveness?
>>>>>> 
>>>>>>> I watched the upstream capacity measurements on the Ookla app, and
>>> could
>>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
>>> most of
>>>>>> the test which determined the final result.
>>>>>> 
>>>>>> I tend to get upset when I see ookla's new test flash a peak result in
>>>>>> the seconds and then settle on some lower number somehow.
>>>>>> So far as I know they are only sampling the latency every 250ms.
>>>>>> 
>>>>>>> 
>>>>>>> The working latency is about 200ms in networkQuality and about 280ms
>>> as
>>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
>> delay
>>> is
>>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
>>> difference
>>>>>> in working latency.  Also, we used the very dynamic Type C load
>>>>>> adjustment/search algorithm in UDPST during all of this testing, which
>>> could
>>>>>> explain the higher working latency to some degree.
>>>>>>> 
>>>>>>> So, it's worth noting that the measurements needed for assessing
>>> working
>>>>>> latency/responsiveness are available in the UDPST utility, and that
>> the
>>> UDPST
>>>>>> measurements are conducted on UDP transport (used by a growing
>> fraction
>>> of
>>>>>> Internet traffic).
>>>>>> 
>>>>>> Thx, didn't know of this work til now!
>>>>>> 
>>>>>> have you tried irtt?
>>>>>> 
>>>>>>> 
>>>>>>> comments welcome of course,
>>>>>>> Al
>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
>>>>>>>> To: ippm@ietf.org
>>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
>>>>>> Latency"
>>>>>>>> metrics
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hi again RPM friends and IPPM'ers,
>>>>>>>> 
>>>>>>>> As promised, I repeated the tests shared last week, this time
>> using
>>> both
>>>>>> the
>>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
>>> networkQuality. I
>>>>>>>> followed Sebastian's calculations as well.
>>>>>>>> 
>>>>>>>> Working Latency & Capacity Summary
>>>>>>>> 
>>>>>>>> Net Qual                           UDPST
>>> Ookla
>>>>>>>> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
>>> DnCap
>>>>>>>> Ping(no load)
>>>>>>>> 885       916 m  66ms   8ms        970      28       0-20
>> 940
>>> 8
>>>>>>>> 888      1355 h  44ms   8ms        966      28       0-23
>> 940
>>> 8
>>>>>>>> 891      1109 h  54ms   8ms        968      27       0-19
>> 940
>>> 9
>>>>>>>> 887      1141 h  53ms   11ms       966      27       0-18
>> 937
>>> 7
>>>>>>>> 884      1151 h  52ms   9ms        968      28       0-20
>> 937
>>> 9
>>>>>>>> 
>>>>>>>> With the sequential test option, I noticed that networkQuality
>>> achieved
>>>>>> nearly
>>>>>>>> the maximum capacity reported almost immediately at the start of a
>>> test.
>>>>>>>> However, the reported capacities are low by about 60Mbps,
>> especially
>>> when
>>>>>>>> compared to the Ookla TCP measurements.
>>>>>>>> 
>>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
>> high
>>> end of
>>>>>> the
>>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
>>>>>> networkQuality
>>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
>>> be much
>>>>>>>> buffering in the downstream direction.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
>>>>>>>>> To: ippm@ietf.org
>>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
>>> Latency"
>>>>>>>>> metrics
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi RPM friends and IPPM'ers,
>>>>>>>>> 
>>>>>>>>> I was wondering what a comparison of some of the "working
>> latency"
>>>>>> metrics
>>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
>>> 3.1, with
>>>>>> the
>>>>>>>>> downlink provisioned for 1Gbps.
>>>>>>>>> 
>>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
>>> Ookla
>>>>>>>> Speedtest
>>>>>>>>> with as similar connectivity as possible (but we know that the
>>> traffic
>>>>>> will
>>>>>>>>> diverge to different servers and we can't change that aspect).
>>>>>>>>> 
>>>>>>>>> Here's a quick summary of yesterday's results:
>>>>>>>>> 
>>>>>>>>> Working Latency & Capacity Summary
>>>>>>>>> 
>>>>>>>>> Net Qual                UDPST                        Ookla
>>>>>>>>> DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
>>> Ping(no
>>>>>> load)
>>>>>>>>> 878       62            970      28       0-19       941      6
>>>>>>>>> 891       92            970      27       0-20       940      7
>>>>>>>>> 891       120           966      28       0-22       937      9
>>>>>>>>> 890       112           970      28       0-21       940      8
>>>>>>>>> 903       70            970      28       0-16       935      9
>>>>>>>>> 
>>>>>>>>> Note: all RPM values were categorized as Low.
>>>>>>>>> 
>>>>>>>>> networkQuality downstream capacities are always on the low side
>>> compared
>>>>>> to
>>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
>>> what
>>>>>> Ookla
>>>>>>>>> achieved. I think that a longer test duration might be needed to
>>> achieve
>>>>>> the
>>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
>>> observed
>>>>>> were
>>>>>>>>> certainly headed in the right direction. (I recently upgraded to
>>>>>> Monterey
>>>>>>>> 12.6
>>>>>>>>> on my MacBook, so should have the latest version.)
>>>>>>>>> 
>>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
>>> should
>>>>>> have
>>>>>>>>> run the tests with the -v option to help with comparisons. I'll
>>> repeat
>>>>>> this
>>>>>>>>> test when I can make time.
>>>>>>>>> 
>>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
>> the
>>> test)
>>>>>> and
>>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
>>> very
>>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
>>> this
>>>>>> access
>>>>>>>> and
>>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
>>> Chicago
>>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
>>> measurements
>>>>>> were
>>>>>>>> run
>>>>>>>>> on an Ubuntu VM in my MacBook.
>>>>>>>>> 
>>>>>>>>> The big disappointment was that the Ookla desktop app I updated
>>> over the
>>>>>>>>> weekend did not include the new responsiveness metric! I
>> included
>>> the
>>>>>> ping
>>>>>>>>> results anyway, and it was clearly using a server in the nearby
>>> area.
>>>>>>>>> 
>>>>>>>>> So, I have some more work to do, but I hope this is interesting-
>>> enough
>>>>>> to
>>>>>>>>> start some comparison discussions, and bring-out some
>> suggestions.
>>>>>>>>> 
>>>>>>>>> happy testing all,
>>>>>>>>> Al
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> ippm mailing list
>>>>>>>>> ippm@ietf.org
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>> 
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>> 
>> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> ippm mailing list
>>>>>>>> ippm@ietf.org
>>>>>>>> 
>>>>>> 
>>> 
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>>>> T!g-
>>>>>> 
>>> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> ippm mailing list
>>>>>>> ippm@ietf.org
>>>>>>> 
>>>>>> 
>>> 
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>> 
>>> 
>> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
>>>>>> KXU$
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> This song goes out to all the folk that thought Stadia would work:
>>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>>> mushroom-
>>>>>> song-activity-6981366665607352320-
>>>>>> 
>>> 
>> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
>>>>>> c48iQFub34zz4iE$
>>>>>> Dave Täht CEO, TekLibre, LLC
>>>> 
>>>> 
>>>> 
>>>> --
>>>> This song goes out to all the folk that thought Stadia would work:
>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>>> mushroom-song-activity-6981366665607352320-
>>> 
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHLHDpSWs$
>>>> Dave Täht CEO, TekLibre, LLC
>>> 
>>> 
>>> 
>>> --
>>> This song goes out to all the folk that thought Stadia would work:
>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> mushroom-
>>> song-activity-6981366665607352320-
>>> 
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHLHDpSWs$
>>> Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

Hi Sebastian, thanks for your comments and information.
I'll reply to a few key points in this top-post.

Sebastian wrote:
> 	[SM] DOCSIS ISPs traditionally provision higher peak rates than they
> advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> 1050 Mbps range. But typically gigabit ethernet limits the practically
> achievable throughput somewhat:
[acm]
Right, my ISP's CPE has Gig-E ports so I won't see any benefit from over-rate provisioning.

Sebastian wrote:
...
> IPv4/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 20)/(1500 + 38 + 4)) = 939.04 Mbps
...
> Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS,
> but on-line speedtest almost never return the simple average rate over the
> measurement duration, but play "clever" tricks to exclude the start-up phase
> and to aggregate over multiple flows that invariably ending with results that
> tend to exceed the hard limits shown above...
[acm]
My measurements with the Ookla desktop app on MacOS (shared earlier this week) are very consistent at ~940Mbps Dn-stream, so timestamps calculation sounds right.
My ISP specifies their service speed at 940Mbps, as though they assume subscribers will measure using Ookla or other TCP tool. In fact, our team hasn't seen such consistent results from Ookla or any TCP-based test in the 1Gbps range, and this makes me wonder if there might be some test-recognition here.

FYI - UDPST uses a max packet size with is sub-MTU, to avoid fragmentation when various encapsulations are encountered. Also, the definition of IP-Layer Capacity in the various standards (e.g., RFC 9097) includes the bits in the IP header in the total Capacity.

So, instead of:
> Ethernet payload rate +VLAN: 		1000 * ((1500)/(1500 + 38 + 4)) =  972.76 Mbps
We have
Ethernet payload rate +VLAN: 		1000 * ((1222)/(1222 + 38 + 4)) =  966.77 Mbps
and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps

UDPST has an option to use datagrams for the traditional 1500 octet MTU (-T), but a user could cause a lot of fragmentation on links with encapsulations wit this option.

acm wrote:
> > The comparison of networkQuality and goresponsiveness is somewhat confounded
> > by the need to use the Apple server infrastructure for both these methods ...
Sebastian wrote:
> 	[SM] Puzzled, I thought when comparing the two networkQuality variants
> having the same back-end sort of helps be reducing the differences to the
> clients? But comparison to UDPST suffers somewhat.
[acm]
Yes, I was hoping to match-up the client and server implementations. Instead, this might be more of a Client-X and Server-Y interoperability test (and could explain some results?), but that was not to be. 

Thanks for your suggested command line for gores.

IIRC one of your early messages, Sebastian, your MacOS indicates that 12.6 is the latest. It's the same for me: my sufficiently powerful MacBook cannot upgrade to Ventura for the latest in networkQuality versions. It would help us who are doing some testing if the latest version of networkQuality could be made installable for us, somehow... 

thanks again and regards,
Al


> -----Original Message-----
> From: Sebastian Moeller <moeller0@gmx.de>
> Sent: Friday, November 4, 2022 3:10 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm
> <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <hawkinsw@obs.cr>
> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> Latency" metrics
> 
> Hi Al,
> 
> 
> > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm <rpm@lists.bufferbloat.net>
> wrote:
> >
> > Hi all,
> >
> > I have been working through the threads on misery metrics and lightweight
> sensing of bw and buffering, and with those insights and gold-nuggets in mind,
> I'm iterating through testing the combinations of tools that Dave That and Bob
> McMahon suggested.
> >
> > earlier this week, Dave wrote:
> >> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> >
> > goresponsiveness installed flawlessly - very nice instructions and getting-
> started info.
> >
> > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> gores/goresponsiveness
> >
> > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-
> stream service
> > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
> >
> > NetQual                            UDPST (RFC 9097)             gores
> > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap    RPM
> DelLD
> > 882       788 m  76ms   8ms        967      28       0-16       127
> (1382) 43ms
> > 892      1036 h  58ms   8          966      27       0-18       128
> (2124) 28ms
> > 887      1304 h  46ms   6          969      27       0-18       130
> (1478) 41ms
> > 885      1008 h  60ms   8          967      28       0-22       127
> (1490) 40ms
> > 894      1383 h  43ms   11         967      28       0-15       133
> (2731) 22ms
> >
> > NetQual                            UDPST (RFC 9097)             gores
> > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap    RPM
> DelLD
> > 21        327 m  183ms  8ms        22 (51)  28       0-253      12
> > 21        413 m  145ms  8          22 (43)  28       0-255      15
> > 22        273 m  220ms  6          23 (53)  28       0-259      10
> > 21        377 m  159ms  8          23 (51)  28       0-250      10
> > 22        281 m  214ms  11         23 (52)  28       0-250       6
> >
> > These tests were conducted in a round-robin fashion to minimize the
> possibility of network variations between measurements:
> > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
> >
> > NetQual indicates the same reduced capacity in Downstream when compared to
> UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer
> capacity, dep. on VLAN tag).
> 
> 	[SM] DOCSIS ISPs traditionally provision higher peak rates than they
> advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> 1050 Mbps range. But typically gigabit ethernet limits the practically
> achievable throughput somewhat:
> 
> Ethernet payload rate: 			1000 * ((1500)/(1500 + 38)) =
> 	 975.29 Mbps
> Ethernet payload rate +VLAN: 		1000 * ((1500)/(1500 + 38 + 4)) =
> 	 972.76 Mbps
> IPv4 payload (ethernet+VLAN): 		1000 * ((1500 - 20)/(1500 + 38 + 4)) =
> 		 959.79 Mbps
> IPv6 payload (ethernet+VLAN): 		1000 * ((1500 - 40)/(1500 + 38 + 4)) =
> 		 946.82 Mbps
> IPv4/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 20)/(1500 + 38 +
> 4)) = 	 946.82 Mbps
> IPv6/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 40)/(1500 + 38 +
> 4)) = 	 933.85 Mbps
> IPv4/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 20)/(1500 +
> 38 + 4)) = 939.04 Mbps
> IPv6/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 40)/(1500 +
> 38 + 4)) = 926.07 Mbps
> 
> 
> Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS,
> but on-line speedtest almost never return the simple average rate over the
> measurement duration, but play "clever" tricks to exclude the start-up phase
> and to aggregate over multiple flows that invariably ending with results that
> tend to exceed the hard limits shown above...
> 
> 
> > Upstream capacities are not very different (a factor that made TCP methods
> more viable many years ago when most consumer access speeds were limited to
> 10's of Megabits).
> 
> 	[SM] My take on this is that this is partly due to the goal of ramping
> up very quickly and get away with a short measurement duration. That causes
> imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I often
> ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and for
> timecourses to look at and reason about).
> 
> > gores reports significantly lower capacity in both downstream and upstream
> measurements, a factor of 7 less than NetQual for downstream. Interestingly,
> the reduced capacity (taken as the working load) results in higher
> responsiveness: RPM meas are higher and loaded delays are lower for
> downstream.
> 
> 	[SM] Yepp, if you only fill-up the queue partially you will harvest less
> queueing delay and hence retain more responsiveness, albeit this really just
> seems to be better interpreted as go-responsiveness failing to achieve
> "working-conditions".
> 
> I tend to run gores like this:
> time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> /api/v1/gm/config  --sattimeout 60 --extended-stats --logger-filename
> go_networkQuality_$(date +%Y%m%d_%H%M%S)
> 
> --sattimeout 60 extends the time out for the saturation measurement somewhat
> (before I saw it simply failing on a 1 Gbps access link, it did give some
> diagnostic message though).
> 
> 
> >
> > The comparison of networkQuality and goresponsiveness is somewhat confounded
> by the need to use the Apple server infrastructure for both these methods (the
> documentation provides this option - thanks!). I don't have admin access to
> our server at the moment. But the measured differences are large despite the
> confounding factor.
> 
> 	[SM] Puzzled, I thought when comparing the two ntworkQuality variants
> having the same back-end sort of helps be reducing the differences to the
> clients? But comparison to UDPST suffers somewhat.
> 
> >
> > goresponsiveness has its own, very different output format than
> networkQuality. There isn't a comparable "-v" option other than -debug (which
> is extremely detailed). gores only reports RPM for the downstream.
> 
> 	[SM] I agree it would be nice if gores would grow a sequential mode as
> well.
> 
> 
> > I hope that these results will prompt more of the principle evangelists and
> coders to weigh-in.
> 
> 	[SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> discuss.
> 
> >
> > It's also worth noting that the RTTrange reported by UDPST is the range
> above the minimum RTT, and represents the entire 10 second test. The
> consistency of the maximum of the range (~255ms) seems to indicate that UDPST
> has characterized the length of the upstream buffer during measurements.
> 
> 	[SM] thanks for explaining that.
> 
> Regards
> 	Sebastian
> 
> >
> > I'm sure there is more to observe that is prompted by these measurements;
> comments welcome!
> >
> > Al
> >
> >
> >> -----Original Message-----
> >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >> Sent: Tuesday, November 1, 2022 10:51 AM
> >> To: Dave Taht <dave.taht@gmail.com>
> >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> >> metrics
> >>
> >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> >>
> >> Hi Dave,
> >> Thanks for trying UDPST (RFC 9097)!
> >>
> >> Something you might try with starlink:
> >> use the -X option and UDPST will generate random payloads.
> >>
> >> The measurements with -X will reflect the uncompressed that are possible.
> >> I tried this on a ship-board Internet access: uncompressed rate was
> 100kbps.
> >>
> >> A few more quick replies below,
> >> Al
> >>
> >>> -----Original Message-----
> >>> From: Dave Taht <dave.taht@gmail.com>
> >>> Sent: Tuesday, November 1, 2022 12:22 AM
> >>> To: MORTON JR., AL <acmorton@att.com>
> >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> >>> metrics
> >>>
> >>> Dear Al:
> >>>
> >>> OK, I took your udpst tool for a spin.
> >>>
> >>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> >>> good feature set, VERY fast,
> >> [acm]
> >> Len Ciavattone (my partner in crime on several measurement projects) is the
> >> lead coder: he has implemented many measurement tools extremely
> efficiently,
> >> this one in C-lang.
> >>
> >>> and in very brief testing, seemed
> >>> to be accurate in the starlink case, though it's hard to tell with
> >>> them as the rate changes every 15s.
> >> [acm]
> >> Great! Our guiding principle developing UDPST has been to test the accuracy
> of
> >> measurements against a ground-truth. It pays-off.
> >>
> >>>
> >>> I filed a couple bug reports on trivial stuff:
> >>>
> >>
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> >>>
> >>
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> >>> SswH_opYmEw$
> >> [acm]
> >> much appreciated...  We have an OB-UDPST project meeting this Friday, can
> >> discuss then.
> >>
> >>>
> >>> (Adding diffserv and ecn washing or marking detection would be a nice
> >>> feature to have)
> >>>
> >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> >>> phase I haven't looked much further.
> >>>
> >>> I left a copy running on one of my starlink testbeds -
> >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> >>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> >>> version from bob, but close), and now udpst, and good to about a gbit.
> >>>
> >>> nice tool!
> >> [acm]
> >> Thanks again!
> >>
> >>>
> >>> Has anyone here played with crusader? (
> >>>
> >>
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> >>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> )
> >>>
> >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> >>>>
> >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> >>>>
> >>>>>> have you tried irtt?
> >>>
> >>
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> >>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> )
> >>>>> I have not. Seems like a reasonable tool for UDP testing. The feature I
> >>> didn't like in my scan of the documentation is the use of Inter-packet
> delay
> >>> variation (IPDV) instead of packet delay variation (PDV): variation from
> the
> >>> minimum (or reference) delay. The morbidly curious can find my analysis in
> >> RFC
> >>> 5481:
> >>>
> >>
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> >>>
> >>
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> >>> T7QSlc$
> >>>>
> >>>> irtt was meant to simulate high speed voip and one day
> >>>> videoconferencing. Please inspect the json output
> >>>> for other metrics. Due to OS limits it is typically only accurate to a
> >>>> 3ms interval. One thing it does admirably is begin to expose the
> >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> >>>> technologies, as well as request/grant systems like cable and gpon,
> >>>> especially when otherwise idle.
> >>>>
> >>>> Here is a highres plot of starlink's behaviors from last year:
> >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> >>> bandwidth-
> >>>
> >>
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> >>>>
> >>>> clearly showing them "optimizing for bandwidth" and changing next sat
> >>>> hop, and about a 40ms interval of buffering between these switches.
> >>>> I'd published elsewhere, if anyone cares, a preliminary study of what
> >>>> starlink's default behaviors did to cubic and BBR...
> >>>>
> >>>>>
> >>>>> irtt's use of IPDV means that the results won’t compare with UDPST, and
> >>> possibly networkQuality. But I may give it a try anyway...
> >>>>
> >>>> The more the merrier! Someday the "right" metrics will arrive.
> >>>>
> >>>> As a side note, this paper focuses on RAN uplink latency
> >>>>
> >>>
> >>
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> >>>
> >>
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHgvqerjg$   which I think
> >>>> is a major barrier to most forms of 5G actually achieving good
> >>>> performance in a FPS game, if it is true for more RANs. I'd like more
> >>>> to be testing uplink latencies idle and with load, on all
> >>>> technologies.
> >>>>
> >>>>>
> >>>>> thanks again, Dave.
> >>>>> Al
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Dave Taht <dave.taht@gmail.com>
> >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> >>>>>> To: MORTON JR., AL <acmorton@att.com>
> >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> >>> Latency"
> >>>>>> metrics
> >>>>>>
> >>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
> >>> that.
> >>>>>>
> >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> >> wrote:
> >>>>>>>
> >>>>>>> (astute readers may have guessed that I pressed "send" too soon on
> >>> previous
> >>>>>> message...)
> >>>>>>>
> >>>>>>> I also conducted upstream tests this time, here are the results:
> >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
> >> and
> >>>>>> Medium)
> >>>>>>>
> >>>>>>> Net Qual                           UDPST (RFC9097)
> >> Ookla
> >>>>>>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
> >> UpCap
> >>>>>> Ping(no load)
> >>>>>>> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
> >>> 8
> >>>>>>> 22         281 m 214ms  8ms        27 (52)  25       5-248      22
> >>> 8
> >>>>>>> 22         290 m 207ms  8ms        27 (55)  28       0-253      22
> >>> 9
> >>>>>>> 21         330 m 182ms  11ms       23 (44)  28       0-255      22
> >>> 7
> >>>>>>> 22         334 m 180ms  9ms        33 (56)  25       0-255      22
> >>> 9
> >>>>>>>
> >>>>>>> The Upstream capacity measurements reflect an interesting feature
> >> that
> >>> we
> >>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
> >> of
> >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> >> this
> >>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
> >>> reporting
> >>>>>> option that divides the complete measurement interval in two time
> >>> intervals to
> >>>>>> report an initial (turbo) max capacity and a steady-state max capacity
> >>> for the
> >>>>>> later intervals. The UDPST capacity results present both measurements:
> >>> steady-
> >>>>>> state first.
> >>>>>>
> >>>>>> Certainly we can expect bi-model distributions from many ISPs, as, for
> >>>>>> one thing, the "speedboost" concept remains popular, except that it's
> >>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> >>>>>> it is often configured "sneakily", in that it doesn't kick in for the
> >>>>>> typical observed duration of the test, for some, they cut the
> >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> >>>>>>
> >>>>>> One of my biggest issues with the rpm spec so far is that it should,
> >>>>>> at least, sometimes, run randomly longer than the overly short
> >>>>>> interval it runs for and the tools also allow for manual override of
> >>> length.
> >>>>>>
> >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> >> default
> >>> for
> >>>>>> 1m.
> >>>>>>
> >>>>>> Also, AQMs on the path can take a while to find the optimal drop or
> >> mark
> >>> rate.
> >>>>>>
> >>>>>>>
> >>>>>>> The capacity processing in networkQuality and Ookla appear to report
> >>> the
> >>>>>> steady-state result.
> >>>>>>
> >>>>>> Ookla used to basically report the last result. Also it's not a good
> >>>>>> indicator of web traffic behavior at all, watching the curve
> >>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> >>>>>> (40ms)....
> >>>>>>
> >>>>>> So adding another mode - how quickly is peak bandwidth actually
> >>>>>> reached, would be nice.
> >>>>>>
> >>>>>> I haven't poked into the current iteration of the goresponsiveness
> >>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
> >>>>>>
> >>>
> >>
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> >>>>>> would be good to try collecting more statistics and histograms and
> >>>>>> methods of analyzing the data in that libre-source version.
> >>>>>>
> >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> >>> goresponsiveness?
> >>>>>>
> >>>>>>> I watched the upstream capacity measurements on the Ookla app, and
> >>> could
> >>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> >>> most of
> >>>>>> the test which determined the final result.
> >>>>>>
> >>>>>> I tend to get upset when I see ookla's new test flash a peak result in
> >>>>>> the seconds and then settle on some lower number somehow.
> >>>>>> So far as I know they are only sampling the latency every 250ms.
> >>>>>>
> >>>>>>>
> >>>>>>> The working latency is about 200ms in networkQuality and about 280ms
> >>> as
> >>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
> >> delay
> >>> is
> >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> >>> difference
> >>>>>> in working latency.  Also, we used the very dynamic Type C load
> >>>>>> adjustment/search algorithm in UDPST during all of this testing, which
> >>> could
> >>>>>> explain the higher working latency to some degree.
> >>>>>>>
> >>>>>>> So, it's worth noting that the measurements needed for assessing
> >>> working
> >>>>>> latency/responsiveness are available in the UDPST utility, and that
> >> the
> >>> UDPST
> >>>>>> measurements are conducted on UDP transport (used by a growing
> >> fraction
> >>> of
> >>>>>> Internet traffic).
> >>>>>>
> >>>>>> Thx, didn't know of this work til now!
> >>>>>>
> >>>>>> have you tried irtt?
> >>>>>>
> >>>>>>>
> >>>>>>> comments welcome of course,
> >>>>>>> Al
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> >>>>>>>> To: ippm@ietf.org
> >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> >>>>>> Latency"
> >>>>>>>> metrics
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi again RPM friends and IPPM'ers,
> >>>>>>>>
> >>>>>>>> As promised, I repeated the tests shared last week, this time
> >> using
> >>> both
> >>>>>> the
> >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> >>> networkQuality. I
> >>>>>>>> followed Sebastian's calculations as well.
> >>>>>>>>
> >>>>>>>> Working Latency & Capacity Summary
> >>>>>>>>
> >>>>>>>> Net Qual                           UDPST
> >>> Ookla
> >>>>>>>> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
> >>> DnCap
> >>>>>>>> Ping(no load)
> >>>>>>>> 885       916 m  66ms   8ms        970      28       0-20
> >> 940
> >>> 8
> >>>>>>>> 888      1355 h  44ms   8ms        966      28       0-23
> >> 940
> >>> 8
> >>>>>>>> 891      1109 h  54ms   8ms        968      27       0-19
> >> 940
> >>> 9
> >>>>>>>> 887      1141 h  53ms   11ms       966      27       0-18
> >> 937
> >>> 7
> >>>>>>>> 884      1151 h  52ms   9ms        968      28       0-20
> >> 937
> >>> 9
> >>>>>>>>
> >>>>>>>> With the sequential test option, I noticed that networkQuality
> >>> achieved
> >>>>>> nearly
> >>>>>>>> the maximum capacity reported almost immediately at the start of a
> >>> test.
> >>>>>>>> However, the reported capacities are low by about 60Mbps,
> >> especially
> >>> when
> >>>>>>>> compared to the Ookla TCP measurements.
> >>>>>>>>
> >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> >> high
> >>> end of
> >>>>>> the
> >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
> >>>>>> networkQuality
> >>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
> >>> be much
> >>>>>>>> buffering in the downstream direction.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> >>>>>>>>> To: ippm@ietf.org
> >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
> >>> Latency"
> >>>>>>>>> metrics
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi RPM friends and IPPM'ers,
> >>>>>>>>>
> >>>>>>>>> I was wondering what a comparison of some of the "working
> >> latency"
> >>>>>> metrics
> >>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
> >>> 3.1, with
> >>>>>> the
> >>>>>>>>> downlink provisioned for 1Gbps.
> >>>>>>>>>
> >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
> >>> Ookla
> >>>>>>>> Speedtest
> >>>>>>>>> with as similar connectivity as possible (but we know that the
> >>> traffic
> >>>>>> will
> >>>>>>>>> diverge to different servers and we can't change that aspect).
> >>>>>>>>>
> >>>>>>>>> Here's a quick summary of yesterday's results:
> >>>>>>>>>
> >>>>>>>>> Working Latency & Capacity Summary
> >>>>>>>>>
> >>>>>>>>> Net Qual                UDPST                        Ookla
> >>>>>>>>> DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
> >>> Ping(no
> >>>>>> load)
> >>>>>>>>> 878       62            970      28       0-19       941      6
> >>>>>>>>> 891       92            970      27       0-20       940      7
> >>>>>>>>> 891       120           966      28       0-22       937      9
> >>>>>>>>> 890       112           970      28       0-21       940      8
> >>>>>>>>> 903       70            970      28       0-16       935      9
> >>>>>>>>>
> >>>>>>>>> Note: all RPM values were categorized as Low.
> >>>>>>>>>
> >>>>>>>>> networkQuality downstream capacities are always on the low side
> >>> compared
> >>>>>> to
> >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
> >>> what
> >>>>>> Ookla
> >>>>>>>>> achieved. I think that a longer test duration might be needed to
> >>> achieve
> >>>>>> the
> >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
> >>> observed
> >>>>>> were
> >>>>>>>>> certainly headed in the right direction. (I recently upgraded to
> >>>>>> Monterey
> >>>>>>>> 12.6
> >>>>>>>>> on my MacBook, so should have the latest version.)
> >>>>>>>>>
> >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
> >>> should
> >>>>>> have
> >>>>>>>>> run the tests with the -v option to help with comparisons. I'll
> >>> repeat
> >>>>>> this
> >>>>>>>>> test when I can make time.
> >>>>>>>>>
> >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> >> the
> >>> test)
> >>>>>> and
> >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
> >>> very
> >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
> >>> this
> >>>>>> access
> >>>>>>>> and
> >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
> >>> Chicago
> >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> >>> measurements
> >>>>>> were
> >>>>>>>> run
> >>>>>>>>> on an Ubuntu VM in my MacBook.
> >>>>>>>>>
> >>>>>>>>> The big disappointment was that the Ookla desktop app I updated
> >>> over the
> >>>>>>>>> weekend did not include the new responsiveness metric! I
> >> included
> >>> the
> >>>>>> ping
> >>>>>>>>> results anyway, and it was clearly using a server in the nearby
> >>> area.
> >>>>>>>>>
> >>>>>>>>> So, I have some more work to do, but I hope this is interesting-
> >>> enough
> >>>>>> to
> >>>>>>>>> start some comparison discussions, and bring-out some
> >> suggestions.
> >>>>>>>>>
> >>>>>>>>> happy testing all,
> >>>>>>>>> Al
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> ippm mailing list
> >>>>>>>>> ippm@ietf.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> >>
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> ippm mailing list
> >>>>>>>> ippm@ietf.org
> >>>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>>> T!g-
> >>>>>>
> >>> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> ippm mailing list
> >>>>>>> ippm@ietf.org
> >>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>
> >>>
> >>
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> >>>>>> KXU$
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> This song goes out to all the folk that thought Stadia would work:
> >>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >>> mushroom-
> >>>>>> song-activity-6981366665607352320-
> >>>>>>
> >>>
> >>
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> >>>>>> c48iQFub34zz4iE$
> >>>>>> Dave Täht CEO, TekLibre, LLC
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> This song goes out to all the folk that thought Stadia would work:
> >>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >>> mushroom-song-activity-6981366665607352320-
> >>>
> >>
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHLHDpSWs$
> >>>> Dave Täht CEO, TekLibre, LLC
> >>>
> >>>
> >>>
> >>> --
> >>> This song goes out to all the folk that thought Stadia would work:
> >>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >> mushroom-
> >>> song-activity-6981366665607352320-
> >>>
> >>
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHLHDpSWs$
> >>> Dave Täht CEO, TekLibre, LLC
> >> _______________________________________________
> >> ippm mailing list
> >> ippm@ietf.org
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> rDt9HLI$
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> mERYCyuCd9e9PY3aNLkSw$

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[MORTON JR., AL]

Hi IPPM,

Prior to IETF-115, I shared a series of measurements with the IPPM list. We're looking at responsiveness and working latency with various metrics and multiple testing utilities. This message continues the discussion with new input.

When I first published some measurements, Dave Taht added his assessment and included other relevant email lists in the discussion. I'm continuing to cross-post to all the lists in this thread.

Dave originally suggested that I try a tool called irtt; I've done that now and these are the results.

Bob McMahon: I queued your request to try your iperf2 tool behind the irtt measurements. I hope to make some more measurements this week... 

-=-=-=-=-=-=-=-=-=-=-=-=-=-

We're testing a DOCIS 3.1 based service with 1Gbps down, nominally 22Mbps up. I used wired ETH connected to the DOCSIS modem's switch.

Dave Taht made his server available for the irtt measurements. I installed irtt on a VM in my Macbook, the same VM that runs UDPST.  I ran quite a few tests to become familiar with irtt, so I'll just summarize the relevant ones here.

I ran irtt with its traffic at the maximum allowed on Dave's server. The test duration was 10sec, with packets spaced at 1ms intervals from my VM client. This is the complete output:

./irtt client -i 1ms -l 1250 -d 10s fremont.starlink.taht.net

                         Min     Mean   Median      Max  Stddev
                         ---     ----   ------      ---  ------
                RTT  46.63ms  51.58ms   51.4ms     58ms  1.55ms
         send delay  20.74ms  25.57ms   25.4ms  32.04ms  1.54ms
      receive delay   25.8ms  26.01ms  25.96ms  30.48ms   219µs
                                                               
      IPDV (jitter)   1.03µs   1.15ms   1.02ms   6.87ms   793µs
          send IPDV    176ns   1.13ms    994µs   6.41ms   776µs
       receive IPDV      7ns   79.8µs   41.7µs   4.54ms   140µs
                                                               
     send call time   10.1µs   55.8µs            1.34ms  30.3µs
        timer error     68ns    431µs            6.49ms   490µs
  server proc. time    680ns   2.43µs            47.7µs  1.73µs

                duration: 10.2s (wait 174ms)
   packets sent/received: 7137/7131 (0.08% loss)
 server packets received: 7131/7137 (0.08%/0.00% loss up/down)
     bytes sent/received: 8921250/8913750
       send/receive rate: 7.14 Mbps / 7.13 Mbps
           packet length: 1250 bytes
             timer stats: 2863/10000 (28.63%) missed, 43.10% error
acm@acm-ubuntu1804-1:~/goirtt/irtt$
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
irtt supplies lots of info/stats about the RTT distribution. In the lightly loaded (7.1Mbps) scenario above, the RTT range is about 12 ms. The Mean and the Median are approximately the same.
irtt also supplies Inter-Packet Delay Varition (IPDV), showing that the packets seem to be delayed more than the sending interval occasionally (Max of 6.8ms).
irtt measurements indicate very low packet loss: no congestion at 7Mbps.

For the opposite end of the congestion spectrum, I ran irtt with UDPST (RFC 9097) running in parallel (and using the Type C search algorithm). We pick-up a lot more RTT and wider RTT range in this scenario:

irtt with udpst using Type C search = max load:

                         Min     Mean   Median      Max   Stddev
                         ---     ----   ------      ---   ------
                RTT  47.58ms    118ms  56.53ms  301.6ms  90.28ms
         send delay  24.05ms  94.85ms  33.38ms  278.5ms  90.26ms
      receive delay  22.99ms  23.17ms  23.13ms  25.42ms    156µs
                                                                
      IPDV (jitter)    162ns   1.04ms    733µs   6.36ms   1.02ms
          send IPDV   3.81µs   1.01ms    697µs   6.24ms   1.02ms
       receive IPDV     88ns     93µs   49.8µs   1.48ms    145µs
                                                                
     send call time   4.28µs   39.3µs             903µs   32.4µs
        timer error     86ns    287µs            6.13ms    214µs
  server proc. time    670ns   3.59µs            19.3µs   2.26µs

                duration: 10.9s (wait 904.8ms)
   packets sent/received: 8305/2408 (71.01% loss)
 server packets received: 2408/8305 (71.01%/0.00% loss up/down)
     bytes sent/received: 10381250/3010000
       send/receive rate: 8.31 Mbps / 2.47 Mbps
           packet length: 1250 bytes
             timer stats: 1695/10000 (16.95%) missed, 28.75% error
acm@acm-ubuntu1804-1:~/goirtt/irtt$ 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
The irtt measurement of RTT range is now about 250ms in this maximally-loaded scenario.
One key RTT difference is the Mean delay, which is now twice the Median with working load added by UDPST.
Congestion is evident in the irtt loss measurements = 71%. However, the competing traffic did not break irtt's process or measurements.
UDPST's measurements were ~22Mbps Capacity (steady-state), with high loss, and similar RTT range of 251 ms.


Additional tests included load from UDPST at fixed rates: 14Mbps and 20Mbps. We compare the RTT range for all four conditions in the table below:

Load with irtt   irtt RTT range    UDPST RTT range
==================================================
irtt alone            12ms               ---
UDPST at 14Mbps       11ms               22ms 
UDPST at 20Mbps       14ms               46ms
UDPST at MaxCap      250ms              251ms

The unexpected result with irtt measurements is that the RTT range did not increase with load, where the UDPST RTT range increases with load. We're assuming that the majority of delay increases occur in the DOCSIS upstream queue and both test streams should see similar delay range as they do with maximum load. Perhaps there are some differences owing to the periodic irtt stream and the bursty UDPST stream (both are UDP), but this is speculation.

To check that the test path was operating similarly with earlier tests, we ran a couple of NetworkQuality and UDPST tests as before:

Comparison of NQ-vs, UDPST -A c, same columns as defined earlier.
Working Latency & Capacity Summary (Upstream only)
(capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)

Net Qual                           UDPST (RFC 9097)                     
UpCap     RPM    DelLD  DelMin     UpCap(stable)    RTTmin   RTTrange   
22        276 m  217ms  11ms       23               28       0-252       
22        291 m  206ms  11ms       ~22*             28*      0-251*  

* UDPST test result with the ~7Mbps irtt stream present.     

We found that uplink test results are similar to previous tests, and the new irtt results were collected under similar conditions.

In conclusion, irtt provides a clear summary of the RTT distribution. Minimum, Mean, Median and Max RTT are useful individually and in combinations/comparisons. The irtt measurements compare favorably to those collected during IP-Layer Capacity tests with the UDPST utility.

comments welcome, 
Al

> -----Original Message-----
> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> Sent: Saturday, November 5, 2022 3:37 PM
> To: Sebastian Moeller <moeller0@gmx.de>
> Cc: Rpm <rpm@lists.bufferbloat.net>; Will Hawkins <hawkinsw@obs.cr>;
> ippm@ietf.org
> Subject: Re: [ippm] [Rpm] Preliminary measurement comparison of "Working
> Latency" metrics
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> Hi Sebastian, thanks for your comments and information.
> I'll reply to a few key points in this top-post.
> 
> Sebastian wrote:
> > 	[SM] DOCSIS ISPs traditionally provision higher peak rates than they
> > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > achievable throughput somewhat:
> [acm]
> Right, my ISP's CPE has Gig-E ports so I won't see any benefit from over-rate
> provisioning.
> 
> Sebastian wrote:
> ...
> > IPv4/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 20)/(1500
> + 38 + 4)) = 939.04 Mbps
> ...
> > Speedtests tend to report IPv4/TCP timestamps might be on depending on the
> OS,
> > but on-line speedtest almost never return the simple average rate over the
> > measurement duration, but play "clever" tricks to exclude the start-up phase
> > and to aggregate over multiple flows that invariably ending with results
> that
> > tend to exceed the hard limits shown above...
> [acm]
> My measurements with the Ookla desktop app on MacOS (shared earlier this week)
> are very consistent at ~940Mbps Dn-stream, so timestamps calculation sounds
> right.
> My ISP specifies their service speed at 940Mbps, as though they assume
> subscribers will measure using Ookla or other TCP tool. In fact, our team
> hasn't seen such consistent results from Ookla or any TCP-based test in the
> 1Gbps range, and this makes me wonder if there might be some test-recognition
> here.
> 
> FYI - UDPST uses a max packet size with is sub-MTU, to avoid fragmentation
> when various encapsulations are encountered. Also, the definition of IP-Layer
> Capacity in the various standards (e.g., RFC 9097) includes the bits in the IP
> header in the total Capacity.
> 
> So, instead of:
> > Ethernet payload rate +VLAN: 		1000 * ((1500)/(1500 + 38 + 4)) =
> 972.76 Mbps
> We have
> Ethernet payload rate +VLAN: 		1000 * ((1222)/(1222 + 38 + 4)) =  966.77
> Mbps
> and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps
> 
> UDPST has an option to use datagrams for the traditional 1500 octet MTU (-T),
> but a user could cause a lot of fragmentation on links with encapsulations wit
> this option.
> 
> acm wrote:
> > > The comparison of networkQuality and goresponsiveness is somewhat
> confounded
> > > by the need to use the Apple server infrastructure for both these methods
> ...
> Sebastian wrote:
> > 	[SM] Puzzled, I thought when comparing the two networkQuality variants
> > having the same back-end sort of helps be reducing the differences to the
> > clients? But comparison to UDPST suffers somewhat.
> [acm]
> Yes, I was hoping to match-up the client and server implementations. Instead,
> this might be more of a Client-X and Server-Y interoperability test (and could
> explain some results?), but that was not to be.
> 
> Thanks for your suggested command line for gores.
> 
> IIRC one of your early messages, Sebastian, your MacOS indicates that 12.6 is
> the latest. It's the same for me: my sufficiently powerful MacBook cannot
> upgrade to Ventura for the latest in networkQuality versions. It would help us
> who are doing some testing if the latest version of networkQuality could be
> made installable for us, somehow...
> 
> thanks again and regards,
> Al
> 
> 
> > -----Original Message-----
> > From: Sebastian Moeller <moeller0@gmx.de>
> > Sent: Friday, November 4, 2022 3:10 PM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <rjmcmahon@rjmcmahon.com>;
> Rpm
> > <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <hawkinsw@obs.cr>
> > Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> > Latency" metrics
> >
> > Hi Al,
> >
> >
> > > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm
> <rpm@lists.bufferbloat.net>
> > wrote:
> > >
> > > Hi all,
> > >
> > > I have been working through the threads on misery metrics and lightweight
> > sensing of bw and buffering, and with those insights and gold-nuggets in
> mind,
> > I'm iterating through testing the combinations of tools that Dave That and
> Bob
> > McMahon suggested.
> > >
> > > earlier this week, Dave wrote:
> > >> How does networkQuality compare vs a vs your tool vs a vs
> goresponsiveness?
> > >
> > > goresponsiveness installed flawlessly - very nice instructions and
> getting-
> > started info.
> > >
> > > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> > gores/goresponsiveness
> > >
> > > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-
> > stream service
> > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > Medium)
> > >
> > > NetQual                            UDPST (RFC 9097)             gores
> > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> RPM
> > DelLD
> > > 882       788 m  76ms   8ms        967      28       0-16       127
> > (1382) 43ms
> > > 892      1036 h  58ms   8          966      27       0-18       128
> > (2124) 28ms
> > > 887      1304 h  46ms   6          969      27       0-18       130
> > (1478) 41ms
> > > 885      1008 h  60ms   8          967      28       0-22       127
> > (1490) 40ms
> > > 894      1383 h  43ms   11         967      28       0-15       133
> > (2731) 22ms
> > >
> > > NetQual                            UDPST (RFC 9097)             gores
> > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> RPM
> > DelLD
> > > 21        327 m  183ms  8ms        22 (51)  28       0-253      12
> > > 21        413 m  145ms  8          22 (43)  28       0-255      15
> > > 22        273 m  220ms  6          23 (53)  28       0-259      10
> > > 21        377 m  159ms  8          23 (51)  28       0-250      10
> > > 22        281 m  214ms  11         23 (52)  28       0-250       6
> > >
> > > These tests were conducted in a round-robin fashion to minimize the
> > possibility of network variations between measurements:
> > > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
> > >
> > > NetQual indicates the same reduced capacity in Downstream when compared to
> > UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-
> layer
> > capacity, dep. on VLAN tag).
> >
> > 	[SM] DOCSIS ISPs traditionally provision higher peak rates than they
> > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > achievable throughput somewhat:
> >
> > Ethernet payload rate: 			1000 * ((1500)/(1500 + 38)) =
> > 	 975.29 Mbps
> > Ethernet payload rate +VLAN: 		1000 * ((1500)/(1500 + 38 + 4)) =
> > 	 972.76 Mbps
> > IPv4 payload (ethernet+VLAN): 		1000 * ((1500 - 20)/(1500 + 38 + 4))
> =
> > 		 959.79 Mbps
> > IPv6 payload (ethernet+VLAN): 		1000 * ((1500 - 40)/(1500 + 38 + 4))
> =
> > 		 946.82 Mbps
> > IPv4/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 20)/(1500 + 38
> +
> > 4)) = 	 946.82 Mbps
> > IPv6/TCP payload (ethernet+VLAN): 	1000 * ((1500 - 20 - 40)/(1500 + 38
> +
> > 4)) = 	 933.85 Mbps
> > IPv4/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 20)/(1500
> +
> > 38 + 4)) = 939.04 Mbps
> > IPv6/TCP/RFC1323 timestamps payload: 	1000 * ((1500 - 12 - 20 - 40)/(1500
> +
> > 38 + 4)) = 926.07 Mbps
> >
> >
> > Speedtests tend to report IPv4/TCP timestamps might be on depending on the
> OS,
> > but on-line speedtest almost never return the simple average rate over the
> > measurement duration, but play "clever" tricks to exclude the start-up phase
> > and to aggregate over multiple flows that invariably ending with results
> that
> > tend to exceed the hard limits shown above...
> >
> >
> > > Upstream capacities are not very different (a factor that made TCP methods
> > more viable many years ago when most consumer access speeds were limited to
> > 10's of Megabits).
> >
> > 	[SM] My take on this is that this is partly due to the goal of ramping
> > up very quickly and get away with a short measurement duration. That causes
> > imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I
> often
> > ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and
> for
> > timecourses to look at and reason about).
> >
> > > gores reports significantly lower capacity in both downstream and upstream
> > measurements, a factor of 7 less than NetQual for downstream. Interestingly,
> > the reduced capacity (taken as the working load) results in higher
> > responsiveness: RPM meas are higher and loaded delays are lower for
> > downstream.
> >
> > 	[SM] Yepp, if you only fill-up the queue partially you will harvest less
> > queueing delay and hence retain more responsiveness, albeit this really just
> > seems to be better interpreted as go-responsiveness failing to achieve
> > "working-conditions".
> >
> > I tend to run gores like this:
> > time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> > /api/v1/gm/config  --sattimeout 60 --extended-stats --logger-filename
> > go_networkQuality_$(date +%Y%m%d_%H%M%S)
> >
> > --sattimeout 60 extends the time out for the saturation measurement somewhat
> > (before I saw it simply failing on a 1 Gbps access link, it did give some
> > diagnostic message though).
> >
> >
> > >
> > > The comparison of networkQuality and goresponsiveness is somewhat
> confounded
> > by the need to use the Apple server infrastructure for both these methods
> (the
> > documentation provides this option - thanks!). I don't have admin access to
> > our server at the moment. But the measured differences are large despite the
> > confounding factor.
> >
> > 	[SM] Puzzled, I thought when comparing the two ntworkQuality variants
> > having the same back-end sort of helps be reducing the differences to the
> > clients? But comparison to UDPST suffers somewhat.
> >
> > >
> > > goresponsiveness has its own, very different output format than
> > networkQuality. There isn't a comparable "-v" option other than -debug
> (which
> > is extremely detailed). gores only reports RPM for the downstream.
> >
> > 	[SM] I agree it would be nice if gores would grow a sequential mode as
> > well.
> >
> >
> > > I hope that these results will prompt more of the principle evangelists
> and
> > coders to weigh-in.
> >
> > 	[SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> > discuss.
> >
> > >
> > > It's also worth noting that the RTTrange reported by UDPST is the range
> > above the minimum RTT, and represents the entire 10 second test. The
> > consistency of the maximum of the range (~255ms) seems to indicate that
> UDPST
> > has characterized the length of the upstream buffer during measurements.
> >
> > 	[SM] thanks for explaining that.
> >
> > Regards
> > 	Sebastian
> >
> > >
> > > I'm sure there is more to observe that is prompted by these measurements;
> > comments welcome!
> > >
> > > Al
> > >
> > >
> > >> -----Original Message-----
> > >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >> Sent: Tuesday, November 1, 2022 10:51 AM
> > >> To: Dave Taht <dave.taht@gmail.com>
> > >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > >> metrics
> > >>
> > >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> > >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> > >>
> > >> Hi Dave,
> > >> Thanks for trying UDPST (RFC 9097)!
> > >>
> > >> Something you might try with starlink:
> > >> use the -X option and UDPST will generate random payloads.
> > >>
> > >> The measurements with -X will reflect the uncompressed that are possible.
> > >> I tried this on a ship-board Internet access: uncompressed rate was
> > 100kbps.
> > >>
> > >> A few more quick replies below,
> > >> Al
> > >>
> > >>> -----Original Message-----
> > >>> From: Dave Taht <dave.taht@gmail.com>
> > >>> Sent: Tuesday, November 1, 2022 12:22 AM
> > >>> To: MORTON JR., AL <acmorton@att.com>
> > >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > >>> metrics
> > >>>
> > >>> Dear Al:
> > >>>
> > >>> OK, I took your udpst tool for a spin.
> > >>>
> > >>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> > >>> good feature set, VERY fast,
> > >> [acm]
> > >> Len Ciavattone (my partner in crime on several measurement projects) is
> the
> > >> lead coder: he has implemented many measurement tools extremely
> > efficiently,
> > >> this one in C-lang.
> > >>
> > >>> and in very brief testing, seemed
> > >>> to be accurate in the starlink case, though it's hard to tell with
> > >>> them as the rate changes every 15s.
> > >> [acm]
> > >> Great! Our guiding principle developing UDPST has been to test the
> accuracy
> > of
> > >> measurements against a ground-truth. It pays-off.
> > >>
> > >>>
> > >>> I filed a couple bug reports on trivial stuff:
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> > >>>
> > >>
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > >>> SswH_opYmEw$
> > >> [acm]
> > >> much appreciated...  We have an OB-UDPST project meeting this Friday, can
> > >> discuss then.
> > >>
> > >>>
> > >>> (Adding diffserv and ecn washing or marking detection would be a nice
> > >>> feature to have)
> > >>>
> > >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> > >>> phase I haven't looked much further.
> > >>>
> > >>> I left a copy running on one of my starlink testbeds -
> > >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> > >>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> > >>> version from bob, but close), and now udpst, and good to about a gbit.
> > >>>
> > >>> nice tool!
> > >> [acm]
> > >> Thanks again!
> > >>
> > >>>
> > >>> Has anyone here played with crusader? (
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > >>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> > )
> > >>>
> > >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> > >>>>
> > >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com>
> wrote:
> > >>>>
> > >>>>>> have you tried irtt?
> > >>>
> > >>
> >
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > >>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> > )
> > >>>>> I have not. Seems like a reasonable tool for UDP testing. The feature
> I
> > >>> didn't like in my scan of the documentation is the use of Inter-packet
> > delay
> > >>> variation (IPDV) instead of packet delay variation (PDV): variation from
> > the
> > >>> minimum (or reference) delay. The morbidly curious can find my analysis
> in
> > >> RFC
> > >>> 5481:
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> > >>>
> > >>
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > >>> T7QSlc$
> > >>>>
> > >>>> irtt was meant to simulate high speed voip and one day
> > >>>> videoconferencing. Please inspect the json output
> > >>>> for other metrics. Due to OS limits it is typically only accurate to a
> > >>>> 3ms interval. One thing it does admirably is begin to expose the
> > >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > >>>> technologies, as well as request/grant systems like cable and gpon,
> > >>>> especially when otherwise idle.
> > >>>>
> > >>>> Here is a highres plot of starlink's behaviors from last year:
> > >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-
> adaptive-
> > >>> bandwidth-
> > >>>
> > >>
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > >>>>
> > >>>> clearly showing them "optimizing for bandwidth" and changing next sat
> > >>>> hop, and about a 40ms interval of buffering between these switches.
> > >>>> I'd published elsewhere, if anyone cares, a preliminary study of what
> > >>>> starlink's default behaviors did to cubic and BBR...
> > >>>>
> > >>>>>
> > >>>>> irtt's use of IPDV means that the results won’t compare with UDPST,
> and
> > >>> possibly networkQuality. But I may give it a try anyway...
> > >>>>
> > >>>> The more the merrier! Someday the "right" metrics will arrive.
> > >>>>
> > >>>> As a side note, this paper focuses on RAN uplink latency
> > >>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> > >>>
> > >>
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHgvqerjg$   which I think
> > >>>> is a major barrier to most forms of 5G actually achieving good
> > >>>> performance in a FPS game, if it is true for more RANs. I'd like more
> > >>>> to be testing uplink latencies idle and with load, on all
> > >>>> technologies.
> > >>>>
> > >>>>>
> > >>>>> thanks again, Dave.
> > >>>>> Al
> > >>>>>
> > >>>>>> -----Original Message-----
> > >>>>>> From: Dave Taht <dave.taht@gmail.com>
> > >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> > >>>>>> To: MORTON JR., AL <acmorton@att.com>
> > >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > >>> Latency"
> > >>>>>> metrics
> > >>>>>>
> > >>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
> > >>> that.
> > >>>>>>
> > >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> (astute readers may have guessed that I pressed "send" too soon on
> > >>> previous
> > >>>>>> message...)
> > >>>>>>>
> > >>>>>>> I also conducted upstream tests this time, here are the results:
> > >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
> > >> and
> > >>>>>> Medium)
> > >>>>>>>
> > >>>>>>> Net Qual                           UDPST (RFC9097)
> > >> Ookla
> > >>>>>>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
> > >> UpCap
> > >>>>>> Ping(no load)
> > >>>>>>> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22
> > >>> 8
> > >>>>>>> 22         281 m 214ms  8ms        27 (52)  25       5-248      22
> > >>> 8
> > >>>>>>> 22         290 m 207ms  8ms        27 (55)  28       0-253      22
> > >>> 9
> > >>>>>>> 21         330 m 182ms  11ms       23 (44)  28       0-255      22
> > >>> 7
> > >>>>>>> 22         334 m 180ms  9ms        33 (56)  25       0-255      22
> > >>> 9
> > >>>>>>>
> > >>>>>>> The Upstream capacity measurements reflect an interesting feature
> > >> that
> > >>> we
> > >>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
> > >> of
> > >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> > >> this
> > >>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
> > >>> reporting
> > >>>>>> option that divides the complete measurement interval in two time
> > >>> intervals to
> > >>>>>> report an initial (turbo) max capacity and a steady-state max
> capacity
> > >>> for the
> > >>>>>> later intervals. The UDPST capacity results present both
> measurements:
> > >>> steady-
> > >>>>>> state first.
> > >>>>>>
> > >>>>>> Certainly we can expect bi-model distributions from many ISPs, as,
> for
> > >>>>>> one thing, the "speedboost" concept remains popular, except that it's
> > >>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > >>>>>> it is often configured "sneakily", in that it doesn't kick in for the
> > >>>>>> typical observed duration of the test, for some, they cut the
> > >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> > >>>>>>
> > >>>>>> One of my biggest issues with the rpm spec so far is that it should,
> > >>>>>> at least, sometimes, run randomly longer than the overly short
> > >>>>>> interval it runs for and the tools also allow for manual override of
> > >>> length.
> > >>>>>>
> > >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> > >> default
> > >>> for
> > >>>>>> 1m.
> > >>>>>>
> > >>>>>> Also, AQMs on the path can take a while to find the optimal drop or
> > >> mark
> > >>> rate.
> > >>>>>>
> > >>>>>>>
> > >>>>>>> The capacity processing in networkQuality and Ookla appear to report
> > >>> the
> > >>>>>> steady-state result.
> > >>>>>>
> > >>>>>> Ookla used to basically report the last result. Also it's not a good
> > >>>>>> indicator of web traffic behavior at all, watching the curve
> > >>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > >>>>>> (40ms)....
> > >>>>>>
> > >>>>>> So adding another mode - how quickly is peak bandwidth actually
> > >>>>>> reached, would be nice.
> > >>>>>>
> > >>>>>> I haven't poked into the current iteration of the goresponsiveness
> > >>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
> > >>>>>>
> > >>>
> > >>
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > >>>>>> would be good to try collecting more statistics and histograms and
> > >>>>>> methods of analyzing the data in that libre-source version.
> > >>>>>>
> > >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> > >>> goresponsiveness?
> > >>>>>>
> > >>>>>>> I watched the upstream capacity measurements on the Ookla app, and
> > >>> could
> > >>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps
> for
> > >>> most of
> > >>>>>> the test which determined the final result.
> > >>>>>>
> > >>>>>> I tend to get upset when I see ookla's new test flash a peak result
> in
> > >>>>>> the seconds and then settle on some lower number somehow.
> > >>>>>> So far as I know they are only sampling the latency every 250ms.
> > >>>>>>
> > >>>>>>>
> > >>>>>>> The working latency is about 200ms in networkQuality and about 280ms
> > >>> as
> > >>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
> > >> delay
> > >>> is
> > >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> > >>> difference
> > >>>>>> in working latency.  Also, we used the very dynamic Type C load
> > >>>>>> adjustment/search algorithm in UDPST during all of this testing,
> which
> > >>> could
> > >>>>>> explain the higher working latency to some degree.
> > >>>>>>>
> > >>>>>>> So, it's worth noting that the measurements needed for assessing
> > >>> working
> > >>>>>> latency/responsiveness are available in the UDPST utility, and that
> > >> the
> > >>> UDPST
> > >>>>>> measurements are conducted on UDP transport (used by a growing
> > >> fraction
> > >>> of
> > >>>>>> Internet traffic).
> > >>>>>>
> > >>>>>> Thx, didn't know of this work til now!
> > >>>>>>
> > >>>>>> have you tried irtt?
> > >>>>>>
> > >>>>>>>
> > >>>>>>> comments welcome of course,
> > >>>>>>> Al
> > >>>>>>>
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> > >>>>>>>> To: ippm@ietf.org
> > >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > >>>>>> Latency"
> > >>>>>>>> metrics
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Hi again RPM friends and IPPM'ers,
> > >>>>>>>>
> > >>>>>>>> As promised, I repeated the tests shared last week, this time
> > >> using
> > >>> both
> > >>>>>> the
> > >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> > >>> networkQuality. I
> > >>>>>>>> followed Sebastian's calculations as well.
> > >>>>>>>>
> > >>>>>>>> Working Latency & Capacity Summary
> > >>>>>>>>
> > >>>>>>>> Net Qual                           UDPST
> > >>> Ookla
> > >>>>>>>> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
> > >>> DnCap
> > >>>>>>>> Ping(no load)
> > >>>>>>>> 885       916 m  66ms   8ms        970      28       0-20
> > >> 940
> > >>> 8
> > >>>>>>>> 888      1355 h  44ms   8ms        966      28       0-23
> > >> 940
> > >>> 8
> > >>>>>>>> 891      1109 h  54ms   8ms        968      27       0-19
> > >> 940
> > >>> 9
> > >>>>>>>> 887      1141 h  53ms   11ms       966      27       0-18
> > >> 937
> > >>> 7
> > >>>>>>>> 884      1151 h  52ms   9ms        968      28       0-20
> > >> 937
> > >>> 9
> > >>>>>>>>
> > >>>>>>>> With the sequential test option, I noticed that networkQuality
> > >>> achieved
> > >>>>>> nearly
> > >>>>>>>> the maximum capacity reported almost immediately at the start of a
> > >>> test.
> > >>>>>>>> However, the reported capacities are low by about 60Mbps,
> > >> especially
> > >>> when
> > >>>>>>>> compared to the Ookla TCP measurements.
> > >>>>>>>>
> > >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> > >> high
> > >>> end of
> > >>>>>> the
> > >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
> > >>>>>> networkQuality
> > >>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
> > >>> be much
> > >>>>>>>> buffering in the downstream direction.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> > >>>>>>>>> To: ippm@ietf.org
> > >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
> > >>> Latency"
> > >>>>>>>>> metrics
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Hi RPM friends and IPPM'ers,
> > >>>>>>>>>
> > >>>>>>>>> I was wondering what a comparison of some of the "working
> > >> latency"
> > >>>>>> metrics
> > >>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
> > >>> 3.1, with
> > >>>>>> the
> > >>>>>>>>> downlink provisioned for 1Gbps.
> > >>>>>>>>>
> > >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
> > >>> Ookla
> > >>>>>>>> Speedtest
> > >>>>>>>>> with as similar connectivity as possible (but we know that the
> > >>> traffic
> > >>>>>> will
> > >>>>>>>>> diverge to different servers and we can't change that aspect).
> > >>>>>>>>>
> > >>>>>>>>> Here's a quick summary of yesterday's results:
> > >>>>>>>>>
> > >>>>>>>>> Working Latency & Capacity Summary
> > >>>>>>>>>
> > >>>>>>>>> Net Qual                UDPST                        Ookla
> > >>>>>>>>> DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
> > >>> Ping(no
> > >>>>>> load)
> > >>>>>>>>> 878       62            970      28       0-19       941      6
> > >>>>>>>>> 891       92            970      27       0-20       940      7
> > >>>>>>>>> 891       120           966      28       0-22       937      9
> > >>>>>>>>> 890       112           970      28       0-21       940      8
> > >>>>>>>>> 903       70            970      28       0-16       935      9
> > >>>>>>>>>
> > >>>>>>>>> Note: all RPM values were categorized as Low.
> > >>>>>>>>>
> > >>>>>>>>> networkQuality downstream capacities are always on the low side
> > >>> compared
> > >>>>>> to
> > >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
> > >>> what
> > >>>>>> Ookla
> > >>>>>>>>> achieved. I think that a longer test duration might be needed to
> > >>> achieve
> > >>>>>> the
> > >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
> > >>> observed
> > >>>>>> were
> > >>>>>>>>> certainly headed in the right direction. (I recently upgraded to
> > >>>>>> Monterey
> > >>>>>>>> 12.6
> > >>>>>>>>> on my MacBook, so should have the latest version.)
> > >>>>>>>>>
> > >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
> > >>> should
> > >>>>>> have
> > >>>>>>>>> run the tests with the -v option to help with comparisons. I'll
> > >>> repeat
> > >>>>>> this
> > >>>>>>>>> test when I can make time.
> > >>>>>>>>>
> > >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> > >> the
> > >>> test)
> > >>>>>> and
> > >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
> > >>> very
> > >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
> > >>> this
> > >>>>>> access
> > >>>>>>>> and
> > >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
> > >>> Chicago
> > >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> > >>> measurements
> > >>>>>> were
> > >>>>>>>> run
> > >>>>>>>>> on an Ubuntu VM in my MacBook.
> > >>>>>>>>>
> > >>>>>>>>> The big disappointment was that the Ookla desktop app I updated
> > >>> over the
> > >>>>>>>>> weekend did not include the new responsiveness metric! I
> > >> included
> > >>> the
> > >>>>>> ping
> > >>>>>>>>> results anyway, and it was clearly using a server in the nearby
> > >>> area.
> > >>>>>>>>>
> > >>>>>>>>> So, I have some more work to do, but I hope this is interesting-
> > >>> enough
> > >>>>>> to
> > >>>>>>>>> start some comparison discussions, and bring-out some
> > >> suggestions.
> > >>>>>>>>>
> > >>>>>>>>> happy testing all,
> > >>>>>>>>> Al
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> ippm mailing list
> > >>>>>>>>> ippm@ietf.org
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > >>>>>>>>
> > >>>>>>>> _______________________________________________
> > >>>>>>>> ippm mailing list
> > >>>>>>>> ippm@ietf.org
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>>> T!g-
> > >>>>>>
> > >>>
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > >>>>>>>
> > >>>>>>> _______________________________________________
> > >>>>>>> ippm mailing list
> > >>>>>>> ippm@ietf.org
> > >>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>
> > >>>
> > >>
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > >>>>>> KXU$
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> This song goes out to all the folk that thought Stadia would work:
> > >>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >>> mushroom-
> > >>>>>> song-activity-6981366665607352320-
> > >>>>>>
> > >>>
> > >>
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > >>>>>> c48iQFub34zz4iE$
> > >>>>>> Dave Täht CEO, TekLibre, LLC
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> This song goes out to all the folk that thought Stadia would work:
> > >>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >>> mushroom-song-activity-6981366665607352320-
> > >>>
> > >>
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHLHDpSWs$
> > >>>> Dave Täht CEO, TekLibre, LLC
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> This song goes out to all the folk that thought Stadia would work:
> > >>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >> mushroom-
> > >>> song-activity-6981366665607352320-
> > >>>
> > >>
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHLHDpSWs$
> > >>> Dave Täht CEO, TekLibre, LLC
> > >> _______________________________________________
> > >> ippm mailing list
> > >> ippm@ietf.org
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> > rDt9HLI$
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > >
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> > T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> > mERYCyuCd9e9PY3aNLkSw$
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!h-glaezufaCxidYk1xzTF48dbEz67JPIJjPA_nweL8YvDu6Z3TmG0A37k_DQ15FIzzwoeCLOEaw$

Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 44205 bytes --]

Adding in the author of irtt.

You had a significant timer error rate in your 1ms testing. Modern
hardware, particularly vms have great difficulty doing that below 3ms.

Similarly the cloud can be highly noisy. I am presently testing a highly
sdn - and virtualized networking fabric in a major DC... And the results
are rather disturbing. I'd like to repeat your test suite inside that DC.



On Sun, Dec 11, 2022, 11:21 AM MORTON JR., AL <acmorton@att.com> wrote:

> Hi IPPM,
>
> Prior to IETF-115, I shared a series of measurements with the IPPM list.
> We're looking at responsiveness and working latency with various metrics
> and multiple testing utilities. This message continues the discussion with
> new input.
>
> When I first published some measurements, Dave Taht added his assessment
> and included other relevant email lists in the discussion. I'm continuing
> to cross-post to all the lists in this thread.
>
> Dave originally suggested that I try a tool called irtt; I've done that
> now and these are the results.
>
> Bob McMahon: I queued your request to try your iperf2 tool behind the irtt
> measurements. I hope to make some more measurements this week...
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> We're testing a DOCIS 3.1 based service with 1Gbps down, nominally 22Mbps
> up. I used wired ETH connected to the DOCSIS modem's switch.
>
> Dave Taht made his server available for the irtt measurements. I installed
> irtt on a VM in my Macbook, the same VM that runs UDPST.  I ran quite a few
> tests to become familiar with irtt, so I'll just summarize the relevant
> ones here.
>
> I ran irtt with its traffic at the maximum allowed on Dave's server. The
> test duration was 10sec, with packets spaced at 1ms intervals from my VM
> client. This is the complete output:
>
> ./irtt client -i 1ms -l 1250 -d 10s fremont.starlink.taht.net
>
>                          Min     Mean   Median      Max  Stddev
>                          ---     ----   ------      ---  ------
>                 RTT  46.63ms  51.58ms   51.4ms     58ms  1.55ms
>          send delay  20.74ms  25.57ms   25.4ms  32.04ms  1.54ms
>       receive delay   25.8ms  26.01ms  25.96ms  30.48ms   219µs
>
>       IPDV (jitter)   1.03µs   1.15ms   1.02ms   6.87ms   793µs
>           send IPDV    176ns   1.13ms    994µs   6.41ms   776µs
>        receive IPDV      7ns   79.8µs   41.7µs   4.54ms   140µs
>
>      send call time   10.1µs   55.8µs            1.34ms  30.3µs
>         timer error     68ns    431µs            6.49ms   490µs
>   server proc. time    680ns   2.43µs            47.7µs  1.73µs
>
>                 duration: 10.2s (wait 174ms)
>    packets sent/received: 7137/7131 (0.08% loss)
>  server packets received: 7131/7137 (0.08%/0.00% loss up/down)
>      bytes sent/received: 8921250/8913750
>        send/receive rate: 7.14 Mbps / 7.13 Mbps
>            packet length: 1250 bytes
>              timer stats: 2863/10000 (28.63%) missed, 43.10% error
> acm@acm-ubuntu1804-1:~/goirtt/irtt$
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> irtt supplies lots of info/stats about the RTT distribution. In the
> lightly loaded (7.1Mbps) scenario above, the RTT range is about 12 ms. The
> Mean and the Median are approximately the same.
> irtt also supplies Inter-Packet Delay Varition (IPDV), showing that the
> packets seem to be delayed more than the sending interval occasionally (Max
> of 6.8ms).
> irtt measurements indicate very low packet loss: no congestion at 7Mbps.
>
> For the opposite end of the congestion spectrum, I ran irtt with UDPST
> (RFC 9097) running in parallel (and using the Type C search algorithm). We
> pick-up a lot more RTT and wider RTT range in this scenario:
>
> irtt with udpst using Type C search = max load:
>
>                          Min     Mean   Median      Max   Stddev
>                          ---     ----   ------      ---   ------
>                 RTT  47.58ms    118ms  56.53ms  301.6ms  90.28ms
>          send delay  24.05ms  94.85ms  33.38ms  278.5ms  90.26ms
>       receive delay  22.99ms  23.17ms  23.13ms  25.42ms    156µs
>
>       IPDV (jitter)    162ns   1.04ms    733µs   6.36ms   1.02ms
>           send IPDV   3.81µs   1.01ms    697µs   6.24ms   1.02ms
>        receive IPDV     88ns     93µs   49.8µs   1.48ms    145µs
>
>      send call time   4.28µs   39.3µs             903µs   32.4µs
>         timer error     86ns    287µs            6.13ms    214µs
>   server proc. time    670ns   3.59µs            19.3µs   2.26µs
>
>                 duration: 10.9s (wait 904.8ms)
>    packets sent/received: 8305/2408 (71.01% loss)
>  server packets received: 2408/8305 (71.01%/0.00% loss up/down)
>      bytes sent/received: 10381250/3010000
>        send/receive rate: 8.31 Mbps / 2.47 Mbps
>            packet length: 1250 bytes
>              timer stats: 1695/10000 (16.95%) missed, 28.75% error
> acm@acm-ubuntu1804-1:~/goirtt/irtt$
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> The irtt measurement of RTT range is now about 250ms in this
> maximally-loaded scenario.
> One key RTT difference is the Mean delay, which is now twice the Median
> with working load added by UDPST.
> Congestion is evident in the irtt loss measurements = 71%. However, the
> competing traffic did not break irtt's process or measurements.
> UDPST's measurements were ~22Mbps Capacity (steady-state), with high loss,
> and similar RTT range of 251 ms.
>
>
> Additional tests included load from UDPST at fixed rates: 14Mbps and
> 20Mbps. We compare the RTT range for all four conditions in the table below:
>
> Load with irtt   irtt RTT range    UDPST RTT range
> ==================================================
> irtt alone            12ms               ---
> UDPST at 14Mbps       11ms               22ms
> UDPST at 20Mbps       14ms               46ms
> UDPST at MaxCap      250ms              251ms
>
> The unexpected result with irtt measurements is that the RTT range did not
> increase with load, where the UDPST RTT range increases with load. We're
> assuming that the majority of delay increases occur in the DOCSIS upstream
> queue and both test streams should see similar delay range as they do with
> maximum load. Perhaps there are some differences owing to the periodic irtt
> stream and the bursty UDPST stream (both are UDP), but this is speculation.
>
> To check that the test path was operating similarly with earlier tests, we
> ran a couple of NetworkQuality and UDPST tests as before:
>
> Comparison of NQ-vs, UDPST -A c, same columns as defined earlier.
> Working Latency & Capacity Summary (Upstream only)
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
>
> Net Qual                           UDPST (RFC 9097)
> UpCap     RPM    DelLD  DelMin     UpCap(stable)    RTTmin   RTTrange
> 22        276 m  217ms  11ms       23               28       0-252
> 22        291 m  206ms  11ms       ~22*             28*      0-251*
>
> * UDPST test result with the ~7Mbps irtt stream present.
>
> We found that uplink test results are similar to previous tests, and the
> new irtt results were collected under similar conditions.
>
> In conclusion, irtt provides a clear summary of the RTT distribution.
> Minimum, Mean, Median and Max RTT are useful individually and in
> combinations/comparisons. The irtt measurements compare favorably to those
> collected during IP-Layer Capacity tests with the UDPST utility.
>
> comments welcome,
> Al
>
> > -----Original Message-----
> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > Sent: Saturday, November 5, 2022 3:37 PM
> > To: Sebastian Moeller <moeller0@gmx.de>
> > Cc: Rpm <rpm@lists.bufferbloat.net>; Will Hawkins <hawkinsw@obs.cr>;
> > ippm@ietf.org
> > Subject: Re: [ippm] [Rpm] Preliminary measurement comparison of "Working
> > Latency" metrics
> >
> > *** Security Advisory: This Message Originated Outside of AT&T ***.
> > Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> >
> > Hi Sebastian, thanks for your comments and information.
> > I'll reply to a few key points in this top-post.
> >
> > Sebastian wrote:
> > >     [SM] DOCSIS ISPs traditionally provision higher peak rates than
> they
> > > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even
> via
> > > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput
> in the
> > > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > > achievable throughput somewhat:
> > [acm]
> > Right, my ISP's CPE has Gig-E ports so I won't see any benefit from
> over-rate
> > provisioning.
> >
> > Sebastian wrote:
> > ...
> > > IPv4/TCP/RFC1323 timestamps payload:        1000 * ((1500 - 12 - 20 -
> 20)/(1500
> > + 38 + 4)) = 939.04 Mbps
> > ...
> > > Speedtests tend to report IPv4/TCP timestamps might be on depending on
> the
> > OS,
> > > but on-line speedtest almost never return the simple average rate over
> the
> > > measurement duration, but play "clever" tricks to exclude the start-up
> phase
> > > and to aggregate over multiple flows that invariably ending with
> results
> > that
> > > tend to exceed the hard limits shown above...
> > [acm]
> > My measurements with the Ookla desktop app on MacOS (shared earlier this
> week)
> > are very consistent at ~940Mbps Dn-stream, so timestamps calculation
> sounds
> > right.
> > My ISP specifies their service speed at 940Mbps, as though they assume
> > subscribers will measure using Ookla or other TCP tool. In fact, our team
> > hasn't seen such consistent results from Ookla or any TCP-based test in
> the
> > 1Gbps range, and this makes me wonder if there might be some
> test-recognition
> > here.
> >
> > FYI - UDPST uses a max packet size with is sub-MTU, to avoid
> fragmentation
> > when various encapsulations are encountered. Also, the definition of
> IP-Layer
> > Capacity in the various standards (e.g., RFC 9097) includes the bits in
> the IP
> > header in the total Capacity.
> >
> > So, instead of:
> > > Ethernet payload rate +VLAN:                1000 * ((1500)/(1500 + 38
> + 4)) =
> > 972.76 Mbps
> > We have
> > Ethernet payload rate +VLAN:          1000 * ((1222)/(1222 + 38 + 4)) =
> 966.77
> > Mbps
> > and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps
> >
> > UDPST has an option to use datagrams for the traditional 1500 octet MTU
> (-T),
> > but a user could cause a lot of fragmentation on links with
> encapsulations wit
> > this option.
> >
> > acm wrote:
> > > > The comparison of networkQuality and goresponsiveness is somewhat
> > confounded
> > > > by the need to use the Apple server infrastructure for both these
> methods
> > ...
> > Sebastian wrote:
> > >     [SM] Puzzled, I thought when comparing the two networkQuality
> variants
> > > having the same back-end sort of helps be reducing the differences to
> the
> > > clients? But comparison to UDPST suffers somewhat.
> > [acm]
> > Yes, I was hoping to match-up the client and server implementations.
> Instead,
> > this might be more of a Client-X and Server-Y interoperability test (and
> could
> > explain some results?), but that was not to be.
> >
> > Thanks for your suggested command line for gores.
> >
> > IIRC one of your early messages, Sebastian, your MacOS indicates that
> 12.6 is
> > the latest. It's the same for me: my sufficiently powerful MacBook cannot
> > upgrade to Ventura for the latest in networkQuality versions. It would
> help us
> > who are doing some testing if the latest version of networkQuality could
> be
> > made installable for us, somehow...
> >
> > thanks again and regards,
> > Al
> >
> >
> > > -----Original Message-----
> > > From: Sebastian Moeller <moeller0@gmx.de>
> > > Sent: Friday, November 4, 2022 3:10 PM
> > > To: MORTON JR., AL <acmorton@att.com>
> > > Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <
> rjmcmahon@rjmcmahon.com>;
> > Rpm
> > > <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <
> hawkinsw@obs.cr>
> > > Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of
> "Working
> > > Latency" metrics
> > >
> > > Hi Al,
> > >
> > >
> > > > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm
> > <rpm@lists.bufferbloat.net>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I have been working through the threads on misery metrics and
> lightweight
> > > sensing of bw and buffering, and with those insights and gold-nuggets
> in
> > mind,
> > > I'm iterating through testing the combinations of tools that Dave That
> and
> > Bob
> > > McMahon suggested.
> > > >
> > > > earlier this week, Dave wrote:
> > > >> How does networkQuality compare vs a vs your tool vs a vs
> > goresponsiveness?
> > > >
> > > > goresponsiveness installed flawlessly - very nice instructions and
> > getting-
> > > started info.
> > > >
> > > > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> > > gores/goresponsiveness
> > > >
> > > > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps
> down-
> > > stream service
> > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > Medium)
> > > >
> > > > NetQual                            UDPST (RFC 9097)             gores
> > > > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
> > RPM
> > > DelLD
> > > > 882       788 m  76ms   8ms        967      28       0-16       127
> > > (1382) 43ms
> > > > 892      1036 h  58ms   8          966      27       0-18       128
> > > (2124) 28ms
> > > > 887      1304 h  46ms   6          969      27       0-18       130
> > > (1478) 41ms
> > > > 885      1008 h  60ms   8          967      28       0-22       127
> > > (1490) 40ms
> > > > 894      1383 h  43ms   11         967      28       0-15       133
> > > (2731) 22ms
> > > >
> > > > NetQual                            UDPST (RFC 9097)             gores
> > > > UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap
> > RPM
> > > DelLD
> > > > 21        327 m  183ms  8ms        22 (51)  28       0-253      12
> > > > 21        413 m  145ms  8          22 (43)  28       0-255      15
> > > > 22        273 m  220ms  6          23 (53)  28       0-259      10
> > > > 21        377 m  159ms  8          23 (51)  28       0-250      10
> > > > 22        281 m  214ms  11         23 (52)  28       0-250       6
> > > >
> > > > These tests were conducted in a round-robin fashion to minimize the
> > > possibility of network variations between measurements:
> > > > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest -
> repeat
> > > >
> > > > NetQual indicates the same reduced capacity in Downstream when
> compared to
> > > UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for
> IP-
> > layer
> > > capacity, dep. on VLAN tag).
> > >
> > >     [SM] DOCSIS ISPs traditionally provision higher peak rates than
> they
> > > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even
> via
> > > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput
> in the
> > > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > > achievable throughput somewhat:
> > >
> > > Ethernet payload rate:                      1000 * ((1500)/(1500 +
> 38)) =
> > >      975.29 Mbps
> > > Ethernet payload rate +VLAN:                1000 * ((1500)/(1500 + 38
> + 4)) =
> > >      972.76 Mbps
> > > IPv4 payload (ethernet+VLAN):               1000 * ((1500 - 20)/(1500
> + 38 + 4))
> > =
> > >              959.79 Mbps
> > > IPv6 payload (ethernet+VLAN):               1000 * ((1500 - 40)/(1500
> + 38 + 4))
> > =
> > >              946.82 Mbps
> > > IPv4/TCP payload (ethernet+VLAN):   1000 * ((1500 - 20 - 20)/(1500 + 38
> > +
> > > 4)) =        946.82 Mbps
> > > IPv6/TCP payload (ethernet+VLAN):   1000 * ((1500 - 20 - 40)/(1500 + 38
> > +
> > > 4)) =        933.85 Mbps
> > > IPv4/TCP/RFC1323 timestamps payload:        1000 * ((1500 - 12 - 20 -
> 20)/(1500
> > +
> > > 38 + 4)) = 939.04 Mbps
> > > IPv6/TCP/RFC1323 timestamps payload:        1000 * ((1500 - 12 - 20 -
> 40)/(1500
> > +
> > > 38 + 4)) = 926.07 Mbps
> > >
> > >
> > > Speedtests tend to report IPv4/TCP timestamps might be on depending on
> the
> > OS,
> > > but on-line speedtest almost never return the simple average rate over
> the
> > > measurement duration, but play "clever" tricks to exclude the start-up
> phase
> > > and to aggregate over multiple flows that invariably ending with
> results
> > that
> > > tend to exceed the hard limits shown above...
> > >
> > >
> > > > Upstream capacities are not very different (a factor that made TCP
> methods
> > > more viable many years ago when most consumer access speeds were
> limited to
> > > 10's of Megabits).
> > >
> > >     [SM] My take on this is that this is partly due to the goal of
> ramping
> > > up very quickly and get away with a short measurement duration. That
> causes
> > > imprecision. As Dave said flent's RRUL test defaults to 60 seconds and
> I
> > often
> > > ran/run it for 5 to 10 minutes to get somewhat more reliable numbers
> (and
> > for
> > > timecourses to look at and reason about).
> > >
> > > > gores reports significantly lower capacity in both downstream and
> upstream
> > > measurements, a factor of 7 less than NetQual for downstream.
> Interestingly,
> > > the reduced capacity (taken as the working load) results in higher
> > > responsiveness: RPM meas are higher and loaded delays are lower for
> > > downstream.
> > >
> > >     [SM] Yepp, if you only fill-up the queue partially you will
> harvest less
> > > queueing delay and hence retain more responsiveness, albeit this
> really just
> > > seems to be better interpreted as go-responsiveness failing to achieve
> > > "working-conditions".
> > >
> > > I tend to run gores like this:
> > > time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> > > /api/v1/gm/config  --sattimeout 60 --extended-stats --logger-filename
> > > go_networkQuality_$(date +%Y%m%d_%H%M%S)
> > >
> > > --sattimeout 60 extends the time out for the saturation measurement
> somewhat
> > > (before I saw it simply failing on a 1 Gbps access link, it did give
> some
> > > diagnostic message though).
> > >
> > >
> > > >
> > > > The comparison of networkQuality and goresponsiveness is somewhat
> > confounded
> > > by the need to use the Apple server infrastructure for both these
> methods
> > (the
> > > documentation provides this option - thanks!). I don't have admin
> access to
> > > our server at the moment. But the measured differences are large
> despite the
> > > confounding factor.
> > >
> > >     [SM] Puzzled, I thought when comparing the two ntworkQuality
> variants
> > > having the same back-end sort of helps be reducing the differences to
> the
> > > clients? But comparison to UDPST suffers somewhat.
> > >
> > > >
> > > > goresponsiveness has its own, very different output format than
> > > networkQuality. There isn't a comparable "-v" option other than -debug
> > (which
> > > is extremely detailed). gores only reports RPM for the downstream.
> > >
> > >     [SM] I agree it would be nice if gores would grow a sequential
> mode as
> > > well.
> > >
> > >
> > > > I hope that these results will prompt more of the principle
> evangelists
> > and
> > > coders to weigh-in.
> > >
> > >     [SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> > > discuss.
> > >
> > > >
> > > > It's also worth noting that the RTTrange reported by UDPST is the
> range
> > > above the minimum RTT, and represents the entire 10 second test. The
> > > consistency of the maximum of the range (~255ms) seems to indicate that
> > UDPST
> > > has characterized the length of the upstream buffer during
> measurements.
> > >
> > >     [SM] thanks for explaining that.
> > >
> > > Regards
> > >     Sebastian
> > >
> > > >
> > > > I'm sure there is more to observe that is prompted by these
> measurements;
> > > comments welcome!
> > > >
> > > > Al
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > >> Sent: Tuesday, November 1, 2022 10:51 AM
> > > >> To: Dave Taht <dave.taht@gmail.com>
> > > >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > >> metrics
> > > >>
> > > >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> > > >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> > information.
> > > >>
> > > >> Hi Dave,
> > > >> Thanks for trying UDPST (RFC 9097)!
> > > >>
> > > >> Something you might try with starlink:
> > > >> use the -X option and UDPST will generate random payloads.
> > > >>
> > > >> The measurements with -X will reflect the uncompressed that are
> possible.
> > > >> I tried this on a ship-board Internet access: uncompressed rate was
> > > 100kbps.
> > > >>
> > > >> A few more quick replies below,
> > > >> Al
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: Dave Taht <dave.taht@gmail.com>
> > > >>> Sent: Tuesday, November 1, 2022 12:22 AM
> > > >>> To: MORTON JR., AL <acmorton@att.com>
> > > >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > Latency"
> > > >>> metrics
> > > >>>
> > > >>> Dear Al:
> > > >>>
> > > >>> OK, I took your udpst tool for a spin.
> > > >>>
> > > >>> NICE! 120k binary (I STILL work on machines with only 4MB of
> flash),
> > > >>> good feature set, VERY fast,
> > > >> [acm]
> > > >> Len Ciavattone (my partner in crime on several measurement
> projects) is
> > the
> > > >> lead coder: he has implemented many measurement tools extremely
> > > efficiently,
> > > >> this one in C-lang.
> > > >>
> > > >>> and in very brief testing, seemed
> > > >>> to be accurate in the starlink case, though it's hard to tell with
> > > >>> them as the rate changes every 15s.
> > > >> [acm]
> > > >> Great! Our guiding principle developing UDPST has been to test the
> > accuracy
> > > of
> > > >> measurements against a ground-truth. It pays-off.
> > > >>
> > > >>>
> > > >>> I filed a couple bug reports on trivial stuff:
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> > > >>>
> > > >>
> > >
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > > >>> SswH_opYmEw$
> > > >> [acm]
> > > >> much appreciated...  We have an OB-UDPST project meeting this
> Friday, can
> > > >> discuss then.
> > > >>
> > > >>>
> > > >>> (Adding diffserv and ecn washing or marking detection would be a
> nice
> > > >>> feature to have)
> > > >>>
> > > >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> > > >>> phase I haven't looked much further.
> > > >>>
> > > >>> I left a copy running on one of my starlink testbeds -
> > > >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> > > >>> instrumented with netperf, flent, irtt, iperf2 (not quite the
> latest
> > > >>> version from bob, but close), and now udpst, and good to about a
> gbit.
> > > >>>
> > > >>> nice tool!
> > > >> [acm]
> > > >> Thanks again!
> > > >>
> > > >>>
> > > >>> Has anyone here played with crusader? (
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > > >>>
> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> > > )
> > > >>>
> > > >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com>
> wrote:
> > > >>>>
> > > >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com>
> > wrote:
> > > >>>>
> > > >>>>>> have you tried irtt?
> > > >>>
> > > >>
> > >
> > (
> https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > > >>>
> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> > > )
> > > >>>>> I have not. Seems like a reasonable tool for UDP testing. The
> feature
> > I
> > > >>> didn't like in my scan of the documentation is the use of
> Inter-packet
> > > delay
> > > >>> variation (IPDV) instead of packet delay variation (PDV):
> variation from
> > > the
> > > >>> minimum (or reference) delay. The morbidly curious can find my
> analysis
> > in
> > > >> RFC
> > > >>> 5481:
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;
> !!
> > > >>>
> > > >>
> > >
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > > >>> T7QSlc$
> > > >>>>
> > > >>>> irtt was meant to simulate high speed voip and one day
> > > >>>> videoconferencing. Please inspect the json output
> > > >>>> for other metrics. Due to OS limits it is typically only accurate
> to a
> > > >>>> 3ms interval. One thing it does admirably is begin to expose the
> > > >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > > >>>> technologies, as well as request/grant systems like cable and
> gpon,
> > > >>>> especially when otherwise idle.
> > > >>>>
> > > >>>> Here is a highres plot of starlink's behaviors from last year:
> > > >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-
> > adaptive-
> > > >>> bandwidth-
> > > >>>
> > > >>
> > >
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > > >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > > >>>>
> > > >>>> clearly showing them "optimizing for bandwidth" and changing next
> sat
> > > >>>> hop, and about a 40ms interval of buffering between these
> switches.
> > > >>>> I'd published elsewhere, if anyone cares, a preliminary study of
> what
> > > >>>> starlink's default behaviors did to cubic and BBR...
> > > >>>>
> > > >>>>>
> > > >>>>> irtt's use of IPDV means that the results won’t compare with
> UDPST,
> > and
> > > >>> possibly networkQuality. But I may give it a try anyway...
> > > >>>>
> > > >>>> The more the merrier! Someday the "right" metrics will arrive.
> > > >>>>
> > > >>>> As a side note, this paper focuses on RAN uplink latency
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> > > >>>
> > > >>
> > >
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHgvqerjg$   which I think
> > > >>>> is a major barrier to most forms of 5G actually achieving good
> > > >>>> performance in a FPS game, if it is true for more RANs. I'd like
> more
> > > >>>> to be testing uplink latencies idle and with load, on all
> > > >>>> technologies.
> > > >>>>
> > > >>>>>
> > > >>>>> thanks again, Dave.
> > > >>>>> Al
> > > >>>>>
> > > >>>>>> -----Original Message-----
> > > >>>>>> From: Dave Taht <dave.taht@gmail.com>
> > > >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> > > >>>>>> To: MORTON JR., AL <acmorton@att.com>
> > > >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of
> "Working
> > > >>> Latency"
> > > >>>>>> metrics
> > > >>>>>>
> > > >>>>>> Thank you very much for the steer to RFC9097. I'd completely
> missed
> > > >>> that.
> > > >>>>>>
> > > >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <
> acmorton@att.com>
> > > >> wrote:
> > > >>>>>>>
> > > >>>>>>> (astute readers may have guessed that I pressed "send" too
> soon on
> > > >>> previous
> > > >>>>>> message...)
> > > >>>>>>>
> > > >>>>>>> I also conducted upstream tests this time, here are the
> results:
> > > >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories,
> High
> > > >> and
> > > >>>>>> Medium)
> > > >>>>>>>
> > > >>>>>>> Net Qual                           UDPST (RFC9097)
> > > >> Ookla
> > > >>>>>>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange
> > > >> UpCap
> > > >>>>>> Ping(no load)
> > > >>>>>>> 34        1821 h 33ms   11ms       23 (42)  28       0-252
>   22
> > > >>> 8
> > > >>>>>>> 22         281 m 214ms  8ms        27 (52)  25       5-248
>   22
> > > >>> 8
> > > >>>>>>> 22         290 m 207ms  8ms        27 (55)  28       0-253
>   22
> > > >>> 9
> > > >>>>>>> 21         330 m 182ms  11ms       23 (44)  28       0-255
>   22
> > > >>> 7
> > > >>>>>>> 22         334 m 180ms  9ms        33 (56)  25       0-255
>   22
> > > >>> 9
> > > >>>>>>>
> > > >>>>>>> The Upstream capacity measurements reflect an interesting
> feature
> > > >> that
> > > >>> we
> > > >>>>>> can reliably and repeatably measure with UDPST. The first ~3
> seconds
> > > >> of
> > > >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST
> displays
> > > >> this
> > > >>>>>> behavior in its 1 second sub-interval measurements and has a
> bimodal
> > > >>> reporting
> > > >>>>>> option that divides the complete measurement interval in two
> time
> > > >>> intervals to
> > > >>>>>> report an initial (turbo) max capacity and a steady-state max
> > capacity
> > > >>> for the
> > > >>>>>> later intervals. The UDPST capacity results present both
> > measurements:
> > > >>> steady-
> > > >>>>>> state first.
> > > >>>>>>
> > > >>>>>> Certainly we can expect bi-model distributions from many ISPs,
> as,
> > for
> > > >>>>>> one thing, the "speedboost" concept remains popular, except
> that it's
> > > >>>>>> misnamed, as it should be called speed-subtract or speed-lose.
> Worse,
> > > >>>>>> it is often configured "sneakily", in that it doesn't kick in
> for the
> > > >>>>>> typical observed duration of the test, for some, they cut the
> > > >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> > > >>>>>>
> > > >>>>>> One of my biggest issues with the rpm spec so far is that it
> should,
> > > >>>>>> at least, sometimes, run randomly longer than the overly short
> > > >>>>>> interval it runs for and the tools also allow for manual
> override of
> > > >>> length.
> > > >>>>>>
> > > >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> > > >> default
> > > >>> for
> > > >>>>>> 1m.
> > > >>>>>>
> > > >>>>>> Also, AQMs on the path can take a while to find the optimal
> drop or
> > > >> mark
> > > >>> rate.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> The capacity processing in networkQuality and Ookla appear to
> report
> > > >>> the
> > > >>>>>> steady-state result.
> > > >>>>>>
> > > >>>>>> Ookla used to basically report the last result. Also it's not a
> good
> > > >>>>>> indicator of web traffic behavior at all, watching the curve
> > > >>>>>> go up much more slowly in their test on say, fiber 2ms, vs
> starlink,
> > > >>>>>> (40ms)....
> > > >>>>>>
> > > >>>>>> So adding another mode - how quickly is peak bandwidth actually
> > > >>>>>> reached, would be nice.
> > > >>>>>>
> > > >>>>>> I haven't poked into the current iteration of the
> goresponsiveness
> > > >>>>>> test at all:
> https://urldefense.com/v3/__https://github.com/network-
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$   it
> > > >>>>>> would be good to try collecting more statistics and histograms
> and
> > > >>>>>> methods of analyzing the data in that libre-source version.
> > > >>>>>>
> > > >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> > > >>> goresponsiveness?
> > > >>>>>>
> > > >>>>>>> I watched the upstream capacity measurements on the Ookla app,
> and
> > > >>> could
> > > >>>>>> easily see the initial rise to 40-50Mbps, then the drop to
> ~22Mbps
> > for
> > > >>> most of
> > > >>>>>> the test which determined the final result.
> > > >>>>>>
> > > >>>>>> I tend to get upset when I see ookla's new test flash a peak
> result
> > in
> > > >>>>>> the seconds and then settle on some lower number somehow.
> > > >>>>>> So far as I know they are only sampling the latency every 250ms.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> The working latency is about 200ms in networkQuality and about
> 280ms
> > > >>> as
> > > >>>>>> measured by UDPST (RFC9097). Note that the networkQuality
> minimum
> > > >> delay
> > > >>> is
> > > >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of
> the
> > > >>> difference
> > > >>>>>> in working latency.  Also, we used the very dynamic Type C load
> > > >>>>>> adjustment/search algorithm in UDPST during all of this testing,
> > which
> > > >>> could
> > > >>>>>> explain the higher working latency to some degree.
> > > >>>>>>>
> > > >>>>>>> So, it's worth noting that the measurements needed for
> assessing
> > > >>> working
> > > >>>>>> latency/responsiveness are available in the UDPST utility, and
> that
> > > >> the
> > > >>> UDPST
> > > >>>>>> measurements are conducted on UDP transport (used by a growing
> > > >> fraction
> > > >>> of
> > > >>>>>> Internet traffic).
> > > >>>>>>
> > > >>>>>> Thx, didn't know of this work til now!
> > > >>>>>>
> > > >>>>>> have you tried irtt?
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> comments welcome of course,
> > > >>>>>>> Al
> > > >>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR.,
> AL
> > > >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> > > >>>>>>>> To: ippm@ietf.org
> > > >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of
> "Working
> > > >>>>>> Latency"
> > > >>>>>>>> metrics
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Hi again RPM friends and IPPM'ers,
> > > >>>>>>>>
> > > >>>>>>>> As promised, I repeated the tests shared last week, this time
> > > >> using
> > > >>> both
> > > >>>>>> the
> > > >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> > > >>> networkQuality. I
> > > >>>>>>>> followed Sebastian's calculations as well.
> > > >>>>>>>>
> > > >>>>>>>> Working Latency & Capacity Summary
> > > >>>>>>>>
> > > >>>>>>>> Net Qual                           UDPST
> > > >>> Ookla
> > > >>>>>>>> DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange
> > > >>> DnCap
> > > >>>>>>>> Ping(no load)
> > > >>>>>>>> 885       916 m  66ms   8ms        970      28       0-20
> > > >> 940
> > > >>> 8
> > > >>>>>>>> 888      1355 h  44ms   8ms        966      28       0-23
> > > >> 940
> > > >>> 8
> > > >>>>>>>> 891      1109 h  54ms   8ms        968      27       0-19
> > > >> 940
> > > >>> 9
> > > >>>>>>>> 887      1141 h  53ms   11ms       966      27       0-18
> > > >> 937
> > > >>> 7
> > > >>>>>>>> 884      1151 h  52ms   9ms        968      28       0-20
> > > >> 937
> > > >>> 9
> > > >>>>>>>>
> > > >>>>>>>> With the sequential test option, I noticed that networkQuality
> > > >>> achieved
> > > >>>>>> nearly
> > > >>>>>>>> the maximum capacity reported almost immediately at the start
> of a
> > > >>> test.
> > > >>>>>>>> However, the reported capacities are low by about 60Mbps,
> > > >> especially
> > > >>> when
> > > >>>>>>>> compared to the Ookla TCP measurements.
> > > >>>>>>>>
> > > >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> > > >> high
> > > >>> end of
> > > >>>>>> the
> > > >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of
> the
> > > >>>>>> networkQuality
> > > >>>>>>>> RPM measurements were categorized as "High". There doesn't
> seem to
> > > >>> be much
> > > >>>>>>>> buffering in the downstream direction.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> -----Original Message-----
> > > >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR.,
> AL
> > > >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> > > >>>>>>>>> To: ippm@ietf.org
> > > >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of
> "Working
> > > >>> Latency"
> > > >>>>>>>>> metrics
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Hi RPM friends and IPPM'ers,
> > > >>>>>>>>>
> > > >>>>>>>>> I was wondering what a comparison of some of the "working
> > > >> latency"
> > > >>>>>> metrics
> > > >>>>>>>>> would look like, so I ran some tests using a service on
> DOCSIS
> > > >>> 3.1, with
> > > >>>>>> the
> > > >>>>>>>>> downlink provisioned for 1Gbps.
> > > >>>>>>>>>
> > > >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097),
> and
> > > >>> Ookla
> > > >>>>>>>> Speedtest
> > > >>>>>>>>> with as similar connectivity as possible (but we know that
> the
> > > >>> traffic
> > > >>>>>> will
> > > >>>>>>>>> diverge to different servers and we can't change that
> aspect).
> > > >>>>>>>>>
> > > >>>>>>>>> Here's a quick summary of yesterday's results:
> > > >>>>>>>>>
> > > >>>>>>>>> Working Latency & Capacity Summary
> > > >>>>>>>>>
> > > >>>>>>>>> Net Qual                UDPST                        Ookla
> > > >>>>>>>>> DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap
> > > >>> Ping(no
> > > >>>>>> load)
> > > >>>>>>>>> 878       62            970      28       0-19       941
>   6
> > > >>>>>>>>> 891       92            970      27       0-20       940
>   7
> > > >>>>>>>>> 891       120           966      28       0-22       937
>   9
> > > >>>>>>>>> 890       112           970      28       0-21       940
>   8
> > > >>>>>>>>> 903       70            970      28       0-16       935
>   9
> > > >>>>>>>>>
> > > >>>>>>>>> Note: all RPM values were categorized as Low.
> > > >>>>>>>>>
> > > >>>>>>>>> networkQuality downstream capacities are always on the low
> side
> > > >>> compared
> > > >>>>>> to
> > > >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's
> mostly
> > > >>> what
> > > >>>>>> Ookla
> > > >>>>>>>>> achieved. I think that a longer test duration might be
> needed to
> > > >>> achieve
> > > >>>>>> the
> > > >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate
> values
> > > >>> observed
> > > >>>>>> were
> > > >>>>>>>>> certainly headed in the right direction. (I recently
> upgraded to
> > > >>>>>> Monterey
> > > >>>>>>>> 12.6
> > > >>>>>>>>> on my MacBook, so should have the latest version.)
> > > >>>>>>>>>
> > > >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded
> me, I
> > > >>> should
> > > >>>>>> have
> > > >>>>>>>>> run the tests with the -v option to help with comparisons.
> I'll
> > > >>> repeat
> > > >>>>>> this
> > > >>>>>>>>> test when I can make time.
> > > >>>>>>>>>
> > > >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> > > >> the
> > > >>> test)
> > > >>>>>> and
> > > >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up
> to
> > > >>> very
> > > >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM
> graded
> > > >>> this
> > > >>>>>> access
> > > >>>>>>>> and
> > > >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm
> in
> > > >>> Chicago
> > > >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> > > >>> measurements
> > > >>>>>> were
> > > >>>>>>>> run
> > > >>>>>>>>> on an Ubuntu VM in my MacBook.
> > > >>>>>>>>>
> > > >>>>>>>>> The big disappointment was that the Ookla desktop app I
> updated
> > > >>> over the
> > > >>>>>>>>> weekend did not include the new responsiveness metric! I
> > > >> included
> > > >>> the
> > > >>>>>> ping
> > > >>>>>>>>> results anyway, and it was clearly using a server in the
> nearby
> > > >>> area.
> > > >>>>>>>>>
> > > >>>>>>>>> So, I have some more work to do, but I hope this is
> interesting-
> > > >>> enough
> > > >>>>>> to
> > > >>>>>>>>> start some comparison discussions, and bring-out some
> > > >> suggestions.
> > > >>>>>>>>>
> > > >>>>>>>>> happy testing all,
> > > >>>>>>>>> Al
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> _______________________________________________
> > > >>>>>>>>> ippm mailing list
> > > >>>>>>>>> ippm@ietf.org
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > >>>>>>>>
> > > >>>>>>>> _______________________________________________
> > > >>>>>>>> ippm mailing list
> > > >>>>>>>> ippm@ietf.org
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>>> T!g-
> > > >>>>>>
> > > >>>
> >
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > >>>>>>>
> > > >>>>>>> _______________________________________________
> > > >>>>>>> ippm mailing list
> > > >>>>>>> ippm@ietf.org
> > > >>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > >>>>>> KXU$
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> This song goes out to all the folk that thought Stadia would
> work:
> > > >>>>>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >>> mushroom-
> > > >>>>>> song-activity-6981366665607352320-
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > >>>>>> c48iQFub34zz4iE$
> > > >>>>>> Dave Täht CEO, TekLibre, LLC
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> This song goes out to all the folk that thought Stadia would work:
> > > >>>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >>> mushroom-song-activity-6981366665607352320-
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHLHDpSWs$
> > > >>>> Dave Täht CEO, TekLibre, LLC
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> This song goes out to all the folk that thought Stadia would work:
> > > >>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >> mushroom-
> > > >>> song-activity-6981366665607352320-
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHLHDpSWs$
> > > >>> Dave Täht CEO, TekLibre, LLC
> > > >> _______________________________________________
> > > >> ippm mailing list
> > > >> ippm@ietf.org
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>
> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> > > rDt9HLI$
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> > > T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> > > mERYCyuCd9e9PY3aNLkSw$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >
> T!h-glaezufaCxidYk1xzTF48dbEz67JPIJjPA_nweL8YvDu6Z3TmG0A37k_DQ15FIzzwoeCLOEaw$
>

[-- Attachment #2: Type: text/html, Size: 66594 bytes --]