Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics

[rjmcmahon <rjmcmahon@rjmcmahon.com>]

Would it be possible to get some iperf 2 bounceback test results too?

https://sourceforge.net/projects/iperf2/

Also, for the hunt algo, maybe use TCP first to get a starting point and 
then hunt? Just a thought.

Thanks,
Bob
> Thank you very much for the steer to RFC9097. I'd completely missed 
> that.
> 
> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> 
> wrote:
>> 
>> (astute readers may have guessed that I pressed "send" too soon on 
>> previous message...)
>> 
>> I also conducted upstream tests this time, here are the results:
>> (capacity in Mbps, delays in ms, h and m are RPM categories, High and 
>> Medium)
>> 
>> Net Qual                           UDPST (RFC9097)              Ookla
>> UpCap     RPM    DelLD  DelMin     UpCap    RTTmin   RTTrange   UpCap  
>>   Ping(no load)
>> 34        1821 h 33ms   11ms       23 (42)  28       0-252      22     
>>   8
>> 22         281 m 214ms  8ms        27 (52)  25       5-248      22     
>>   8
>> 22         290 m 207ms  8ms        27 (55)  28       0-253      22     
>>   9
>> 21         330 m 182ms  11ms       23 (44)  28       0-255      22     
>>   7
>> 22         334 m 180ms  9ms        33 (56)  25       0-255      22     
>>   9
>> 
>> The Upstream capacity measurements reflect an interesting feature that 
>> we can reliably and repeatably measure with UDPST. The first ~3 
>> seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST 
>> displays this behavior in its 1 second sub-interval measurements and 
>> has a bimodal reporting option that divides the complete measurement 
>> interval in two time intervals to report an initial (turbo) max 
>> capacity and a steady-state max capacity for the later intervals. The 
>> UDPST capacity results present both measurements: steady-state first.
> 
> Certainly we can expect bi-model distributions from many ISPs, as, for
> one thing, the "speedboost" concept remains popular, except that it's
> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> it is often configured "sneakily", in that it doesn't kick in for the
> typical observed duration of the test, for some, they cut the
> available bandwidth about 20s in, others, 1 or 5 minutes.
> 
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short
> interval it runs for and the tools also allow for manual override of 
> length.
> 
> we caught a lot of tomfoolery with flent's rrul test running by default 
> for 1m.
> 
> Also, AQMs on the path can take a while to find the optimal drop or 
> mark rate.
> 
>> 
>> The capacity processing in networkQuality and Ookla appear to report 
>> the steady-state result.
> 
> Ookla used to basically report the last result. Also it's not a good
> indicator of web traffic behavior at all, watching the curve
> go up much more slowly in their test on say, fiber 2ms, vs starlink, 
> (40ms)....
> 
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
> 
> I haven't poked into the current iteration of the goresponsiveness
> test at all: https://github.com/network-quality/goresponsiveness it
> would be good to try collecting more statistics and histograms and
> methods of analyzing the data in that libre-source version.
> 
> How does networkQuality compare vs a vs your tool vs a vs 
> goresponsiveness?
> 
>> I watched the upstream capacity measurements on the Ookla app, and 
>> could easily see the initial rise to 40-50Mbps, then the drop to 
>> ~22Mbps for most of the test which determined the final result.
> 
> I tend to get upset when I see ookla's new test flash a peak result in
> the seconds and then settle on some lower number somehow.
> So far as I know they are only sampling the latency every 250ms.
> 
>> 
>> The working latency is about 200ms in networkQuality and about 280ms 
>> as measured by UDPST (RFC9097). Note that the networkQuality minimum 
>> delay is ~20ms lower than the UDPST RTTmin, so this accounts for some 
>> of the difference in working latency.  Also, we used the very dynamic 
>> Type C load adjustment/search algorithm in UDPST during all of this 
>> testing, which could explain the higher working latency to some 
>> degree.
>> 
>> So, it's worth noting that the measurements needed for assessing 
>> working latency/responsiveness are available in the UDPST utility, and 
>> that the UDPST measurements are conducted on UDP transport (used by a 
>> growing fraction of Internet traffic).
> 
> Thx, didn't know of this work til now!
> 
> have you tried irtt?
> 
>> 
>> comments welcome of course,
>> Al
>> 
>> > -----Original Message-----
>> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > Sent: Sunday, October 30, 2022 8:09 PM
>> > To: ippm@ietf.org
>> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> > metrics
>> >
>> >
>> > Hi again RPM friends and IPPM'ers,
>> >
>> > As promised, I repeated the tests shared last week, this time using both the
>> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
>> > followed Sebastian's calculations as well.
>> >
>> > Working Latency & Capacity Summary
>> >
>> > Net Qual                           UDPST                        Ookla
>> > DnCap     RPM    DelLD  DelMin     DnCap    RTTmin   RTTrange   DnCap
>> > Ping(no load)
>> > 885       916 m  66ms   8ms        970      28       0-20       940      8
>> > 888      1355 h  44ms   8ms        966      28       0-23       940      8
>> > 891      1109 h  54ms   8ms        968      27       0-19       940      9
>> > 887      1141 h  53ms   11ms       966      27       0-18       937      7
>> > 884      1151 h  52ms   9ms        968      28       0-20       937      9
>> >
>> > With the sequential test option, I noticed that networkQuality achieved nearly
>> > the maximum capacity reported almost immediately at the start of a test.
>> > However, the reported capacities are low by about 60Mbps, especially when
>> > compared to the Ookla TCP measurements.
>> >
>> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the
>> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality
>> > RPM measurements were categorized as "High". There doesn't seem to be much
>> > buffering in the downstream direction.
>> >
>> >
>> >
>> > > -----Original Message-----
>> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > Sent: Monday, October 24, 2022 6:36 PM
>> > > To: ippm@ietf.org
>> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
>> > > metrics
>> > >
>> > >
>> > > Hi RPM friends and IPPM'ers,
>> > >
>> > > I was wondering what a comparison of some of the "working latency" metrics
>> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the
>> > > downlink provisioned for 1Gbps.
>> > >
>> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
>> > Speedtest
>> > > with as similar connectivity as possible (but we know that the traffic will
>> > > diverge to different servers and we can't change that aspect).
>> > >
>> > > Here's a quick summary of yesterday's results:
>> > >
>> > > Working Latency & Capacity Summary
>> > >
>> > > Net Qual                UDPST                        Ookla
>> > > DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no load)
>> > > 878       62            970      28       0-19       941      6
>> > > 891       92            970      27       0-20       940      7
>> > > 891       120           966      28       0-22       937      9
>> > > 890       112           970      28       0-21       940      8
>> > > 903       70            970      28       0-16       935      9
>> > >
>> > > Note: all RPM values were categorized as Low.
>> > >
>> > > networkQuality downstream capacities are always on the low side compared to
>> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla
>> > > achieved. I think that a longer test duration might be needed to achieve the
>> > > actual 1Gbps capacity with networkQuality; intermediate values observed were
>> > > certainly headed in the right direction. (I recently upgraded to Monterey
>> > 12.6
>> > > on my MacBook, so should have the latest version.)
>> > >
>> > > Also, as Sebastian Moeller's message to the list reminded me, I should have
>> > > run the tests with the -v option to help with comparisons. I'll repeat this
>> > > test when I can make time.
>> > >
>> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and
>> > > the range of variation above the minimum (RTTVarRnge) add-up to very
>> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access
>> > and
>> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
>> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were
>> > run
>> > > on an Ubuntu VM in my MacBook.
>> > >
>> > > The big disappointment was that the Ookla desktop app I updated over the
>> > > weekend did not include the new responsiveness metric! I included the ping
>> > > results anyway, and it was clearly using a server in the nearby area.
>> > >
>> > > So, I have some more work to do, but I hope this is interesting-enough to
>> > > start some comparison discussions, and bring-out some suggestions.
>> > >
>> > > happy testing all,
>> > > Al
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > ippm mailing list
>> > > ippm@ietf.org
>> > >
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > >
>> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>> >
>> > _______________________________________________
>> > ippm mailing list
>> > ippm@ietf.org
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>> 
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm