* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
[not found] ` <CH0PR02MB7980D3036BF700A074D902A1D3379@CH0PR02MB7980.namprd02.prod.outlook.com>
@ 2022-10-31 16:52 ` Dave Taht
2022-10-31 18:52 ` rjmcmahon
2022-10-31 20:40 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics MORTON JR., AL
0 siblings, 2 replies; 29+ messages in thread
From: Dave Taht @ 2022-10-31 16:52 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: ippm, Rpm
Thank you very much for the steer to RFC9097. I'd completely missed that.
On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
>
> (astute readers may have guessed that I pressed "send" too soon on previous message...)
>
> I also conducted upstream tests this time, here are the results:
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
>
> Net Qual UDPST (RFC9097) Ookla
> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap Ping(no load)
> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 8
> 22 281 m 214ms 8ms 27 (52) 25 5-248 22 8
> 22 290 m 207ms 8ms 27 (55) 28 0-253 22 9
> 21 330 m 182ms 11ms 23 (44) 28 0-255 22 7
> 22 334 m 180ms 9ms 33 (56) 25 0-255 22 9
>
> The Upstream capacity measurements reflect an interesting feature that we can reliably and repeatably measure with UDPST. The first ~3 seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this behavior in its 1 second sub-interval measurements and has a bimodal reporting option that divides the complete measurement interval in two time intervals to report an initial (turbo) max capacity and a steady-state max capacity for the later intervals. The UDPST capacity results present both measurements: steady-state first.
Certainly we can expect bi-model distributions from many ISPs, as, for
one thing, the "speedboost" concept remains popular, except that it's
misnamed, as it should be called speed-subtract or speed-lose. Worse,
it is often configured "sneakily", in that it doesn't kick in for the
typical observed duration of the test, for some, they cut the
available bandwidth about 20s in, others, 1 or 5 minutes.
One of my biggest issues with the rpm spec so far is that it should,
at least, sometimes, run randomly longer than the overly short
interval it runs for and the tools also allow for manual override of length.
we caught a lot of tomfoolery with flent's rrul test running by default for 1m.
Also, AQMs on the path can take a while to find the optimal drop or mark rate.
>
> The capacity processing in networkQuality and Ookla appear to report the steady-state result.
Ookla used to basically report the last result. Also it's not a good
indicator of web traffic behavior at all, watching the curve
go up much more slowly in their test on say, fiber 2ms, vs starlink, (40ms)....
So adding another mode - how quickly is peak bandwidth actually
reached, would be nice.
I haven't poked into the current iteration of the goresponsiveness
test at all: https://github.com/network-quality/goresponsiveness it
would be good to try collecting more statistics and histograms and
methods of analyzing the data in that libre-source version.
How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
>I watched the upstream capacity measurements on the Ookla app, and could easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of the test which determined the final result.
I tend to get upset when I see ookla's new test flash a peak result in
the seconds and then settle on some lower number somehow.
So far as I know they are only sampling the latency every 250ms.
>
> The working latency is about 200ms in networkQuality and about 280ms as measured by UDPST (RFC9097). Note that the networkQuality minimum delay is ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference in working latency. Also, we used the very dynamic Type C load adjustment/search algorithm in UDPST during all of this testing, which could explain the higher working latency to some degree.
>
> So, it's worth noting that the measurements needed for assessing working latency/responsiveness are available in the UDPST utility, and that the UDPST measurements are conducted on UDP transport (used by a growing fraction of Internet traffic).
Thx, didn't know of this work til now!
have you tried irtt?
>
> comments welcome of course,
> Al
>
> > -----Original Message-----
> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > Sent: Sunday, October 30, 2022 8:09 PM
> > To: ippm@ietf.org
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> >
> > Hi again RPM friends and IPPM'ers,
> >
> > As promised, I repeated the tests shared last week, this time using both the
> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > followed Sebastian's calculations as well.
> >
> > Working Latency & Capacity Summary
> >
> > Net Qual UDPST Ookla
> > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> > Ping(no load)
> > 885 916 m 66ms 8ms 970 28 0-20 940 8
> > 888 1355 h 44ms 8ms 966 28 0-23 940 8
> > 891 1109 h 54ms 8ms 968 27 0-19 940 9
> > 887 1141 h 53ms 11ms 966 27 0-18 937 7
> > 884 1151 h 52ms 9ms 968 28 0-20 937 9
> >
> > With the sequential test option, I noticed that networkQuality achieved nearly
> > the maximum capacity reported almost immediately at the start of a test.
> > However, the reported capacities are low by about 60Mbps, especially when
> > compared to the Ookla TCP measurements.
> >
> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the
> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality
> > RPM measurements were categorized as "High". There doesn't seem to be much
> > buffering in the downstream direction.
> >
> >
> >
> > > -----Original Message-----
> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > Sent: Monday, October 24, 2022 6:36 PM
> > > To: ippm@ietf.org
> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > metrics
> > >
> > >
> > > Hi RPM friends and IPPM'ers,
> > >
> > > I was wondering what a comparison of some of the "working latency" metrics
> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the
> > > downlink provisioned for 1Gbps.
> > >
> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > Speedtest
> > > with as similar connectivity as possible (but we know that the traffic will
> > > diverge to different servers and we can't change that aspect).
> > >
> > > Here's a quick summary of yesterday's results:
> > >
> > > Working Latency & Capacity Summary
> > >
> > > Net Qual UDPST Ookla
> > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no load)
> > > 878 62 970 28 0-19 941 6
> > > 891 92 970 27 0-20 940 7
> > > 891 120 966 28 0-22 937 9
> > > 890 112 970 28 0-21 940 8
> > > 903 70 970 28 0-16 935 9
> > >
> > > Note: all RPM values were categorized as Low.
> > >
> > > networkQuality downstream capacities are always on the low side compared to
> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla
> > > achieved. I think that a longer test duration might be needed to achieve the
> > > actual 1Gbps capacity with networkQuality; intermediate values observed were
> > > certainly headed in the right direction. (I recently upgraded to Monterey
> > 12.6
> > > on my MacBook, so should have the latest version.)
> > >
> > > Also, as Sebastian Moeller's message to the list reminded me, I should have
> > > run the tests with the -v option to help with comparisons. I'll repeat this
> > > test when I can make time.
> > >
> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and
> > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access
> > and
> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were
> > run
> > > on an Ubuntu VM in my MacBook.
> > >
> > > The big disappointment was that the Ookla desktop app I updated over the
> > > weekend did not include the new responsiveness metric! I included the ping
> > > results anyway, and it was clearly using a server in the nearby area.
> > >
> > > So, I have some more work to do, but I hope this is interesting-enough to
> > > start some comparison discussions, and bring-out some suggestions.
> > >
> > > happy testing all,
> > > Al
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >
> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 16:52 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics Dave Taht
@ 2022-10-31 18:52 ` rjmcmahon
2022-10-31 22:08 ` MORTON JR., AL
2022-10-31 20:40 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics MORTON JR., AL
1 sibling, 1 reply; 29+ messages in thread
From: rjmcmahon @ 2022-10-31 18:52 UTC (permalink / raw)
To: Dave Taht; +Cc: MORTON JR., AL, Rpm, ippm
Would it be possible to get some iperf 2 bounceback test results too?
https://sourceforge.net/projects/iperf2/
Also, for the hunt algo, maybe use TCP first to get a starting point and
then hunt? Just a thought.
Thanks,
Bob
> Thank you very much for the steer to RFC9097. I'd completely missed
> that.
>
> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> wrote:
>>
>> (astute readers may have guessed that I pressed "send" too soon on
>> previous message...)
>>
>> I also conducted upstream tests this time, here are the results:
>> (capacity in Mbps, delays in ms, h and m are RPM categories, High and
>> Medium)
>>
>> Net Qual UDPST (RFC9097) Ookla
>> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
>> Ping(no load)
>> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
>> 8
>> 22 281 m 214ms 8ms 27 (52) 25 5-248 22
>> 8
>> 22 290 m 207ms 8ms 27 (55) 28 0-253 22
>> 9
>> 21 330 m 182ms 11ms 23 (44) 28 0-255 22
>> 7
>> 22 334 m 180ms 9ms 33 (56) 25 0-255 22
>> 9
>>
>> The Upstream capacity measurements reflect an interesting feature that
>> we can reliably and repeatably measure with UDPST. The first ~3
>> seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST
>> displays this behavior in its 1 second sub-interval measurements and
>> has a bimodal reporting option that divides the complete measurement
>> interval in two time intervals to report an initial (turbo) max
>> capacity and a steady-state max capacity for the later intervals. The
>> UDPST capacity results present both measurements: steady-state first.
>
> Certainly we can expect bi-model distributions from many ISPs, as, for
> one thing, the "speedboost" concept remains popular, except that it's
> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> it is often configured "sneakily", in that it doesn't kick in for the
> typical observed duration of the test, for some, they cut the
> available bandwidth about 20s in, others, 1 or 5 minutes.
>
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short
> interval it runs for and the tools also allow for manual override of
> length.
>
> we caught a lot of tomfoolery with flent's rrul test running by default
> for 1m.
>
> Also, AQMs on the path can take a while to find the optimal drop or
> mark rate.
>
>>
>> The capacity processing in networkQuality and Ookla appear to report
>> the steady-state result.
>
> Ookla used to basically report the last result. Also it's not a good
> indicator of web traffic behavior at all, watching the curve
> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> (40ms)....
>
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
>
> I haven't poked into the current iteration of the goresponsiveness
> test at all: https://github.com/network-quality/goresponsiveness it
> would be good to try collecting more statistics and histograms and
> methods of analyzing the data in that libre-source version.
>
> How does networkQuality compare vs a vs your tool vs a vs
> goresponsiveness?
>
>> I watched the upstream capacity measurements on the Ookla app, and
>> could easily see the initial rise to 40-50Mbps, then the drop to
>> ~22Mbps for most of the test which determined the final result.
>
> I tend to get upset when I see ookla's new test flash a peak result in
> the seconds and then settle on some lower number somehow.
> So far as I know they are only sampling the latency every 250ms.
>
>>
>> The working latency is about 200ms in networkQuality and about 280ms
>> as measured by UDPST (RFC9097). Note that the networkQuality minimum
>> delay is ~20ms lower than the UDPST RTTmin, so this accounts for some
>> of the difference in working latency. Also, we used the very dynamic
>> Type C load adjustment/search algorithm in UDPST during all of this
>> testing, which could explain the higher working latency to some
>> degree.
>>
>> So, it's worth noting that the measurements needed for assessing
>> working latency/responsiveness are available in the UDPST utility, and
>> that the UDPST measurements are conducted on UDP transport (used by a
>> growing fraction of Internet traffic).
>
> Thx, didn't know of this work til now!
>
> have you tried irtt?
>
>>
>> comments welcome of course,
>> Al
>>
>> > -----Original Message-----
>> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > Sent: Sunday, October 30, 2022 8:09 PM
>> > To: ippm@ietf.org
>> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> > metrics
>> >
>> >
>> > Hi again RPM friends and IPPM'ers,
>> >
>> > As promised, I repeated the tests shared last week, this time using both the
>> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
>> > followed Sebastian's calculations as well.
>> >
>> > Working Latency & Capacity Summary
>> >
>> > Net Qual UDPST Ookla
>> > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
>> > Ping(no load)
>> > 885 916 m 66ms 8ms 970 28 0-20 940 8
>> > 888 1355 h 44ms 8ms 966 28 0-23 940 8
>> > 891 1109 h 54ms 8ms 968 27 0-19 940 9
>> > 887 1141 h 53ms 11ms 966 27 0-18 937 7
>> > 884 1151 h 52ms 9ms 968 28 0-20 937 9
>> >
>> > With the sequential test option, I noticed that networkQuality achieved nearly
>> > the maximum capacity reported almost immediately at the start of a test.
>> > However, the reported capacities are low by about 60Mbps, especially when
>> > compared to the Ookla TCP measurements.
>> >
>> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the
>> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality
>> > RPM measurements were categorized as "High". There doesn't seem to be much
>> > buffering in the downstream direction.
>> >
>> >
>> >
>> > > -----Original Message-----
>> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > Sent: Monday, October 24, 2022 6:36 PM
>> > > To: ippm@ietf.org
>> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
>> > > metrics
>> > >
>> > >
>> > > Hi RPM friends and IPPM'ers,
>> > >
>> > > I was wondering what a comparison of some of the "working latency" metrics
>> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the
>> > > downlink provisioned for 1Gbps.
>> > >
>> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
>> > Speedtest
>> > > with as similar connectivity as possible (but we know that the traffic will
>> > > diverge to different servers and we can't change that aspect).
>> > >
>> > > Here's a quick summary of yesterday's results:
>> > >
>> > > Working Latency & Capacity Summary
>> > >
>> > > Net Qual UDPST Ookla
>> > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no load)
>> > > 878 62 970 28 0-19 941 6
>> > > 891 92 970 27 0-20 940 7
>> > > 891 120 966 28 0-22 937 9
>> > > 890 112 970 28 0-21 940 8
>> > > 903 70 970 28 0-16 935 9
>> > >
>> > > Note: all RPM values were categorized as Low.
>> > >
>> > > networkQuality downstream capacities are always on the low side compared to
>> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla
>> > > achieved. I think that a longer test duration might be needed to achieve the
>> > > actual 1Gbps capacity with networkQuality; intermediate values observed were
>> > > certainly headed in the right direction. (I recently upgraded to Monterey
>> > 12.6
>> > > on my MacBook, so should have the latest version.)
>> > >
>> > > Also, as Sebastian Moeller's message to the list reminded me, I should have
>> > > run the tests with the -v option to help with comparisons. I'll repeat this
>> > > test when I can make time.
>> > >
>> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and
>> > > the range of variation above the minimum (RTTVarRnge) add-up to very
>> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access
>> > and
>> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
>> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were
>> > run
>> > > on an Ubuntu VM in my MacBook.
>> > >
>> > > The big disappointment was that the Ookla desktop app I updated over the
>> > > weekend did not include the new responsiveness metric! I included the ping
>> > > results anyway, and it was clearly using a server in the nearby area.
>> > >
>> > > So, I have some more work to do, but I hope this is interesting-enough to
>> > > start some comparison discussions, and bring-out some suggestions.
>> > >
>> > > happy testing all,
>> > > Al
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > ippm mailing list
>> > > ippm@ietf.org
>> > >
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > >
>> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>> >
>> > _______________________________________________
>> > ippm mailing list
>> > ippm@ietf.org
>> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>>
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 16:52 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics Dave Taht
2022-10-31 18:52 ` rjmcmahon
@ 2022-10-31 20:40 ` MORTON JR., AL
2022-10-31 23:30 ` Dave Taht
1 sibling, 1 reply; 29+ messages in thread
From: MORTON JR., AL @ 2022-10-31 20:40 UTC (permalink / raw)
To: Dave Taht; +Cc: ippm, Rpm
Thanks for your read/reply/suggestions, Dave!
Allow me to pull a few of your comments forward to try for a more cogent reply.
Dave wrote:
> Thank you very much for the steer to RFC9097. I'd completely missed that.
You're quite welcome! We all have a hand on the elephant with eyes closed, and only learn the whole story when we talk to each other...
The repo for UDPST is here: https://github.com/BroadbandForum/obudpst
We are also working to standardize the protocol the UDPST uses to measure RFC 9097:
https://datatracker.ietf.org/doc/html/draft-ietf-ippm-capacity-protocol-03
and potentially many aspects of network and application performance.
Dave wrote:
> Certainly we can expect bi-modal distributions ... should be called ... speed-lose.
Agreed. I'm glad we added the bimodal analysis feature in UDPST. It works with our max test duration (60s, we didn't want people to run capacity tests ad nauseum), but we won't be able to detect speed-loose beyond that.
Dave wrote:
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short interval <now> ...
We don't have adaptable duration either. Another perspective on duration comes from folks who test paths with mobile access: they prefer 5-7 second duration, and the Type C algo helps.
Dave wrote:
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
I think we report this in our prolific JSON-formatted output (#sub-interval with the Max Cap).
Dave wrote:
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
I'll try to install goresponsiveness later this week, so that we have a view of this.
Dave wrote:
> have you tried irtt? (https://github.com/heistp/irtt )
I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481
irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...
thanks again, Dave.
Al
> -----Original Message-----
> From: Dave Taht <dave.taht@gmail.com>
> Sent: Monday, October 31, 2022 12:52 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
>
> Thank you very much for the steer to RFC9097. I'd completely missed that.
>
> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> >
> > (astute readers may have guessed that I pressed "send" too soon on previous
> message...)
> >
> > I also conducted upstream tests this time, here are the results:
> > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
> >
> > Net Qual UDPST (RFC9097) Ookla
> > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> Ping(no load)
> > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 8
> > 22 281 m 214ms 8ms 27 (52) 25 5-248 22 8
> > 22 290 m 207ms 8ms 27 (55) 28 0-253 22 9
> > 21 330 m 182ms 11ms 23 (44) 28 0-255 22 7
> > 22 334 m 180ms 9ms 33 (56) 25 0-255 22 9
> >
> > The Upstream capacity measurements reflect an interesting feature that we
> can reliably and repeatably measure with UDPST. The first ~3 seconds of
> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> behavior in its 1 second sub-interval measurements and has a bimodal reporting
> option that divides the complete measurement interval in two time intervals to
> report an initial (turbo) max capacity and a steady-state max capacity for the
> later intervals. The UDPST capacity results present both measurements: steady-
> state first.
>
> Certainly we can expect bi-model distributions from many ISPs, as, for
> one thing, the "speedboost" concept remains popular, except that it's
> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> it is often configured "sneakily", in that it doesn't kick in for the
> typical observed duration of the test, for some, they cut the
> available bandwidth about 20s in, others, 1 or 5 minutes.
>
> One of my biggest issues with the rpm spec so far is that it should,
> at least, sometimes, run randomly longer than the overly short
> interval it runs for and the tools also allow for manual override of length.
>
> we caught a lot of tomfoolery with flent's rrul test running by default for
> 1m.
>
> Also, AQMs on the path can take a while to find the optimal drop or mark rate.
>
> >
> > The capacity processing in networkQuality and Ookla appear to report the
> steady-state result.
>
> Ookla used to basically report the last result. Also it's not a good
> indicator of web traffic behavior at all, watching the curve
> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> (40ms)....
>
> So adding another mode - how quickly is peak bandwidth actually
> reached, would be nice.
>
> I haven't poked into the current iteration of the goresponsiveness
> test at all: https://urldefense.com/v3/__https://github.com/network-
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> would be good to try collecting more statistics and histograms and
> methods of analyzing the data in that libre-source version.
>
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
>
> >I watched the upstream capacity measurements on the Ookla app, and could
> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> the test which determined the final result.
>
> I tend to get upset when I see ookla's new test flash a peak result in
> the seconds and then settle on some lower number somehow.
> So far as I know they are only sampling the latency every 250ms.
>
> >
> > The working latency is about 200ms in networkQuality and about 280ms as
> measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> in working latency. Also, we used the very dynamic Type C load
> adjustment/search algorithm in UDPST during all of this testing, which could
> explain the higher working latency to some degree.
> >
> > So, it's worth noting that the measurements needed for assessing working
> latency/responsiveness are available in the UDPST utility, and that the UDPST
> measurements are conducted on UDP transport (used by a growing fraction of
> Internet traffic).
>
> Thx, didn't know of this work til now!
>
> have you tried irtt?
>
> >
> > comments welcome of course,
> > Al
> >
> > > -----Original Message-----
> > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > Sent: Sunday, October 30, 2022 8:09 PM
> > > To: ippm@ietf.org
> > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > metrics
> > >
> > >
> > > Hi again RPM friends and IPPM'ers,
> > >
> > > As promised, I repeated the tests shared last week, this time using both
> the
> > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > followed Sebastian's calculations as well.
> > >
> > > Working Latency & Capacity Summary
> > >
> > > Net Qual UDPST Ookla
> > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> > > Ping(no load)
> > > 885 916 m 66ms 8ms 970 28 0-20 940 8
> > > 888 1355 h 44ms 8ms 966 28 0-23 940 8
> > > 891 1109 h 54ms 8ms 968 27 0-19 940 9
> > > 887 1141 h 53ms 11ms 966 27 0-18 937 7
> > > 884 1151 h 52ms 9ms 968 28 0-20 937 9
> > >
> > > With the sequential test option, I noticed that networkQuality achieved
> nearly
> > > the maximum capacity reported almost immediately at the start of a test.
> > > However, the reported capacities are low by about 60Mbps, especially when
> > > compared to the Ookla TCP measurements.
> > >
> > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> the
> > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> networkQuality
> > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > buffering in the downstream direction.
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > To: ippm@ietf.org
> > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > metrics
> > > >
> > > >
> > > > Hi RPM friends and IPPM'ers,
> > > >
> > > > I was wondering what a comparison of some of the "working latency"
> metrics
> > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> the
> > > > downlink provisioned for 1Gbps.
> > > >
> > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > Speedtest
> > > > with as similar connectivity as possible (but we know that the traffic
> will
> > > > diverge to different servers and we can't change that aspect).
> > > >
> > > > Here's a quick summary of yesterday's results:
> > > >
> > > > Working Latency & Capacity Summary
> > > >
> > > > Net Qual UDPST Ookla
> > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no
> load)
> > > > 878 62 970 28 0-19 941 6
> > > > 891 92 970 27 0-20 940 7
> > > > 891 120 966 28 0-22 937 9
> > > > 890 112 970 28 0-21 940 8
> > > > 903 70 970 28 0-16 935 9
> > > >
> > > > Note: all RPM values were categorized as Low.
> > > >
> > > > networkQuality downstream capacities are always on the low side compared
> to
> > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> Ookla
> > > > achieved. I think that a longer test duration might be needed to achieve
> the
> > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> were
> > > > certainly headed in the right direction. (I recently upgraded to
> Monterey
> > > 12.6
> > > > on my MacBook, so should have the latest version.)
> > > >
> > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> have
> > > > run the tests with the -v option to help with comparisons. I'll repeat
> this
> > > > test when I can make time.
> > > >
> > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> and
> > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> access
> > > and
> > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> were
> > > run
> > > > on an Ubuntu VM in my MacBook.
> > > >
> > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > weekend did not include the new responsiveness metric! I included the
> ping
> > > > results anyway, and it was clearly using a server in the nearby area.
> > > >
> > > > So, I have some more work to do, but I hope this is interesting-enough
> to
> > > > start some comparison discussions, and bring-out some suggestions.
> > > >
> > > > happy testing all,
> > > > Al
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >
> > >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > T!g-
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> KXU$
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> song-activity-6981366665607352320-
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> c48iQFub34zz4iE$
> Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 18:52 ` rjmcmahon
@ 2022-10-31 22:08 ` MORTON JR., AL
2022-10-31 22:44 ` rjmcmahon
2022-11-01 20:15 ` [Rpm] lightweight active sensing of bandwidth and buffering Dave Taht
0 siblings, 2 replies; 29+ messages in thread
From: MORTON JR., AL @ 2022-10-31 22:08 UTC (permalink / raw)
To: rjmcmahon, Dave Taht; +Cc: Rpm, ippm
> -----Original Message-----
> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
> Sent: Monday, October 31, 2022 2:53 PM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: MORTON JR., AL <acmorton@att.com>; Rpm <rpm@lists.bufferbloat.net>;
> ippm@ietf.org
> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> Latency" metrics
>
> Would it be possible to get some iperf 2 bounceback test results too?
>
> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
> gmYpYN3pBO-aWMfjDRdVRFQ20aHQ5nDHOhEVY1y-
> MkFFyH8YmM4wf8cEtaxzvcwTMaCaJOCNRBtj0tnz9A$
>
> Also, for the hunt algo, maybe use TCP first to get a starting point and
> then hunt? Just a thought.
>
> Thanks,
> Bob
> >
<snip>
Thanks for your suggestion, Bob, and it's nice to meet you!
I was only familiar with the "old" iperf2 at:
https://iperf.fr/iperf-doc.php
before your message and URL arrived (yet another segment of the elephant!).
I didn't quickly find whether your code will run on linux (ubuntu for me). I suppose I use the .tar.gz and compile locally. Let me know, I've got some other tools to check-out first.
You suggest the bounceback test option (I found it on the man page), but there are lots of options and six versions of bounceback! Based on the results I already reported, can you suggest a specific version of bounceback and a set of options that would be a good first try?
(see the man page at https://iperf2.sourceforge.io/iperf-manpage.html )
Regarding our hunt algorithms (search for max), the new Type C algo (in release 7.5.0) locates the Max very quickly. I showed a comparison of our Type B and Type C search algorithms on a slide at the IPPM meeting in July. See Slide 10:
https://datatracker.ietf.org/meeting/114/materials/slides-114-ippm-sec-dir-review-discussion-test-protocol-for-one-way-ip-capacity-measurement-00
The measured capacity reaches 1 Gbps in about 1 second with the Type C algorithm, without choosing or testing for a starting point (our default starting point rate is ~500kbps, very low to accommodate "any" subscribed rate).
regards, and thanks again,
Al
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 22:08 ` MORTON JR., AL
@ 2022-10-31 22:44 ` rjmcmahon
2022-11-01 20:15 ` [Rpm] lightweight active sensing of bandwidth and buffering Dave Taht
1 sibling, 0 replies; 29+ messages in thread
From: rjmcmahon @ 2022-10-31 22:44 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: Dave Taht, Rpm, ippm
One can download iperf version 2.1.8 tarball or use git to get the
latest and compile from master. Sourceforge supports both.
https://sourceforge.net/projects/iperf2/
A simple test is to use the --bounceback and then, with load, add
--bounceback-congest (see below) or maybe use one way congestion vs the
default of full-duplex.
Build instructions is in the INSTALL file. Should work on linux,
windows, BSD, etc.
I think with WiFi the distributions may be more than bi-modal and one
second decisions may be too short. It doesn't mean that the traffic has
to be active and at capacity per the full decision interval, rather the
forwarding/phy state machines & other logic may need more than one
second to find their local optimums. Also, maybe some sort of online
mean & min shift detections could help? Or one can use the inP metric
too. This is calculated per Little's law. Though OWD requires clock sync
indicated as being done to iperf via the --trip-times option.
I added your delay docs into the iperf2 docs directory. Thanks for
mentioning them. They're helpful.
https://sourceforge.net/p/iperf2/code/ci/master/tree/doc/delay_docs/
Bob
[root@ctrl1fc35 ~]# iperf -c 192.168.1.231 --bounceback
------------------------------------------------------------
Client connecting to 192.168.1.231, TCP port 5001 with pid 717413 (1
flows)
Write buffer size: 100 Byte
Bursting: 100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.15%enp2s0 port 43538 connected with 192.168.1.231
port 5001 (bb w/quickack len/hold=100/0) (sock=3)
(icwnd/mss/irtt=14/1448/4442) (ct=4.55 ms) on 2022-10-31 15:25:56 (PDT)
[ ID] Interval Transfer Bandwidth BB
cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
[ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.802/2.008/5.765/1.243 ms 0 14K/3958 us 263 rps
[ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.836/2.179/5.112/0.884 ms 0 14K/3727 us 261 rps
[ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.941/3.576/4.221/0.159 ms 0 14K/3800 us 254 rps
[ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec
10=4.042/3.790/4.847/0.295 ms 0 14K/3853 us 247 rps
[ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.930/3.699/4.162/0.123 ms 0 14K/3834 us 254 rps
[ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec
10=4.037/3.752/4.612/0.238 ms 0 14K/3858 us 248 rps
[ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.941/3.827/4.045/0.078 ms 0 14K/3833 us 254 rps
[ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec
10=4.042/3.805/4.778/0.277 ms 0 14K/3869 us 247 rps
[ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec
10=3.935/3.749/4.099/0.127 ms 0 14K/3822 us 254 rps
[ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec
10=4.041/3.783/4.885/0.320 ms 0 14K/3863 us 247 rps
[ 1] 0.00-10.02 sec 19.5 KBytes 16.0 Kbits/sec
100=3.955/2.008/5.765/0.503 ms 0 14K/3872 us 253 rps
[ 1] 0.00-10.02 sec BB8(f)-PDF:
bin(w=100us):cnt(100)=21:1,22:1,24:1,28:1,30:1,32:1,36:2,37:2,38:6,39:20,40:24,41:25,42:4,43:1,44:1,47:1,48:1,49:3,51:1,52:2,58:1
(5.00/95.00/99.7%=30/49/58,Outliers=0,obl/obu=0/0)
[root@ctrl1fc35 ~]# iperf -c 192.168.1.231 --bounceback
--bounceback-congest
------------------------------------------------------------
Client connecting to 192.168.1.231, TCP port 5001 with pid 717416 (1
flows)
Write buffer size: 100 Byte
Bursting: 100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 2] local 192.168.1.15%enp2s0 port 43540 connected with 192.168.1.231
port 5001 (full-duplex) (sock=4) (qack) (icwnd/mss/irtt=14/1448/4000)
(ct=4.10 ms) on 2022-10-31 15:26:15 (PDT)
[ 1] local 192.168.1.15%enp2s0 port 43542 connected with 192.168.1.231
port 5001 (bb w/quickack len/hold=100/0) (sock=3)
(icwnd/mss/irtt=14/1448/4003) (ct=4.10 ms) on 2022-10-31 15:26:15 (PDT)
[ ID] Interval Transfer Bandwidth BB
cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
[ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
10=13.770/1.855/50.097/15.368 ms 0 14K/14387 us 73 rps
[ ID] Interval Transfer Bandwidth Write/Err Rtry
Cwnd/RTT(var) NetPwr
[ 2] 0.00-1.00 sec 45.2 MBytes 379 Mbits/sec 474213/0 0
7611K/23840(768) us 1989
[ ID] Interval Transfer Bandwidth Reads=Dist
[ *2] 0.00-1.00 sec 29.8 MBytes 250 Mbits/sec
312674=78:10:29:14:57:17:87:14
[ ID] Interval Transfer Bandwidth
[FD2] 0.00-1.00 sec 75.0 MBytes 629 Mbits/sec
[ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
10=25.213/15.954/48.097/9.889 ms 0 14K/23702 us 40 rps
[ *2] 1.00-2.00 sec 45.6 MBytes 383 Mbits/sec
478326=4:19:3:10:16:12:12:13
[ 2] 1.00-2.00 sec 56.4 MBytes 473 Mbits/sec 591760/0 0
7611K/14112(1264) us 4193
[FD2] 1.00-2.00 sec 102 MBytes 856 Mbits/sec
[ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
10=25.220/11.400/51.321/14.320 ms 0 14K/25023 us 40 rps
[ *2] 2.00-3.00 sec 39.1 MBytes 328 Mbits/sec
410533=18:32:13:15:25:27:7:23
[ 2] 2.00-3.00 sec 57.2 MBytes 480 Mbits/sec 599942/0 0
7611K/14356(920) us 4179
[FD2] 2.00-3.00 sec 96.4 MBytes 808 Mbits/sec
[ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec
10=16.023/5.588/50.304/12.940 ms 0 14K/16648 us 62 rps
[ *2] 3.00-4.00 sec 35.9 MBytes 301 Mbits/sec
376791=17:34:11:15:33:37:22:26
[ 2] 3.00-4.00 sec 56.6 MBytes 475 Mbits/sec 593663/0 0
7611K/45828(2395) us 1295
[FD2] 3.00-4.00 sec 92.5 MBytes 776 Mbits/sec
[ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec
10=12.445/7.353/19.663/3.601 ms 0 14K/13359 us 80 rps
[ *2] 4.00-5.00 sec 42.4 MBytes 356 Mbits/sec
444818=29:34:8:15:21:26:15:20
[ 2] 4.00-5.00 sec 54.9 MBytes 461 Mbits/sec 575659/0 0
7611K/13205(2359) us 4359
[FD2] 4.00-5.00 sec 97.3 MBytes 816 Mbits/sec
[ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec
10=12.092/6.493/16.031/3.279 ms 0 14K/12345 us 83 rps
[ *2] 5.00-6.00 sec 37.9 MBytes 318 Mbits/sec
397808=10:22:20:28:24:20:11:10
[ 2] 5.00-6.00 sec 56.9 MBytes 477 Mbits/sec 596477/0 0
7611K/15607(167) us 3822
[FD2] 5.00-6.00 sec 94.8 MBytes 795 Mbits/sec
[ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec
10=13.916/9.555/19.109/2.876 ms 0 14K/13690 us 72 rps
[ *2] 6.00-7.00 sec 46.0 MBytes 386 Mbits/sec
482572=25:46:13:18:34:35:27:32
[ 2] 6.00-7.00 sec 57.2 MBytes 480 Mbits/sec 600055/0 0
7611K/14175(669) us 4233
[FD2] 6.00-7.00 sec 103 MBytes 866 Mbits/sec
[ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec
10=13.125/4.997/19.527/4.239 ms 0 14K/13695 us 76 rps
[ *2] 7.00-8.00 sec 40.1 MBytes 336 Mbits/sec
420304=21:30:11:6:5:17:14:18
[ 2] 7.00-8.00 sec 56.0 MBytes 470 Mbits/sec 587082/0 0
7611K/24914(3187) us 2356
[FD2] 7.00-8.00 sec 96.1 MBytes 806 Mbits/sec
[ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec
10=8.221/2.939/17.274/4.419 ms 0 14K/9861 us 122 rps
[ *2] 8.00-9.00 sec 33.6 MBytes 282 Mbits/sec
352847=17:35:16:19:15:27:17:17
[ 2] 8.00-9.00 sec 56.2 MBytes 471 Mbits/sec 589281/0 0
7611K/13206(403) us 4462
[FD2] 8.00-9.00 sec 89.8 MBytes 754 Mbits/sec
[ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec
10=10.896/6.822/14.941/2.788 ms 0 14K/10419 us 92 rps
[ *2] 9.00-10.00 sec 36.4 MBytes 306 Mbits/sec
382097=18:48:22:25:29:32:28:24
[ 2] 9.00-10.00 sec 57.2 MBytes 480 Mbits/sec 599784/0 0
7611K/15965(850) us 3757
[FD2] 9.00-10.00 sec 93.6 MBytes 785 Mbits/sec
[ 2] 0.00-10.00 sec 554 MBytes 465 Mbits/sec 5807917/0 0
7611K/15965(850) us 3637
[ *2] 0.00-10.07 sec 391 MBytes 326 Mbits/sec
4099553=239:319:147:168:261:251:241:200
[FD2] 0.00-10.07 sec 945 MBytes 787 Mbits/sec
[ 1] 0.00-10.07 sec 21.5 KBytes 17.5 Kbits/sec
110=14.330/1.855/51.321/9.904 ms 0 14K/6896 us 70 rps
[ 1] 0.00-10.07 sec BB8(f)-PDF:
bin(w=100us):cnt(110)=19:1,30:1,33:2,34:1,40:1,45:2,47:1,50:1,51:1,55:1,56:1,58:1,64:1,65:1,66:1,67:1,69:1,70:2,71:1,74:1,77:1,79:2,83:2,84:1,90:1,93:2,96:3,97:2,99:1,100:1,103:2,105:1,111:1,112:1,113:2,114:2,115:1,116:1,118:2,119:1,122:2,123:1,124:1,127:1,129:2,130:3,133:1,143:1,144:1,145:1,147:3,149:4,150:2,156:1,157:2,158:1,160:1,161:1,162:1,163:1,164:1,167:1,173:1,181:1,182:2,186:1,192:1,196:1,197:1,199:1,218:1,221:1,235:1,242:1,246:1,278:1,281:1,282:1,286:1,342:1,481:1,496:1,501:1,504:1,514:1
(5.00/95.00/99.7%=40/342/514,Outliers=0,obl/obu=0/0)
[ CT] final connect times (min/avg/max/stdev) = 4.104/4.104/4.104/0.000
ms (tot/err) = 2/0>> -----Original Message-----
>> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
>> Sent: Monday, October 31, 2022 2:53 PM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: MORTON JR., AL <acmorton@att.com>; Rpm
>> <rpm@lists.bufferbloat.net>;
>> ippm@ietf.org
>> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of
>> "Working
>> Latency" metrics
>>
>> Would it be possible to get some iperf 2 bounceback test results too?
>>
>> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
>> gmYpYN3pBO-aWMfjDRdVRFQ20aHQ5nDHOhEVY1y-
>> MkFFyH8YmM4wf8cEtaxzvcwTMaCaJOCNRBtj0tnz9A$
>>
>> Also, for the hunt algo, maybe use TCP first to get a starting point
>> and
>> then hunt? Just a thought.
>>
>> Thanks,
>> Bob
>> >
> <snip>
>
> Thanks for your suggestion, Bob, and it's nice to meet you!
> I was only familiar with the "old" iperf2 at:
> https://iperf.fr/iperf-doc.php
> before your message and URL arrived (yet another segment of the
> elephant!).
>
> I didn't quickly find whether your code will run on linux (ubuntu for
> me). I suppose I use the .tar.gz and compile locally. Let me know,
> I've got some other tools to check-out first.
>
> You suggest the bounceback test option (I found it on the man page),
> but there are lots of options and six versions of bounceback! Based
> on the results I already reported, can you suggest a specific version
> of bounceback and a set of options that would be a good first try?
> (see the man page at https://iperf2.sourceforge.io/iperf-manpage.html )
>
> Regarding our hunt algorithms (search for max), the new Type C algo
> (in release 7.5.0) locates the Max very quickly. I showed a comparison
> of our Type B and Type C search algorithms on a slide at the IPPM
> meeting in July. See Slide 10:
> https://datatracker.ietf.org/meeting/114/materials/slides-114-ippm-sec-dir-review-discussion-test-protocol-for-one-way-ip-capacity-measurement-00
>
> The measured capacity reaches 1 Gbps in about 1 second with the Type C
> algorithm, without choosing or testing for a starting point (our
> default starting point rate is ~500kbps, very low to accommodate "any"
> subscribed rate).
>
> regards, and thanks again,
> Al
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 20:40 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics MORTON JR., AL
@ 2022-10-31 23:30 ` Dave Taht
2022-11-01 4:21 ` Dave Taht
0 siblings, 1 reply; 29+ messages in thread
From: Dave Taht @ 2022-10-31 23:30 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: ippm, Rpm
On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> > have you tried irtt? (https://github.com/heistp/irtt )
> I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481
irtt was meant to simulate high speed voip and one day
videoconferencing. Please inspect the json output
for other metrics. Due to OS limits it is typically only accurate to a
3ms interval. One thing it does admirably is begin to expose the
sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
technologies, as well as request/grant systems like cable and gpon,
especially when otherwise idle.
Here is a highres plot of starlink's behaviors from last year:
https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848/3238
clearly showing them "optimizing for bandwidth" and changing next sat
hop, and about a 40ms interval of buffering between these switches.
I'd published elsewhere, if anyone cares, a preliminary study of what
starlink's default behaviors did to cubic and BBR...
>
> irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...
The more the merrier! Someday the "right" metrics will arrive.
As a side note, this paper focuses on RAN uplink latency
https://dl.ifip.org/db/conf/itc/itc2021/1570740615.pdf which I think
is a major barrier to most forms of 5G actually achieving good
performance in a FPS game, if it is true for more RANs. I'd like more
to be testing uplink latencies idle and with load, on all
technologies.
>
> thanks again, Dave.
> Al
>
> > -----Original Message-----
> > From: Dave Taht <dave.taht@gmail.com>
> > Sent: Monday, October 31, 2022 12:52 PM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> > Thank you very much for the steer to RFC9097. I'd completely missed that.
> >
> > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > >
> > > (astute readers may have guessed that I pressed "send" too soon on previous
> > message...)
> > >
> > > I also conducted upstream tests this time, here are the results:
> > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > Medium)
> > >
> > > Net Qual UDPST (RFC9097) Ookla
> > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> > Ping(no load)
> > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 8
> > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22 8
> > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22 9
> > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22 7
> > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22 9
> > >
> > > The Upstream capacity measurements reflect an interesting feature that we
> > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > behavior in its 1 second sub-interval measurements and has a bimodal reporting
> > option that divides the complete measurement interval in two time intervals to
> > report an initial (turbo) max capacity and a steady-state max capacity for the
> > later intervals. The UDPST capacity results present both measurements: steady-
> > state first.
> >
> > Certainly we can expect bi-model distributions from many ISPs, as, for
> > one thing, the "speedboost" concept remains popular, except that it's
> > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > it is often configured "sneakily", in that it doesn't kick in for the
> > typical observed duration of the test, for some, they cut the
> > available bandwidth about 20s in, others, 1 or 5 minutes.
> >
> > One of my biggest issues with the rpm spec so far is that it should,
> > at least, sometimes, run randomly longer than the overly short
> > interval it runs for and the tools also allow for manual override of length.
> >
> > we caught a lot of tomfoolery with flent's rrul test running by default for
> > 1m.
> >
> > Also, AQMs on the path can take a while to find the optimal drop or mark rate.
> >
> > >
> > > The capacity processing in networkQuality and Ookla appear to report the
> > steady-state result.
> >
> > Ookla used to basically report the last result. Also it's not a good
> > indicator of web traffic behavior at all, watching the curve
> > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > (40ms)....
> >
> > So adding another mode - how quickly is peak bandwidth actually
> > reached, would be nice.
> >
> > I haven't poked into the current iteration of the goresponsiveness
> > test at all: https://urldefense.com/v3/__https://github.com/network-
> > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > would be good to try collecting more statistics and histograms and
> > methods of analyzing the data in that libre-source version.
> >
> > How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> >
> > >I watched the upstream capacity measurements on the Ookla app, and could
> > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> > the test which determined the final result.
> >
> > I tend to get upset when I see ookla's new test flash a peak result in
> > the seconds and then settle on some lower number somehow.
> > So far as I know they are only sampling the latency every 250ms.
> >
> > >
> > > The working latency is about 200ms in networkQuality and about 280ms as
> > measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> > ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> > in working latency. Also, we used the very dynamic Type C load
> > adjustment/search algorithm in UDPST during all of this testing, which could
> > explain the higher working latency to some degree.
> > >
> > > So, it's worth noting that the measurements needed for assessing working
> > latency/responsiveness are available in the UDPST utility, and that the UDPST
> > measurements are conducted on UDP transport (used by a growing fraction of
> > Internet traffic).
> >
> > Thx, didn't know of this work til now!
> >
> > have you tried irtt?
> >
> > >
> > > comments welcome of course,
> > > Al
> > >
> > > > -----Original Message-----
> > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > To: ippm@ietf.org
> > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > metrics
> > > >
> > > >
> > > > Hi again RPM friends and IPPM'ers,
> > > >
> > > > As promised, I repeated the tests shared last week, this time using both
> > the
> > > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > > followed Sebastian's calculations as well.
> > > >
> > > > Working Latency & Capacity Summary
> > > >
> > > > Net Qual UDPST Ookla
> > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> > > > Ping(no load)
> > > > 885 916 m 66ms 8ms 970 28 0-20 940 8
> > > > 888 1355 h 44ms 8ms 966 28 0-23 940 8
> > > > 891 1109 h 54ms 8ms 968 27 0-19 940 9
> > > > 887 1141 h 53ms 11ms 966 27 0-18 937 7
> > > > 884 1151 h 52ms 9ms 968 28 0-20 937 9
> > > >
> > > > With the sequential test option, I noticed that networkQuality achieved
> > nearly
> > > > the maximum capacity reported almost immediately at the start of a test.
> > > > However, the reported capacities are low by about 60Mbps, especially when
> > > > compared to the Ookla TCP measurements.
> > > >
> > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> > the
> > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > networkQuality
> > > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > > buffering in the downstream direction.
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > To: ippm@ietf.org
> > > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > > metrics
> > > > >
> > > > >
> > > > > Hi RPM friends and IPPM'ers,
> > > > >
> > > > > I was wondering what a comparison of some of the "working latency"
> > metrics
> > > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> > the
> > > > > downlink provisioned for 1Gbps.
> > > > >
> > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > > Speedtest
> > > > > with as similar connectivity as possible (but we know that the traffic
> > will
> > > > > diverge to different servers and we can't change that aspect).
> > > > >
> > > > > Here's a quick summary of yesterday's results:
> > > > >
> > > > > Working Latency & Capacity Summary
> > > > >
> > > > > Net Qual UDPST Ookla
> > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no
> > load)
> > > > > 878 62 970 28 0-19 941 6
> > > > > 891 92 970 27 0-20 940 7
> > > > > 891 120 966 28 0-22 937 9
> > > > > 890 112 970 28 0-21 940 8
> > > > > 903 70 970 28 0-16 935 9
> > > > >
> > > > > Note: all RPM values were categorized as Low.
> > > > >
> > > > > networkQuality downstream capacities are always on the low side compared
> > to
> > > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> > Ookla
> > > > > achieved. I think that a longer test duration might be needed to achieve
> > the
> > > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> > were
> > > > > certainly headed in the right direction. (I recently upgraded to
> > Monterey
> > > > 12.6
> > > > > on my MacBook, so should have the latest version.)
> > > > >
> > > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> > have
> > > > > run the tests with the -v option to help with comparisons. I'll repeat
> > this
> > > > > test when I can make time.
> > > > >
> > > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> > and
> > > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> > access
> > > > and
> > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> > were
> > > > run
> > > > > on an Ubuntu VM in my MacBook.
> > > > >
> > > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > > weekend did not include the new responsiveness metric! I included the
> > ping
> > > > > results anyway, and it was clearly using a server in the nearby area.
> > > > >
> > > > > So, I have some more work to do, but I hope this is interesting-enough
> > to
> > > > > start some comparison discussions, and bring-out some suggestions.
> > > > >
> > > > > happy testing all,
> > > > > Al
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > >
> > > >
> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > T!g-
> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > >
> > > _______________________________________________
> > > ippm mailing list
> > > ippm@ietf.org
> > >
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > KXU$
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> > song-activity-6981366665607352320-
> > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > c48iQFub34zz4iE$
> > Dave Täht CEO, TekLibre, LLC
--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-10-31 23:30 ` Dave Taht
@ 2022-11-01 4:21 ` Dave Taht
2022-11-01 14:51 ` MORTON JR., AL
0 siblings, 1 reply; 29+ messages in thread
From: Dave Taht @ 2022-11-01 4:21 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: ippm, Rpm
Dear Al:
OK, I took your udpst tool for a spin.
NICE! 120k binary (I STILL work on machines with only 4MB of flash),
good feature set, VERY fast, and in very brief testing, seemed
to be accurate in the starlink case, though it's hard to tell with
them as the rate changes every 15s.
I filed a couple bug reports on trivial stuff:
https://github.com/BroadbandForum/obudpst/issues/8
(Adding diffserv and ecn washing or marking detection would be a nice
feature to have)
Aside from the sheer joy coming from the "it compiles! and runs!"
phase I haven't looked much further.
I left a copy running on one of my starlink testbeds -
fremont.starlink.taht.net - if anyone wants to try it. It's
instrumented with netperf, flent, irtt, iperf2 (not quite the latest
version from bob, but close), and now udpst, and good to about a gbit.
nice tool!
Has anyone here played with crusader? ( https://github.com/Zoxc/crusader )
On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>
> > > have you tried irtt? (https://github.com/heistp/irtt )
> > I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481
>
> irtt was meant to simulate high speed voip and one day
> videoconferencing. Please inspect the json output
> for other metrics. Due to OS limits it is typically only accurate to a
> 3ms interval. One thing it does admirably is begin to expose the
> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> technologies, as well as request/grant systems like cable and gpon,
> especially when otherwise idle.
>
> Here is a highres plot of starlink's behaviors from last year:
> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848/3238
>
> clearly showing them "optimizing for bandwidth" and changing next sat
> hop, and about a 40ms interval of buffering between these switches.
> I'd published elsewhere, if anyone cares, a preliminary study of what
> starlink's default behaviors did to cubic and BBR...
>
> >
> > irtt's use of IPDV means that the results won’t compare with UDPST, and possibly networkQuality. But I may give it a try anyway...
>
> The more the merrier! Someday the "right" metrics will arrive.
>
> As a side note, this paper focuses on RAN uplink latency
> https://dl.ifip.org/db/conf/itc/itc2021/1570740615.pdf which I think
> is a major barrier to most forms of 5G actually achieving good
> performance in a FPS game, if it is true for more RANs. I'd like more
> to be testing uplink latencies idle and with load, on all
> technologies.
>
> >
> > thanks again, Dave.
> > Al
> >
> > > -----Original Message-----
> > > From: Dave Taht <dave.taht@gmail.com>
> > > Sent: Monday, October 31, 2022 12:52 PM
> > > To: MORTON JR., AL <acmorton@att.com>
> > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > > metrics
> > >
> > > Thank you very much for the steer to RFC9097. I'd completely missed that.
> > >
> > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > > >
> > > > (astute readers may have guessed that I pressed "send" too soon on previous
> > > message...)
> > > >
> > > > I also conducted upstream tests this time, here are the results:
> > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > Medium)
> > > >
> > > > Net Qual UDPST (RFC9097) Ookla
> > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> > > Ping(no load)
> > > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 8
> > > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22 8
> > > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22 9
> > > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22 7
> > > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22 9
> > > >
> > > > The Upstream capacity measurements reflect an interesting feature that we
> > > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > > behavior in its 1 second sub-interval measurements and has a bimodal reporting
> > > option that divides the complete measurement interval in two time intervals to
> > > report an initial (turbo) max capacity and a steady-state max capacity for the
> > > later intervals. The UDPST capacity results present both measurements: steady-
> > > state first.
> > >
> > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > one thing, the "speedboost" concept remains popular, except that it's
> > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > it is often configured "sneakily", in that it doesn't kick in for the
> > > typical observed duration of the test, for some, they cut the
> > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > >
> > > One of my biggest issues with the rpm spec so far is that it should,
> > > at least, sometimes, run randomly longer than the overly short
> > > interval it runs for and the tools also allow for manual override of length.
> > >
> > > we caught a lot of tomfoolery with flent's rrul test running by default for
> > > 1m.
> > >
> > > Also, AQMs on the path can take a while to find the optimal drop or mark rate.
> > >
> > > >
> > > > The capacity processing in networkQuality and Ookla appear to report the
> > > steady-state result.
> > >
> > > Ookla used to basically report the last result. Also it's not a good
> > > indicator of web traffic behavior at all, watching the curve
> > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > (40ms)....
> > >
> > > So adding another mode - how quickly is peak bandwidth actually
> > > reached, would be nice.
> > >
> > > I haven't poked into the current iteration of the goresponsiveness
> > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > > would be good to try collecting more statistics and histograms and
> > > methods of analyzing the data in that libre-source version.
> > >
> > > How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> > >
> > > >I watched the upstream capacity measurements on the Ookla app, and could
> > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for most of
> > > the test which determined the final result.
> > >
> > > I tend to get upset when I see ookla's new test flash a peak result in
> > > the seconds and then settle on some lower number somehow.
> > > So far as I know they are only sampling the latency every 250ms.
> > >
> > > >
> > > > The working latency is about 200ms in networkQuality and about 280ms as
> > > measured by UDPST (RFC9097). Note that the networkQuality minimum delay is
> > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the difference
> > > in working latency. Also, we used the very dynamic Type C load
> > > adjustment/search algorithm in UDPST during all of this testing, which could
> > > explain the higher working latency to some degree.
> > > >
> > > > So, it's worth noting that the measurements needed for assessing working
> > > latency/responsiveness are available in the UDPST utility, and that the UDPST
> > > measurements are conducted on UDP transport (used by a growing fraction of
> > > Internet traffic).
> > >
> > > Thx, didn't know of this work til now!
> > >
> > > have you tried irtt?
> > >
> > > >
> > > > comments welcome of course,
> > > > Al
> > > >
> > > > > -----Original Message-----
> > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > To: ippm@ietf.org
> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > Latency"
> > > > > metrics
> > > > >
> > > > >
> > > > > Hi again RPM friends and IPPM'ers,
> > > > >
> > > > > As promised, I repeated the tests shared last week, this time using both
> > > the
> > > > > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I
> > > > > followed Sebastian's calculations as well.
> > > > >
> > > > > Working Latency & Capacity Summary
> > > > >
> > > > > Net Qual UDPST Ookla
> > > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> > > > > Ping(no load)
> > > > > 885 916 m 66ms 8ms 970 28 0-20 940 8
> > > > > 888 1355 h 44ms 8ms 966 28 0-23 940 8
> > > > > 891 1109 h 54ms 8ms 968 27 0-19 940 9
> > > > > 887 1141 h 53ms 11ms 966 27 0-18 937 7
> > > > > 884 1151 h 52ms 9ms 968 28 0-20 937 9
> > > > >
> > > > > With the sequential test option, I noticed that networkQuality achieved
> > > nearly
> > > > > the maximum capacity reported almost immediately at the start of a test.
> > > > > However, the reported capacities are low by about 60Mbps, especially when
> > > > > compared to the Ookla TCP measurements.
> > > > >
> > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of
> > > the
> > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > networkQuality
> > > > > RPM measurements were categorized as "High". There doesn't seem to be much
> > > > > buffering in the downstream direction.
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > To: ippm@ietf.org
> > > > > > Subject: [ippm] Preliminary measurement comparison of "Working Latency"
> > > > > > metrics
> > > > > >
> > > > > >
> > > > > > Hi RPM friends and IPPM'ers,
> > > > > >
> > > > > > I was wondering what a comparison of some of the "working latency"
> > > metrics
> > > > > > would look like, so I ran some tests using a service on DOCSIS 3.1, with
> > > the
> > > > > > downlink provisioned for 1Gbps.
> > > > > >
> > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla
> > > > > Speedtest
> > > > > > with as similar connectivity as possible (but we know that the traffic
> > > will
> > > > > > diverge to different servers and we can't change that aspect).
> > > > > >
> > > > > > Here's a quick summary of yesterday's results:
> > > > > >
> > > > > > Working Latency & Capacity Summary
> > > > > >
> > > > > > Net Qual UDPST Ookla
> > > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no
> > > load)
> > > > > > 878 62 970 28 0-19 941 6
> > > > > > 891 92 970 27 0-20 940 7
> > > > > > 891 120 966 28 0-22 937 9
> > > > > > 890 112 970 28 0-21 940 8
> > > > > > 903 70 970 28 0-16 935 9
> > > > > >
> > > > > > Note: all RPM values were categorized as Low.
> > > > > >
> > > > > > networkQuality downstream capacities are always on the low side compared
> > > to
> > > > > > others. We would expect about 940Mbps for TCP, and that's mostly what
> > > Ookla
> > > > > > achieved. I think that a longer test duration might be needed to achieve
> > > the
> > > > > > actual 1Gbps capacity with networkQuality; intermediate values observed
> > > were
> > > > > > certainly headed in the right direction. (I recently upgraded to
> > > Monterey
> > > > > 12.6
> > > > > > on my MacBook, so should have the latest version.)
> > > > > >
> > > > > > Also, as Sebastian Moeller's message to the list reminded me, I should
> > > have
> > > > > > run the tests with the -v option to help with comparisons. I'll repeat
> > > this
> > > > > > test when I can make time.
> > > > > >
> > > > > > The UDPST measurements of RTTmin (minimum RTT observed during the test)
> > > and
> > > > > > the range of variation above the minimum (RTTVarRnge) add-up to very
> > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this
> > > access
> > > > > and
> > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago
> > > > > > conducting tests, so the minimum 28ms is typical. UDPST measurements
> > > were
> > > > > run
> > > > > > on an Ubuntu VM in my MacBook.
> > > > > >
> > > > > > The big disappointment was that the Ookla desktop app I updated over the
> > > > > > weekend did not include the new responsiveness metric! I included the
> > > ping
> > > > > > results anyway, and it was clearly using a server in the nearby area.
> > > > > >
> > > > > > So, I have some more work to do, but I hope this is interesting-enough
> > > to
> > > > > > start some comparison discussions, and bring-out some suggestions.
> > > > > >
> > > > > > happy testing all,
> > > > > > Al
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > >
> > > > >
> > > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > T!g-
> > > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > >
> > > > _______________________________________________
> > > > ippm mailing list
> > > > ippm@ietf.org
> > > >
> > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > KXU$
> > >
> > >
> > >
> > > --
> > > This song goes out to all the folk that thought Stadia would work:
> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> > > song-activity-6981366665607352320-
> > > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > c48iQFub34zz4iE$
> > > Dave Täht CEO, TekLibre, LLC
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-01 4:21 ` Dave Taht
@ 2022-11-01 14:51 ` MORTON JR., AL
2022-11-04 17:14 ` MORTON JR., AL
0 siblings, 1 reply; 29+ messages in thread
From: MORTON JR., AL @ 2022-11-01 14:51 UTC (permalink / raw)
To: Dave Taht; +Cc: ippm, Rpm
Hi Dave,
Thanks for trying UDPST (RFC 9097)!
Something you might try with starlink:
use the -X option and UDPST will generate random payloads.
The measurements with -X will reflect the uncompressed that are possible.
I tried this on a ship-board Internet access: uncompressed rate was 100kbps.
A few more quick replies below,
Al
> -----Original Message-----
> From: Dave Taht <dave.taht@gmail.com>
> Sent: Tuesday, November 1, 2022 12:22 AM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
>
> Dear Al:
>
> OK, I took your udpst tool for a spin.
>
> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> good feature set, VERY fast,
[acm]
Len Ciavattone (my partner in crime on several measurement projects) is the lead coder: he has implemented many measurement tools extremely efficiently, this one in C-lang.
> and in very brief testing, seemed
> to be accurate in the starlink case, though it's hard to tell with
> them as the rate changes every 15s.
[acm]
Great! Our guiding principle developing UDPST has been to test the accuracy of measurements against a ground-truth. It pays-off.
>
> I filed a couple bug reports on trivial stuff:
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> SswH_opYmEw$
[acm]
much appreciated... We have an OB-UDPST project meeting this Friday, can discuss then.
>
> (Adding diffserv and ecn washing or marking detection would be a nice
> feature to have)
>
> Aside from the sheer joy coming from the "it compiles! and runs!"
> phase I haven't looked much further.
>
> I left a copy running on one of my starlink testbeds -
> fremont.starlink.taht.net - if anyone wants to try it. It's
> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> version from bob, but close), and now udpst, and good to about a gbit.
>
> nice tool!
[acm]
Thanks again!
>
> Has anyone here played with crusader? (
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$ )
>
> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> >
> > > > have you tried irtt?
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$ )
> > > I have not. Seems like a reasonable tool for UDP testing. The feature I
> didn't like in my scan of the documentation is the use of Inter-packet delay
> variation (IPDV) instead of packet delay variation (PDV): variation from the
> minimum (or reference) delay. The morbidly curious can find my analysis in RFC
> 5481:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> T7QSlc$
> >
> > irtt was meant to simulate high speed voip and one day
> > videoconferencing. Please inspect the json output
> > for other metrics. Due to OS limits it is typically only accurate to a
> > 3ms interval. One thing it does admirably is begin to expose the
> > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > technologies, as well as request/grant systems like cable and gpon,
> > especially when otherwise idle.
> >
> > Here is a highres plot of starlink's behaviors from last year:
> > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> bandwidth-
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> >
> > clearly showing them "optimizing for bandwidth" and changing next sat
> > hop, and about a 40ms interval of buffering between these switches.
> > I'd published elsewhere, if anyone cares, a preliminary study of what
> > starlink's default behaviors did to cubic and BBR...
> >
> > >
> > > irtt's use of IPDV means that the results won’t compare with UDPST, and
> possibly networkQuality. But I may give it a try anyway...
> >
> > The more the merrier! Someday the "right" metrics will arrive.
> >
> > As a side note, this paper focuses on RAN uplink latency
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHgvqerjg$ which I think
> > is a major barrier to most forms of 5G actually achieving good
> > performance in a FPS game, if it is true for more RANs. I'd like more
> > to be testing uplink latencies idle and with load, on all
> > technologies.
> >
> > >
> > > thanks again, Dave.
> > > Al
> > >
> > > > -----Original Message-----
> > > > From: Dave Taht <dave.taht@gmail.com>
> > > > Sent: Monday, October 31, 2022 12:52 PM
> > > > To: MORTON JR., AL <acmorton@att.com>
> > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > > metrics
> > > >
> > > > Thank you very much for the steer to RFC9097. I'd completely missed
> that.
> > > >
> > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com> wrote:
> > > > >
> > > > > (astute readers may have guessed that I pressed "send" too soon on
> previous
> > > > message...)
> > > > >
> > > > > I also conducted upstream tests this time, here are the results:
> > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > > Medium)
> > > > >
> > > > > Net Qual UDPST (RFC9097) Ookla
> > > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> > > > Ping(no load)
> > > > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
> 8
> > > > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22
> 8
> > > > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22
> 9
> > > > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22
> 7
> > > > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22
> 9
> > > > >
> > > > > The Upstream capacity measurements reflect an interesting feature that
> we
> > > > can reliably and repeatably measure with UDPST. The first ~3 seconds of
> > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this
> > > > behavior in its 1 second sub-interval measurements and has a bimodal
> reporting
> > > > option that divides the complete measurement interval in two time
> intervals to
> > > > report an initial (turbo) max capacity and a steady-state max capacity
> for the
> > > > later intervals. The UDPST capacity results present both measurements:
> steady-
> > > > state first.
> > > >
> > > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > > one thing, the "speedboost" concept remains popular, except that it's
> > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > > it is often configured "sneakily", in that it doesn't kick in for the
> > > > typical observed duration of the test, for some, they cut the
> > > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > > >
> > > > One of my biggest issues with the rpm spec so far is that it should,
> > > > at least, sometimes, run randomly longer than the overly short
> > > > interval it runs for and the tools also allow for manual override of
> length.
> > > >
> > > > we caught a lot of tomfoolery with flent's rrul test running by default
> for
> > > > 1m.
> > > >
> > > > Also, AQMs on the path can take a while to find the optimal drop or mark
> rate.
> > > >
> > > > >
> > > > > The capacity processing in networkQuality and Ookla appear to report
> the
> > > > steady-state result.
> > > >
> > > > Ookla used to basically report the last result. Also it's not a good
> > > > indicator of web traffic behavior at all, watching the curve
> > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > > (40ms)....
> > > >
> > > > So adding another mode - how quickly is peak bandwidth actually
> > > > reached, would be nice.
> > > >
> > > > I haven't poked into the current iteration of the goresponsiveness
> > > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > > > would be good to try collecting more statistics and histograms and
> > > > methods of analyzing the data in that libre-source version.
> > > >
> > > > How does networkQuality compare vs a vs your tool vs a vs
> goresponsiveness?
> > > >
> > > > >I watched the upstream capacity measurements on the Ookla app, and
> could
> > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> most of
> > > > the test which determined the final result.
> > > >
> > > > I tend to get upset when I see ookla's new test flash a peak result in
> > > > the seconds and then settle on some lower number somehow.
> > > > So far as I know they are only sampling the latency every 250ms.
> > > >
> > > > >
> > > > > The working latency is about 200ms in networkQuality and about 280ms
> as
> > > > measured by UDPST (RFC9097). Note that the networkQuality minimum delay
> is
> > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> difference
> > > > in working latency. Also, we used the very dynamic Type C load
> > > > adjustment/search algorithm in UDPST during all of this testing, which
> could
> > > > explain the higher working latency to some degree.
> > > > >
> > > > > So, it's worth noting that the measurements needed for assessing
> working
> > > > latency/responsiveness are available in the UDPST utility, and that the
> UDPST
> > > > measurements are conducted on UDP transport (used by a growing fraction
> of
> > > > Internet traffic).
> > > >
> > > > Thx, didn't know of this work til now!
> > > >
> > > > have you tried irtt?
> > > >
> > > > >
> > > > > comments welcome of course,
> > > > > Al
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > > To: ippm@ietf.org
> > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > > Latency"
> > > > > > metrics
> > > > > >
> > > > > >
> > > > > > Hi again RPM friends and IPPM'ers,
> > > > > >
> > > > > > As promised, I repeated the tests shared last week, this time using
> both
> > > > the
> > > > > > verbose (-v) and sequential (-s) dwn/up test options of
> networkQuality. I
> > > > > > followed Sebastian's calculations as well.
> > > > > >
> > > > > > Working Latency & Capacity Summary
> > > > > >
> > > > > > Net Qual UDPST
> Ookla
> > > > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
> DnCap
> > > > > > Ping(no load)
> > > > > > 885 916 m 66ms 8ms 970 28 0-20 940
> 8
> > > > > > 888 1355 h 44ms 8ms 966 28 0-23 940
> 8
> > > > > > 891 1109 h 54ms 8ms 968 27 0-19 940
> 9
> > > > > > 887 1141 h 53ms 11ms 966 27 0-18 937
> 7
> > > > > > 884 1151 h 52ms 9ms 968 28 0-20 937
> 9
> > > > > >
> > > > > > With the sequential test option, I noticed that networkQuality
> achieved
> > > > nearly
> > > > > > the maximum capacity reported almost immediately at the start of a
> test.
> > > > > > However, the reported capacities are low by about 60Mbps, especially
> when
> > > > > > compared to the Ookla TCP measurements.
> > > > > >
> > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high
> end of
> > > > the
> > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > > networkQuality
> > > > > > RPM measurements were categorized as "High". There doesn't seem to
> be much
> > > > > > buffering in the downstream direction.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > > To: ippm@ietf.org
> > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > > > > > > metrics
> > > > > > >
> > > > > > >
> > > > > > > Hi RPM friends and IPPM'ers,
> > > > > > >
> > > > > > > I was wondering what a comparison of some of the "working latency"
> > > > metrics
> > > > > > > would look like, so I ran some tests using a service on DOCSIS
> 3.1, with
> > > > the
> > > > > > > downlink provisioned for 1Gbps.
> > > > > > >
> > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
> Ookla
> > > > > > Speedtest
> > > > > > > with as similar connectivity as possible (but we know that the
> traffic
> > > > will
> > > > > > > diverge to different servers and we can't change that aspect).
> > > > > > >
> > > > > > > Here's a quick summary of yesterday's results:
> > > > > > >
> > > > > > > Working Latency & Capacity Summary
> > > > > > >
> > > > > > > Net Qual UDPST Ookla
> > > > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap
> Ping(no
> > > > load)
> > > > > > > 878 62 970 28 0-19 941 6
> > > > > > > 891 92 970 27 0-20 940 7
> > > > > > > 891 120 966 28 0-22 937 9
> > > > > > > 890 112 970 28 0-21 940 8
> > > > > > > 903 70 970 28 0-16 935 9
> > > > > > >
> > > > > > > Note: all RPM values were categorized as Low.
> > > > > > >
> > > > > > > networkQuality downstream capacities are always on the low side
> compared
> > > > to
> > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
> what
> > > > Ookla
> > > > > > > achieved. I think that a longer test duration might be needed to
> achieve
> > > > the
> > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
> observed
> > > > were
> > > > > > > certainly headed in the right direction. (I recently upgraded to
> > > > Monterey
> > > > > > 12.6
> > > > > > > on my MacBook, so should have the latest version.)
> > > > > > >
> > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
> should
> > > > have
> > > > > > > run the tests with the -v option to help with comparisons. I'll
> repeat
> > > > this
> > > > > > > test when I can make time.
> > > > > > >
> > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during the
> test)
> > > > and
> > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
> very
> > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
> this
> > > > access
> > > > > > and
> > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
> Chicago
> > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
> measurements
> > > > were
> > > > > > run
> > > > > > > on an Ubuntu VM in my MacBook.
> > > > > > >
> > > > > > > The big disappointment was that the Ookla desktop app I updated
> over the
> > > > > > > weekend did not include the new responsiveness metric! I included
> the
> > > > ping
> > > > > > > results anyway, and it was clearly using a server in the nearby
> area.
> > > > > > >
> > > > > > > So, I have some more work to do, but I hope this is interesting-
> enough
> > > > to
> > > > > > > start some comparison discussions, and bring-out some suggestions.
> > > > > > >
> > > > > > > happy testing all,
> > > > > > > Al
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ippm mailing list
> > > > > > > ippm@ietf.org
> > > > > > >
> > > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > >
> > > > > >
> > > >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > T!g-
> > > >
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > > >
> > > > > _______________________________________________
> > > > > ippm mailing list
> > > > > ippm@ietf.org
> > > > >
> > > >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > > KXU$
> > > >
> > > >
> > > >
> > > > --
> > > > This song goes out to all the folk that thought Stadia would work:
> > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-
> > > > song-activity-6981366665607352320-
> > > >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > > c48iQFub34zz4iE$
> > > > Dave Täht CEO, TekLibre, LLC
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-song-activity-6981366665607352320-
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHLHDpSWs$
> > Dave Täht CEO, TekLibre, LLC
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mushroom-
> song-activity-6981366665607352320-
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> IEUcSswHLHDpSWs$
> Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Rpm] lightweight active sensing of bandwidth and buffering
2022-10-31 22:08 ` MORTON JR., AL
2022-10-31 22:44 ` rjmcmahon
@ 2022-11-01 20:15 ` Dave Taht
2022-11-01 23:39 ` rjmcmahon
1 sibling, 1 reply; 29+ messages in thread
From: Dave Taht @ 2022-11-01 20:15 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: rjmcmahon, Rpm, ippm
For about 2 years now the cake w-adaptive bandwidth project has been
exploring techniques to lightweightedly sense bandwidth and buffering
problems. One of my favorites was their discovery that ICMP type 13
got them working OWD from millions of ipv4 devices!
They've also explored leveraging ntp and multiple other methods, and
have scripts available that do a good job of compensating for 5g and
starlink's misbehaviors.
They've also pioneered a whole bunch of new graphing techniques, which
I do wish were used more than single number summaries especially in
analyzing the behaviors of new metrics like rpm, samknows, ookla, and
RFC9097 - to see what is being missed.
There are thousands of posts about this research topic, a new post on
OWD just went by here.
https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
and of course, I love flent's enormous graphing toolset for simulating
and analyzing complex network behaviors.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] lightweight active sensing of bandwidth and buffering
2022-11-01 20:15 ` [Rpm] lightweight active sensing of bandwidth and buffering Dave Taht
@ 2022-11-01 23:39 ` rjmcmahon
2022-11-02 8:23 ` Sebastian Moeller
0 siblings, 1 reply; 29+ messages in thread
From: rjmcmahon @ 2022-11-01 23:39 UTC (permalink / raw)
To: Dave Taht; +Cc: MORTON JR., AL, Rpm, ippm
Bufferbloat shifts the minimum of the latency or OWD CDF. A suggestion
is to disable x-axis auto-scaling and start from zero.
Bob
> For about 2 years now the cake w-adaptive bandwidth project has been
> exploring techniques to lightweightedly sense bandwidth and buffering
> problems. One of my favorites was their discovery that ICMP type 13
> got them working OWD from millions of ipv4 devices!
>
> They've also explored leveraging ntp and multiple other methods, and
> have scripts available that do a good job of compensating for 5g and
> starlink's misbehaviors.
>
> They've also pioneered a whole bunch of new graphing techniques, which
> I do wish were used more than single number summaries especially in
> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
> RFC9097 - to see what is being missed.
>
> There are thousands of posts about this research topic, a new post on
> OWD just went by here.
>
> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>
> and of course, I love flent's enormous graphing toolset for simulating
> and analyzing complex network behaviors.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] lightweight active sensing of bandwidth and buffering
2022-11-01 23:39 ` rjmcmahon
@ 2022-11-02 8:23 ` Sebastian Moeller
2022-11-02 9:41 ` [Rpm] [ippm] " Ruediger.Geib
2022-11-02 19:21 ` [Rpm] " rjmcmahon
0 siblings, 2 replies; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-02 8:23 UTC (permalink / raw)
To: rjmcmahon; +Cc: Dave Täht, Rpm, MORTON JR., AL, ippm
Hi Bob,
thanks!
> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>
> Bufferbloat shifts the minimum of the latency or OWD CDF.
[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
> A suggestion is to disable x-axis auto-scaling and start from zero.
[SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this
*) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
**) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>
> Bob
>> For about 2 years now the cake w-adaptive bandwidth project has been
>> exploring techniques to lightweightedly sense bandwidth and buffering
>> problems. One of my favorites was their discovery that ICMP type 13
>> got them working OWD from millions of ipv4 devices!
>> They've also explored leveraging ntp and multiple other methods, and
>> have scripts available that do a good job of compensating for 5g and
>> starlink's misbehaviors.
>> They've also pioneered a whole bunch of new graphing techniques, which
>> I do wish were used more than single number summaries especially in
>> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
>> RFC9097 - to see what is being missed.
>> There are thousands of posts about this research topic, a new post on
>> OWD just went by here.
>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> and of course, I love flent's enormous graphing toolset for simulating
>> and analyzing complex network behaviors.
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 8:23 ` Sebastian Moeller
@ 2022-11-02 9:41 ` Ruediger.Geib
2022-11-02 19:29 ` rjmcmahon
2022-11-02 21:41 ` Sebastian Moeller
2022-11-02 19:21 ` [Rpm] " rjmcmahon
1 sibling, 2 replies; 29+ messages in thread
From: Ruediger.Geib @ 2022-11-02 9:41 UTC (permalink / raw)
To: moeller0, rjmcmahon; +Cc: rpm, ippm
Bob, Sebastian,
not being active on your topic, just to add what I observed on congestion:
- starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
- buffer fill reaches a "steady state", called bufferbloat on access I think; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct). I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
- a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill. The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
- in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
Regards,
Ruediger
> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>
> Bufferbloat shifts the minimum of the latency or OWD CDF.
[SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
> A suggestion is to disable x-axis auto-scaling and start from zero.
[SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this
*) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
**) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>
> Bob
>> For about 2 years now the cake w-adaptive bandwidth project has been
>> exploring techniques to lightweightedly sense bandwidth and
>> buffering problems. One of my favorites was their discovery that ICMP
>> type 13 got them working OWD from millions of ipv4 devices!
>> They've also explored leveraging ntp and multiple other methods, and
>> have scripts available that do a good job of compensating for 5g and
>> starlink's misbehaviors.
>> They've also pioneered a whole bunch of new graphing techniques,
>> which I do wish were used more than single number summaries
>> especially in analyzing the behaviors of new metrics like rpm,
>> samknows, ookla, and
>> RFC9097 - to see what is being missed.
>> There are thousands of posts about this research topic, a new post on
>> OWD just went by here.
>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> and of course, I love flent's enormous graphing toolset for
>> simulating and analyzing complex network behaviors.
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] lightweight active sensing of bandwidth and buffering
2022-11-02 8:23 ` Sebastian Moeller
2022-11-02 9:41 ` [Rpm] [ippm] " Ruediger.Geib
@ 2022-11-02 19:21 ` rjmcmahon
1 sibling, 0 replies; 29+ messages in thread
From: rjmcmahon @ 2022-11-02 19:21 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Dave Täht, Rpm, MORTON JR., AL, ippm
Bloat is really a unit of memory so the inP metric in iperf 2 may be
preferred. It's calculated using Little's law. The bloated UDP run over
a WiFi link is ~2470 packets and min e2e latency of ~31 ms. The non
bloat steady state is ~330 packets and min latency of ~4ms. The latter
is likely showing that the WiFi transmits being delayed by ~4ms NAV
though I'd have to dig more to verify. (Note: I'll probably add inP to
burst isoch traffic too. Not there yet as there hasn't been wide
adoption of --isochronous though some use it extensively)
[root@fedora ~]# iperf -s -u -i 1 -e -B 192.168.1.231%eth1
------------------------------------------------------------
Server listening on UDP port 5001 with pid 611171
Binding to local address 192.168.1.231 and iface eth1
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.231%eth1 port 5001 connected with 192.168.1.15
port 50870 (trip-times) (sock=3) (peer 2.1.9-master) on 2022-11-02
12:00:36 (PDT)
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Latency avg/min/max/stdev PPS Rx/inP NetPwr
[ 1] 0.00-1.00 sec 118 MBytes 986 Mbits/sec 0.016 ms 4545/88410
(5.1%) 29.484/5.175/42.872/6.106 ms 83979 pps 83865/2476(513) pkts 4181
[ 1] 0.00-1.00 sec 47 datagrams received out-of-order
[ 1] 1.00-2.00 sec 123 MBytes 1.03 Gbits/sec 0.017 ms 3937/91326
(4.3%) 31.576/29.353/39.704/1.169 ms 87396 pps 87389/2760(102) pkts 4068
[ 1] 2.00-3.00 sec 123 MBytes 1.03 Gbits/sec 0.008 ms 3311/91243
(3.6%) 31.602/29.218/39.096/1.259 ms 87899 pps 87932/2778(111) pkts 4090
[ 1] 3.00-4.00 sec 123 MBytes 1.03 Gbits/sec 0.010 ms 3474/91385
(3.8%) 31.570/29.032/42.499/1.856 ms 87943 pps 87911/2776(163) pkts 4093
[ 1] 4.00-5.00 sec 122 MBytes 1.02 Gbits/sec 0.015 ms 4482/91331
(4.9%) 31.349/29.163/40.234/1.291 ms 86813 pps 86849/2721(112) pkts 4073
[ 1] 5.00-6.00 sec 122 MBytes 1.02 Gbits/sec 0.023 ms 4249/91102
(4.7%) 31.376/29.336/41.963/1.318 ms 86890 pps 86853/2726(115) pkts 4069
[ 1] 6.00-7.00 sec 122 MBytes 1.03 Gbits/sec 0.007 ms 4052/91410
(4.4%) 31.424/29.178/38.683/1.037 ms 87344 pps 87358/2745(91) pkts 4087
[ 1] 7.00-8.00 sec 123 MBytes 1.03 Gbits/sec 0.013 ms 3464/91297
(3.8%) 31.414/29.189/38.161/1.061 ms 87839 pps 87833/2759(93) pkts 4110
[ 1] 8.00-9.00 sec 123 MBytes 1.03 Gbits/sec 0.010 ms 3253/90930
(3.6%) 31.686/29.335/39.989/1.478 ms 87662 pps 87677/2778(130) pkts 4068
[ 1] 9.00-10.00 sec 122 MBytes 1.03 Gbits/sec 0.010 ms 3971/91331
(4.3%) 31.486/29.197/39.750/1.407 ms 87378 pps 87360/2751(123) pkts 4079
[ 1] 0.00-10.03 sec 1.20 GBytes 1.02 Gbits/sec 0.021 ms
39124/913050 (4.3%) 31.314/5.175/42.872/2.365 ms 87095 pps
873926/2727(206) pkts 4089
[ 1] 0.00-10.03 sec 47 datagrams received out-of-order
[ 3] WARNING: ack of last datagram failed.
[ 2] local 192.168.1.231%eth1 port 5001 connected with 192.168.1.15
port 46481 (trip-times) (sock=5) (peer 2.1.9-master) on 2022-11-02
12:01:24 (PDT)
[ 2] 0.00-1.00 sec 106 MBytes 886 Mbits/sec 0.012 ms 799/76100
(1%) 4.377/2.780/9.694/0.547 ms 75391 pps 75301/330(41) pkts 25287
[ 2] 0.00-1.00 sec 46 datagrams received out-of-order
[ 2] 1.00-2.00 sec 106 MBytes 892 Mbits/sec 0.018 ms 746/76624
(0.97%) 4.453/3.148/9.444/0.693 ms 75884 pps 75878/338(53) pkts 25048
[ 2] 2.00-3.00 sec 106 MBytes 892 Mbits/sec 0.010 ms 640/76528
(0.84%) 4.388/3.022/9.848/0.529 ms 75888 pps 75888/333(40) pkts 25425
[ 2] 3.00-4.00 sec 106 MBytes 891 Mbits/sec 0.014 ms 707/76436
(0.92%) 4.412/3.065/7.490/0.514 ms 75727 pps 75729/334(39) pkts 25231
[ 2] 4.00-5.00 sec 106 MBytes 892 Mbits/sec 0.012 ms 764/76623
(1%) 4.408/3.081/8.836/0.571 ms 75859 pps 75859/334(43) pkts 25300
[ 2] 5.00-6.00 sec 106 MBytes 888 Mbits/sec 0.016 ms 941/76483
(1.2%) 4.330/3.097/7.841/0.490 ms 75548 pps 75542/327(37) pkts 25648
[ 2] 6.00-7.00 sec 106 MBytes 893 Mbits/sec 0.015 ms 549/76500
(0.72%) 4.405/3.021/9.493/0.554 ms 75945 pps 75951/335(42) pkts 25346
[ 2] 7.00-8.00 sec 106 MBytes 892 Mbits/sec 0.021 ms 683/76572
(0.89%) 4.362/3.042/8.770/0.488 ms 75892 pps 75889/331(37) pkts 25576
[ 2] 8.00-9.00 sec 106 MBytes 892 Mbits/sec 0.019 ms 688/76520
(0.9%) 4.355/2.964/10.486/0.553 ms 75906 pps 75832/331(42) pkts 25599
[ 2] 9.00-10.00 sec 106 MBytes 892 Mbits/sec 0.015 ms 644/76496
(0.84%) 4.387/3.043/10.469/0.563 ms 75775 pps 75852/332(43) pkts 25419
[ 2] 0.00-10.01 sec 1.04 GBytes 891 Mbits/sec 0.024 ms 7161/765314
(0.94%) 4.388/2.780/10.486/0.554 ms 75775 pps 758153/333(42) pkts 25385
[ 2] 0.00-10.01 sec 46 datagrams received out-of-order
[root@ctrl1fc35 ~]# iperf -c 192.168.1.231%enp2s0 -u --trip-times -b 1G
------------------------------------------------------------
Client connecting to 192.168.1.231, UDP port 5001 with pid 724001 via
enp2s0 (1 flows)
Sending 1470 byte datagrams, IPG target: 10.95 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.15 port 50870 connected with 192.168.1.231 port
5001 (trip-times)
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.00 sec 1.25 GBytes 1.07 Gbits/sec
[ 1] Sent 913051 datagrams
[ 1] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Latency avg/min/max/stdev PPS Rx/inP NetPwr
[ 1] 0.00-10.03 sec 1.20 GBytes 1.02 Gbits/sec 0.020 ms
39124/913050 (4.3%) 31.314/5.175/42.872/0.021 ms 90994 pps 90994/2849(2)
pkts 4089
[ 1] 0.00-10.03 sec 47 datagrams received out-of-order
[root@ctrl1fc35 ~]# iperf -c 192.168.1.231%enp2s0 -u --trip-times -b
900m
------------------------------------------------------------
Client connecting to 192.168.1.231, UDP port 5001 with pid 724015 via
enp2s0 (1 flows)
Sending 1470 byte datagrams, IPG target: 13.07 us (kalman adjust)
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.15 port 46481 connected with 192.168.1.231 port
5001 (trip-times)
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.00 sec 1.05 GBytes 900 Mbits/sec
[ 1] Sent 765315 datagrams
[ 1] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total
Latency avg/min/max/stdev PPS Rx/inP NetPwr
[ 1] 0.00-10.01 sec 1.04 GBytes 891 Mbits/sec 0.024 ms 7161/765314
(0.94%) 4.388/2.780/10.486/0.032 ms 76490 pps 76490/336(2) pkts 25385
[ 1] 0.00-10.01 sec 46 datagrams received out-of-order
Bob
> Hi Bob,
>
> thanks!
>
>
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>> <rpm@lists.bufferbloat.net> wrote:
>>
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>
> [SM] Thank you for spelling this out explicitly, I only worked on a
> vage implicit assumption along those lines. However what I want to
> avoid is using delay magnitude itself as classifier between high and
> low load condition as that seems statistically uncouth to then show
> that the delay differs between the two classes;).
> Yet, your comment convinced me that my current load threshold (at
> least for the high load condition) probably is too small, exactly
> because the "base" of the high-load CDFs coincides with the base of
> the low-load CDFs implying that the high-load class contains too many
> samples with decent delay (which after all is one of the goals of the
> whole autorate endeavor).
>
>
>> A suggestion is to disable x-axis auto-scaling and start from zero.
>
> [SM] Will reconsider. I started with start at zero, end then switched
> to an x-range that starts with the delay corresponding to 0.01% for
> the reflector/condition with the lowest such value and stops at 97.5%
> for the reflector/condition with the highest delay value. My rationale
> is that the base delay/path delay of each reflector is not all that
> informative* (and it can still be learned from reading the x-axis),
> the long tail > 50% however is where I expect most differences so I
> want to emphasize this and finally I wanted to avoid that the actual
> "curvy" part gets compressed so much that all lines more or less
> coincide. As I said, I will reconsider this
>
>
> *) We also maintain individual baselines per reflector, so I could
> just plot the differences from baseline, but that would essentially
> equalize all reflectors, and I think having a plot that easily shows
> reflectors with outlying base delay can be informative when selecting
> reflector candidates. However once we actually switch to OWDs baseline
> correction might be required anyways, as due to colck differences ICMP
> type 13/14 data can have massive offsets that are mostly indicative of
> un synched clocks**.
>
> **) This is whyI would prefer to use NTP servers as reflectors with
> NTP requests, my expectation is all of these should be reasonably
> synced by default so that offsets should be in the sane range....
>
>
>>
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense bandwidth and
>>> buffering
>>> problems. One of my favorites was their discovery that ICMP type 13
>>> got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques,
>>> which
>>> I do wish were used more than single number summaries especially in
>>> analyzing the behaviors of new metrics like rpm, samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for
>>> simulating
>>> and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 9:41 ` [Rpm] [ippm] " Ruediger.Geib
@ 2022-11-02 19:29 ` rjmcmahon
2022-11-02 19:44 ` Dave Taht
2022-11-02 21:41 ` Sebastian Moeller
1 sibling, 1 reply; 29+ messages in thread
From: rjmcmahon @ 2022-11-02 19:29 UTC (permalink / raw)
To: Ruediger.Geib; +Cc: moeller0, rpm, ippm
Most measuring bloat are ignoring queue build up phase and rather start
taking measurements after the bottleneck queue is in a standing state.
My opinion, the best units for bloat is packets for UDP or bytes for
TCP. Min delay is a proxy measurement.
Little's law allows one to compute this though does assume the network
is in a stable state over the measurement interval. In the real world,
this probably is rarely true. So we, in test & measurement engineering,
force the standing state with some sort of measurement co-traffic and
call it "working conditions" or equivalent. ;)
Bob
> Bob, Sebastian,
>
> not being active on your topic, just to add what I observed on
> congestion:
> - starts with an increase of jitter, but measured minimum delays still
> remain constant. Technically, a queue builds up some of the time, but
> it isn't present permanently.
> - buffer fill reaches a "steady state", called bufferbloat on access I
> think; technically, OWD increases also for the minimum delays, jitter
> now decreases (what you've described that as "the delay magnitude"
> decreases or "minimum CDF shift" respectively, if I'm correct). I'd
> expect packet loss to occur, once the buffer fill is on steady state,
> but loss might be randomly distributed and could be of a low
> percentage.
> - a sudden rather long load burst may cause a jump-start to
> "steady-state" buffer fill. The above holds for a slow but steady load
> increase (where the measurement frequency determines the timescale
> qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't
> an easy to handle single metric to identify congestion.
>
> Regards,
>
> Ruediger
>
>
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>> <rpm@lists.bufferbloat.net> wrote:
>>
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>
> [SM] Thank you for spelling this out explicitly, I only worked on a
> vage implicit assumption along those lines. However what I want to
> avoid is using delay magnitude itself as classifier between high and
> low load condition as that seems statistically uncouth to then show
> that the delay differs between the two classes;).
> Yet, your comment convinced me that my current load threshold (at
> least for the high load condition) probably is too small, exactly
> because the "base" of the high-load CDFs coincides with the base of
> the low-load CDFs implying that the high-load class contains too many
> samples with decent delay (which after all is one of the goals of the
> whole autorate endeavor).
>
>
>> A suggestion is to disable x-axis auto-scaling and start from zero.
>
> [SM] Will reconsider. I started with start at zero, end then switched
> to an x-range that starts with the delay corresponding to 0.01% for
> the reflector/condition with the lowest such value and stops at 97.5%
> for the reflector/condition with the highest delay value. My rationale
> is that the base delay/path delay of each reflector is not all that
> informative* (and it can still be learned from reading the x-axis),
> the long tail > 50% however is where I expect most differences so I
> want to emphasize this and finally I wanted to avoid that the actual
> "curvy" part gets compressed so much that all lines more or less
> coincide. As I said, I will reconsider this
>
>
> *) We also maintain individual baselines per reflector, so I could
> just plot the differences from baseline, but that would essentially
> equalize all reflectors, and I think having a plot that easily shows
> reflectors with outlying base delay can be informative when selecting
> reflector candidates. However once we actually switch to OWDs baseline
> correction might be required anyways, as due to colck differences ICMP
> type 13/14 data can have massive offsets that are mostly indicative of
> un synched clocks**.
>
> **) This is whyI would prefer to use NTP servers as reflectors with
> NTP requests, my expectation is all of these should be reasonably
> synced by default so that offsets should be in the sane range....
>
>
>>
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense bandwidth and
>>> buffering problems. One of my favorites was their discovery that ICMP
>>> type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques,
>>> which I do wish were used more than single number summaries
>>> especially in analyzing the behaviors of new metrics like rpm,
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 19:29 ` rjmcmahon
@ 2022-11-02 19:44 ` Dave Taht
2022-11-02 20:37 ` rjmcmahon
2022-11-02 21:13 ` Sebastian Moeller
0 siblings, 2 replies; 29+ messages in thread
From: Dave Taht @ 2022-11-02 19:44 UTC (permalink / raw)
To: rjmcmahon; +Cc: Ruediger.Geib, rpm, ippm
On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
<rpm@lists.bufferbloat.net> wrote:
>
> Most measuring bloat are ignoring queue build up phase and rather start
> taking measurements after the bottleneck queue is in a standing state.
+10. It's the slow start transient that is holding things back. If we
could, for example
open up the 110+ objects and flows web pages require all at once, and
let 'em rip, instead of 15 at a time, without destroying the network,
web PLT would get much better.
> My opinion, the best units for bloat is packets for UDP or bytes for
> TCP. Min delay is a proxy measurement.
bytes, period. bytes = time. Sure most udp today is small packets but
quic and videconferencing change that.
>
> Little's law allows one to compute this though does assume the network
> is in a stable state over the measurement interval. In the real world,
> this probably is rarely true. So we, in test & measurement engineering,
> force the standing state with some sort of measurement co-traffic and
> call it "working conditions" or equivalent. ;)
There was an extremely long, nuanced debate about little's law and
where it applies, last year, here:
https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html
I don't want to go into it, again.
>
> Bob
> > Bob, Sebastian,
> >
> > not being active on your topic, just to add what I observed on
> > congestion:
> > - starts with an increase of jitter, but measured minimum delays still
> > remain constant. Technically, a queue builds up some of the time, but
> > it isn't present permanently.
> > - buffer fill reaches a "steady state", called bufferbloat on access I
> > think; technically, OWD increases also for the minimum delays, jitter
> > now decreases (what you've described that as "the delay magnitude"
> > decreases or "minimum CDF shift" respectively, if I'm correct). I'd
> > expect packet loss to occur, once the buffer fill is on steady state,
> > but loss might be randomly distributed and could be of a low
> > percentage.
> > - a sudden rather long load burst may cause a jump-start to
> > "steady-state" buffer fill. The above holds for a slow but steady load
> > increase (where the measurement frequency determines the timescale
> > qualifying "slow").
> > - in the end, max-min delay or delay distribution/jitter likely isn't
> > an easy to handle single metric to identify congestion.
> >
> > Regards,
> >
> > Ruediger
> >
> >
> >> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
> >> <rpm@lists.bufferbloat.net> wrote:
> >>
> >> Bufferbloat shifts the minimum of the latency or OWD CDF.
> >
> > [SM] Thank you for spelling this out explicitly, I only worked on a
> > vage implicit assumption along those lines. However what I want to
> > avoid is using delay magnitude itself as classifier between high and
> > low load condition as that seems statistically uncouth to then show
> > that the delay differs between the two classes;).
> > Yet, your comment convinced me that my current load threshold (at
> > least for the high load condition) probably is too small, exactly
> > because the "base" of the high-load CDFs coincides with the base of
> > the low-load CDFs implying that the high-load class contains too many
> > samples with decent delay (which after all is one of the goals of the
> > whole autorate endeavor).
> >
> >
> >> A suggestion is to disable x-axis auto-scaling and start from zero.
> >
> > [SM] Will reconsider. I started with start at zero, end then switched
> > to an x-range that starts with the delay corresponding to 0.01% for
> > the reflector/condition with the lowest such value and stops at 97.5%
> > for the reflector/condition with the highest delay value. My rationale
> > is that the base delay/path delay of each reflector is not all that
> > informative* (and it can still be learned from reading the x-axis),
> > the long tail > 50% however is where I expect most differences so I
> > want to emphasize this and finally I wanted to avoid that the actual
> > "curvy" part gets compressed so much that all lines more or less
> > coincide. As I said, I will reconsider this
> >
> >
> > *) We also maintain individual baselines per reflector, so I could
> > just plot the differences from baseline, but that would essentially
> > equalize all reflectors, and I think having a plot that easily shows
> > reflectors with outlying base delay can be informative when selecting
> > reflector candidates. However once we actually switch to OWDs baseline
> > correction might be required anyways, as due to colck differences ICMP
> > type 13/14 data can have massive offsets that are mostly indicative of
> > un synched clocks**.
> >
> > **) This is whyI would prefer to use NTP servers as reflectors with
> > NTP requests, my expectation is all of these should be reasonably
> > synced by default so that offsets should be in the sane range....
> >
> >
> >>
> >> Bob
> >>> For about 2 years now the cake w-adaptive bandwidth project has been
> >>> exploring techniques to lightweightedly sense bandwidth and
> >>> buffering problems. One of my favorites was their discovery that ICMP
> >>> type 13 got them working OWD from millions of ipv4 devices!
> >>> They've also explored leveraging ntp and multiple other methods, and
> >>> have scripts available that do a good job of compensating for 5g and
> >>> starlink's misbehaviors.
> >>> They've also pioneered a whole bunch of new graphing techniques,
> >>> which I do wish were used more than single number summaries
> >>> especially in analyzing the behaviors of new metrics like rpm,
> >>> samknows, ookla, and
> >>> RFC9097 - to see what is being missed.
> >>> There are thousands of posts about this research topic, a new post on
> >>> OWD just went by here.
> >>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
> >>> and of course, I love flent's enormous graphing toolset for
> >>> simulating and analyzing complex network behaviors.
> >> _______________________________________________
> >> Rpm mailing list
> >> Rpm@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> > https://www.ietf.org/mailman/listinfo/ippm
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
--
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 19:44 ` Dave Taht
@ 2022-11-02 20:37 ` rjmcmahon
2022-11-02 21:13 ` Sebastian Moeller
1 sibling, 0 replies; 29+ messages in thread
From: rjmcmahon @ 2022-11-02 20:37 UTC (permalink / raw)
To: Dave Taht; +Cc: Ruediger.Geib, rpm, ippm
I used iperf 2's Little's law calculation to find the buffer sizes
designed in by our hardware team(s). They were surprised that the
numbers exactly matched their designs - Little applied - and I never saw
either the hardware nor its design spec.
It seems reasonable to use something if & when it works and is useful.
The challenge seems to be knowing the limits of any claims (or
simulations.) I think engineers do this much when we assume linearity
over some small interval as an example in finite element analysis and
structures:
https://control.com/technical-articles/the-difference-between-linear-and-nonlinear-finite-element-analysis-fea/
Bob
> On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
> <rpm@lists.bufferbloat.net> wrote:
>>
>> Most measuring bloat are ignoring queue build up phase and rather
>> start
>> taking measurements after the bottleneck queue is in a standing state.
>
> +10. It's the slow start transient that is holding things back. If we
> could, for example
> open up the 110+ objects and flows web pages require all at once, and
> let 'em rip, instead of 15 at a time, without destroying the network,
> web PLT would get much better.
>
>> My opinion, the best units for bloat is packets for UDP or bytes for
>> TCP. Min delay is a proxy measurement.
>
> bytes, period. bytes = time. Sure most udp today is small packets but
> quic and videconferencing change that.
>
>>
>> Little's law allows one to compute this though does assume the network
>> is in a stable state over the measurement interval. In the real world,
>> this probably is rarely true. So we, in test & measurement
>> engineering,
>> force the standing state with some sort of measurement co-traffic and
>> call it "working conditions" or equivalent. ;)
>
> There was an extremely long, nuanced debate about little's law and
> where it applies, last year, here:
>
> https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html
>
> I don't want to go into it, again.
>
>>
>> Bob
>> > Bob, Sebastian,
>> >
>> > not being active on your topic, just to add what I observed on
>> > congestion:
>> > - starts with an increase of jitter, but measured minimum delays still
>> > remain constant. Technically, a queue builds up some of the time, but
>> > it isn't present permanently.
>> > - buffer fill reaches a "steady state", called bufferbloat on access I
>> > think; technically, OWD increases also for the minimum delays, jitter
>> > now decreases (what you've described that as "the delay magnitude"
>> > decreases or "minimum CDF shift" respectively, if I'm correct). I'd
>> > expect packet loss to occur, once the buffer fill is on steady state,
>> > but loss might be randomly distributed and could be of a low
>> > percentage.
>> > - a sudden rather long load burst may cause a jump-start to
>> > "steady-state" buffer fill. The above holds for a slow but steady load
>> > increase (where the measurement frequency determines the timescale
>> > qualifying "slow").
>> > - in the end, max-min delay or delay distribution/jitter likely isn't
>> > an easy to handle single metric to identify congestion.
>> >
>> > Regards,
>> >
>> > Ruediger
>> >
>> >
>> >> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>> >> <rpm@lists.bufferbloat.net> wrote:
>> >>
>> >> Bufferbloat shifts the minimum of the latency or OWD CDF.
>> >
>> > [SM] Thank you for spelling this out explicitly, I only worked on a
>> > vage implicit assumption along those lines. However what I want to
>> > avoid is using delay magnitude itself as classifier between high and
>> > low load condition as that seems statistically uncouth to then show
>> > that the delay differs between the two classes;).
>> > Yet, your comment convinced me that my current load threshold (at
>> > least for the high load condition) probably is too small, exactly
>> > because the "base" of the high-load CDFs coincides with the base of
>> > the low-load CDFs implying that the high-load class contains too many
>> > samples with decent delay (which after all is one of the goals of the
>> > whole autorate endeavor).
>> >
>> >
>> >> A suggestion is to disable x-axis auto-scaling and start from zero.
>> >
>> > [SM] Will reconsider. I started with start at zero, end then switched
>> > to an x-range that starts with the delay corresponding to 0.01% for
>> > the reflector/condition with the lowest such value and stops at 97.5%
>> > for the reflector/condition with the highest delay value. My rationale
>> > is that the base delay/path delay of each reflector is not all that
>> > informative* (and it can still be learned from reading the x-axis),
>> > the long tail > 50% however is where I expect most differences so I
>> > want to emphasize this and finally I wanted to avoid that the actual
>> > "curvy" part gets compressed so much that all lines more or less
>> > coincide. As I said, I will reconsider this
>> >
>> >
>> > *) We also maintain individual baselines per reflector, so I could
>> > just plot the differences from baseline, but that would essentially
>> > equalize all reflectors, and I think having a plot that easily shows
>> > reflectors with outlying base delay can be informative when selecting
>> > reflector candidates. However once we actually switch to OWDs baseline
>> > correction might be required anyways, as due to colck differences ICMP
>> > type 13/14 data can have massive offsets that are mostly indicative of
>> > un synched clocks**.
>> >
>> > **) This is whyI would prefer to use NTP servers as reflectors with
>> > NTP requests, my expectation is all of these should be reasonably
>> > synced by default so that offsets should be in the sane range....
>> >
>> >
>> >>
>> >> Bob
>> >>> For about 2 years now the cake w-adaptive bandwidth project has been
>> >>> exploring techniques to lightweightedly sense bandwidth and
>> >>> buffering problems. One of my favorites was their discovery that ICMP
>> >>> type 13 got them working OWD from millions of ipv4 devices!
>> >>> They've also explored leveraging ntp and multiple other methods, and
>> >>> have scripts available that do a good job of compensating for 5g and
>> >>> starlink's misbehaviors.
>> >>> They've also pioneered a whole bunch of new graphing techniques,
>> >>> which I do wish were used more than single number summaries
>> >>> especially in analyzing the behaviors of new metrics like rpm,
>> >>> samknows, ookla, and
>> >>> RFC9097 - to see what is being missed.
>> >>> There are thousands of posts about this research topic, a new post on
>> >>> OWD just went by here.
>> >>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>> >>> and of course, I love flent's enormous graphing toolset for
>> >>> simulating and analyzing complex network behaviors.
>> >> _______________________________________________
>> >> Rpm mailing list
>> >> Rpm@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/rpm
>> >
>> > _______________________________________________
>> > ippm mailing list
>> > ippm@ietf.org
>> > https://www.ietf.org/mailman/listinfo/ippm
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 19:44 ` Dave Taht
2022-11-02 20:37 ` rjmcmahon
@ 2022-11-02 21:13 ` Sebastian Moeller
1 sibling, 0 replies; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-02 21:13 UTC (permalink / raw)
To: Dave Täht; +Cc: rjmcmahon, Rpm, Ruediger.Geib, ippm
> On Nov 2, 2022, at 20:44, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>
> On Wed, Nov 2, 2022 at 12:29 PM rjmcmahon via Rpm
> <rpm@lists.bufferbloat.net> wrote:
>>
>> Most measuring bloat are ignoring queue build up phase and rather start
>> taking measurements after the bottleneck queue is in a standing state.
>
> +10. It's the slow start transient that is holding things back.
[SM] From my naive perspective slow start has a few facets:
a) it is absolutely the right approach conceptually with no reliable prior knowledge of the capacity the best we can do is to probe it by increasing the sending rate over time (and the current exponential growth is already plenty aggressive*)
b) Since this needs feedback from the remote endpoint at best we can figure out 1 RTT later whether we sent at acceptable rate
c) we want to go as fast as reasonable
d) but not any faster ;)
e) the trickiest part is really deciding when to leave slow start's aggressive rate increase per RTT regime so that the >= 1 RTT "blind" spot does not lead to too big a transient queue spike.
f) The slow start phase ca be relatively short, so the less averaging we need to do to decide when to drop out of slow start the better as averaging costs time.
IMHO the best way forward would be to switch from bit-banging the queue state (as in L4S' design) over multiple packets to sending a multi-bit queue occupancy signal per-packet, then the sender should be able to figure out the rate of queueing change as a function of sending rate change and predict a reasonable time to switch to congestion avoidance without having to wait for a drop (assuming the remote end signals these queue occupancy signals back to the sender in a timely fashion)...
(This is really just transfering the ideas from Arslan, Serhat, and Nick McKeown. ‘Switches Know the Exact Amount of Congestion’. In Proceedings of the 2019 Workshop on Buffer Sizing, 1–6, 2019. to slow-start).
*) Two factors to play with is size of the starting batch (aka initial window) and the actual factor of increase per RTT, people statred playing with the first but so far seem reasonable enough not to touch the second ;)
> If we
> could, for example
> open up the 110+ objects and flows web pages require all at once, and
> let 'em rip, instead of 15 at a time, without destroying the network,
> web PLT would get much better.
>
>> My opinion, the best units for bloat is packets for UDP or bytes for
>> TCP. Min delay is a proxy measurement.
>
> bytes, period. bytes = time. Sure most udp today is small packets but
> quic and videconferencing change that.
>
>>
>> Little's law allows one to compute this though does assume the network
>> is in a stable state over the measurement interval. In the real world,
>> this probably is rarely true. So we, in test & measurement engineering,
>> force the standing state with some sort of measurement co-traffic and
>> call it "working conditions" or equivalent. ;)
>
> There was an extremely long, nuanced debate about little's law and
> where it applies, last year, here:
>
> https://lists.bufferbloat.net/pipermail/cake/2021-July/005540.html
>
> I don't want to go into it, again.
>
>>
>> Bob
>>> Bob, Sebastian,
>>>
>>> not being active on your topic, just to add what I observed on
>>> congestion:
>>> - starts with an increase of jitter, but measured minimum delays still
>>> remain constant. Technically, a queue builds up some of the time, but
>>> it isn't present permanently.
>>> - buffer fill reaches a "steady state", called bufferbloat on access I
>>> think; technically, OWD increases also for the minimum delays, jitter
>>> now decreases (what you've described that as "the delay magnitude"
>>> decreases or "minimum CDF shift" respectively, if I'm correct). I'd
>>> expect packet loss to occur, once the buffer fill is on steady state,
>>> but loss might be randomly distributed and could be of a low
>>> percentage.
>>> - a sudden rather long load burst may cause a jump-start to
>>> "steady-state" buffer fill. The above holds for a slow but steady load
>>> increase (where the measurement frequency determines the timescale
>>> qualifying "slow").
>>> - in the end, max-min delay or delay distribution/jitter likely isn't
>>> an easy to handle single metric to identify congestion.
>>>
>>> Regards,
>>>
>>> Ruediger
>>>
>>>
>>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm
>>>> <rpm@lists.bufferbloat.net> wrote:
>>>>
>>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>>
>>> [SM] Thank you for spelling this out explicitly, I only worked on a
>>> vage implicit assumption along those lines. However what I want to
>>> avoid is using delay magnitude itself as classifier between high and
>>> low load condition as that seems statistically uncouth to then show
>>> that the delay differs between the two classes;).
>>> Yet, your comment convinced me that my current load threshold (at
>>> least for the high load condition) probably is too small, exactly
>>> because the "base" of the high-load CDFs coincides with the base of
>>> the low-load CDFs implying that the high-load class contains too many
>>> samples with decent delay (which after all is one of the goals of the
>>> whole autorate endeavor).
>>>
>>>
>>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>>
>>> [SM] Will reconsider. I started with start at zero, end then switched
>>> to an x-range that starts with the delay corresponding to 0.01% for
>>> the reflector/condition with the lowest such value and stops at 97.5%
>>> for the reflector/condition with the highest delay value. My rationale
>>> is that the base delay/path delay of each reflector is not all that
>>> informative* (and it can still be learned from reading the x-axis),
>>> the long tail > 50% however is where I expect most differences so I
>>> want to emphasize this and finally I wanted to avoid that the actual
>>> "curvy" part gets compressed so much that all lines more or less
>>> coincide. As I said, I will reconsider this
>>>
>>>
>>> *) We also maintain individual baselines per reflector, so I could
>>> just plot the differences from baseline, but that would essentially
>>> equalize all reflectors, and I think having a plot that easily shows
>>> reflectors with outlying base delay can be informative when selecting
>>> reflector candidates. However once we actually switch to OWDs baseline
>>> correction might be required anyways, as due to colck differences ICMP
>>> type 13/14 data can have massive offsets that are mostly indicative of
>>> un synched clocks**.
>>>
>>> **) This is whyI would prefer to use NTP servers as reflectors with
>>> NTP requests, my expectation is all of these should be reasonably
>>> synced by default so that offsets should be in the sane range....
>>>
>>>
>>>>
>>>> Bob
>>>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>>>> exploring techniques to lightweightedly sense bandwidth and
>>>>> buffering problems. One of my favorites was their discovery that ICMP
>>>>> type 13 got them working OWD from millions of ipv4 devices!
>>>>> They've also explored leveraging ntp and multiple other methods, and
>>>>> have scripts available that do a good job of compensating for 5g and
>>>>> starlink's misbehaviors.
>>>>> They've also pioneered a whole bunch of new graphing techniques,
>>>>> which I do wish were used more than single number summaries
>>>>> especially in analyzing the behaviors of new metrics like rpm,
>>>>> samknows, ookla, and
>>>>> RFC9097 - to see what is being missed.
>>>>> There are thousands of posts about this research topic, a new post on
>>>>> OWD just went by here.
>>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>>> and of course, I love flent's enormous graphing toolset for
>>>>> simulating and analyzing complex network behaviors.
>>>> _______________________________________________
>>>> Rpm mailing list
>>>> Rpm@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/rpm
>>>
>>> _______________________________________________
>>> ippm mailing list
>>> ippm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/ippm
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
>
>
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 9:41 ` [Rpm] [ippm] " Ruediger.Geib
2022-11-02 19:29 ` rjmcmahon
@ 2022-11-02 21:41 ` Sebastian Moeller
2022-11-03 8:20 ` Ruediger.Geib
1 sibling, 1 reply; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-02 21:41 UTC (permalink / raw)
To: Ruediger.Geib; +Cc: rjmcmahon, Rpm, ippm
Dear Ruediger,
thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow.
The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
*) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
**) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>
> Bob, Sebastian,
>
> not being active on your topic, just to add what I observed on congestion:
[SM] I will try to explain how/if we could exploit your observations for our controller
> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
> - buffer fill reaches a "steady state", called bufferbloat on access I think
[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
> - a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill.
[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
Many Thanks & Kind Regards
Sebastian
>
> Regards,
>
> Ruediger
>
>
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>
> [SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
> Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>
>
>> A suggestion is to disable x-axis auto-scaling and start from zero.
>
> [SM] Will reconsider. I started with start at zero, end then switched to an x-range that starts with the delay corresponding to 0.01% for the reflector/condition with the lowest such value and stops at 97.5% for the reflector/condition with the highest delay value. My rationale is that the base delay/path delay of each reflector is not all that informative* (and it can still be learned from reading the x-axis), the long tail > 50% however is where I expect most differences so I want to emphasize this and finally I wanted to avoid that the actual "curvy" part gets compressed so much that all lines more or less coincide. As I said, I will reconsider this
>
>
> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>
> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>
>
>>
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense bandwidth and
>>> buffering problems. One of my favorites was their discovery that ICMP
>>> type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques,
>>> which I do wish were used more than single number summaries
>>> especially in analyzing the behaviors of new metrics like rpm,
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post on
>>> OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-02 21:41 ` Sebastian Moeller
@ 2022-11-03 8:20 ` Ruediger.Geib
2022-11-03 8:57 ` Sebastian Moeller
0 siblings, 1 reply; 29+ messages in thread
From: Ruediger.Geib @ 2022-11-03 8:20 UTC (permalink / raw)
To: moeller0; +Cc: rjmcmahon, rpm, ippm
Hi Sebastian,
[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.
Regards,
Ruediger
-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <moeller0@gmx.de>
Gesendet: Mittwoch, 2. November 2022 22:41
An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm <rpm@lists.bufferbloat.net>; ippm@ietf.org
Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and buffering
Dear Ruediger,
thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow.
The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
*) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
**) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>
> Bob, Sebastian,
>
> not being active on your topic, just to add what I observed on congestion:
[SM] I will try to explain how/if we could exploit your observations for our controller
> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
[SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
> - buffer fill reaches a "steady state", called bufferbloat on access I
> think
[SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
[SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
[SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
> - a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill.
[SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
[SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
Many Thanks & Kind Regards
Sebastian
>
> Regards,
>
> Ruediger
>
>
>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>
>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>
> [SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
> Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>
>
>> A suggestion is to disable x-axis auto-scaling and start from zero.
>
> [SM] Will reconsider. I started with start at zero, end then switched
> to an x-range that starts with the delay corresponding to 0.01% for
> the reflector/condition with the lowest such value and stops at 97.5%
> for the reflector/condition with the highest delay value. My rationale
> is that the base delay/path delay of each reflector is not all that
> informative* (and it can still be learned from reading the x-axis),
> the long tail > 50% however is where I expect most differences so I
> want to emphasize this and finally I wanted to avoid that the actual
> "curvy" part gets compressed so much that all lines more or less
> coincide. As I said, I will reconsider this
>
>
> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>
> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>
>
>>
>> Bob
>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>> exploring techniques to lightweightedly sense bandwidth and
>>> buffering problems. One of my favorites was their discovery that
>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>> They've also explored leveraging ntp and multiple other methods, and
>>> have scripts available that do a good job of compensating for 5g and
>>> starlink's misbehaviors.
>>> They've also pioneered a whole bunch of new graphing techniques,
>>> which I do wish were used more than single number summaries
>>> especially in analyzing the behaviors of new metrics like rpm,
>>> samknows, ookla, and
>>> RFC9097 - to see what is being missed.
>>> There are thousands of posts about this research topic, a new post
>>> on OWD just went by here.
>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>> and of course, I love flent's enormous graphing toolset for
>>> simulating and analyzing complex network behaviors.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-03 8:20 ` Ruediger.Geib
@ 2022-11-03 8:57 ` Sebastian Moeller
2022-11-03 11:25 ` Ruediger.Geib
0 siblings, 1 reply; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-03 8:57 UTC (permalink / raw)
To: Ruediger.Geib; +Cc: rjmcmahon, Rpm, IETF IPPM WG, Andrew Somerville
Hi Ruediger,
> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>
> Hi Sebastian,
>
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
>
> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.
[SM] Exactly! We do try to account for some of these factors to some degree:
a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).
And yes, the goal is not to model congestion veridically, but really to offer something that is "good enough" to help while not being excessively complicated (though admittedly as often happens with heuristics we added more and more code once testing showed our previous approach as too simplistic). Also this effort over on the OpenWrt list is actually a set of multiple parallel efforts with cake-autorate* only one ~4 alternatives (implemented in perl, lua, awk, and bash) that cross-pollinated each other and IMHO all approaches improved from open discussion and friendly competition.
*) And cake-autorate is neither the first nor the most innovative of the bunch (that would IMHO the perl implementation that had existed first and had single-handedly demonstrated the suitability of ICMP type 13/14 timestamps for directional "congestion" measurements) but the one with the most dedicated and open-minded main developer** (CCd) integrating all ideas (and coming up with new ones), without whom neither of the approaches would have become public or even happened I think. It is also the approach that appears to have the most active development momentum and testing community.
**) A developer that also uses this code on a day-to-day basis, so "dog-fooding" at its finest, that really helps I think to stay pragmatic ;)
>
> Regards,
>
> Ruediger
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Sebastian Moeller <moeller0@gmx.de>
> Gesendet: Mittwoch, 2. November 2022 22:41
> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm <rpm@lists.bufferbloat.net>; ippm@ietf.org
> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and buffering
>
> Dear Ruediger,
>
> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow.
> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
>
> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
>
>
>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>>
>> Bob, Sebastian,
>>
>> not being active on your topic, just to add what I observed on congestion:
>
> [SM] I will try to explain how/if we could exploit your observations for our controller
>
>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
>
> [SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
>
>> - buffer fill reaches a "steady state", called bufferbloat on access I
>> think
>
> [SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
>
>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
>
> [SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
>
>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
>
> [SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
>
>> - a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill.
>
> [SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
>
>
>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
>
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
>
> Many Thanks & Kind Regards
> Sebastian
>
>
>>
>> Regards,
>>
>> Ruediger
>>
>>
>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>>
>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>
>> [SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
>> Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>>
>>
>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>
>> [SM] Will reconsider. I started with start at zero, end then switched
>> to an x-range that starts with the delay corresponding to 0.01% for
>> the reflector/condition with the lowest such value and stops at 97.5%
>> for the reflector/condition with the highest delay value. My rationale
>> is that the base delay/path delay of each reflector is not all that
>> informative* (and it can still be learned from reading the x-axis),
>> the long tail > 50% however is where I expect most differences so I
>> want to emphasize this and finally I wanted to avoid that the actual
>> "curvy" part gets compressed so much that all lines more or less
>> coincide. As I said, I will reconsider this
>>
>>
>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>>
>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>>
>>
>>>
>>> Bob
>>>> For about 2 years now the cake w-adaptive bandwidth project has been
>>>> exploring techniques to lightweightedly sense bandwidth and
>>>> buffering problems. One of my favorites was their discovery that
>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>> They've also explored leveraging ntp and multiple other methods, and
>>>> have scripts available that do a good job of compensating for 5g and
>>>> starlink's misbehaviors.
>>>> They've also pioneered a whole bunch of new graphing techniques,
>>>> which I do wish were used more than single number summaries
>>>> especially in analyzing the behaviors of new metrics like rpm,
>>>> samknows, ookla, and
>>>> RFC9097 - to see what is being missed.
>>>> There are thousands of posts about this research topic, a new post
>>>> on OWD just went by here.
>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>> and of course, I love flent's enormous graphing toolset for
>>>> simulating and analyzing complex network behaviors.
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>>
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-03 8:57 ` Sebastian Moeller
@ 2022-11-03 11:25 ` Ruediger.Geib
2022-11-03 11:48 ` Sebastian Moeller
0 siblings, 1 reply; 29+ messages in thread
From: Ruediger.Geib @ 2022-11-03 11:25 UTC (permalink / raw)
To: moeller0; +Cc: rjmcmahon, rpm, ippm, aesomerville
Hi Sebastian,
Quick feedback [RG2]
-----------------------------------------
Hi Ruediger,
> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>
> Hi Sebastian,
>
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
>
> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.
[SM] Exactly! We do try to account for some of these factors to some degree:
a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
[RG2] Allright, that approach is reasonable.
b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).
[RG2] That's a tricky and absolutely required part, differentiating a path change from congestion. A comment on B) - please keep in mind, that the baseline path may have a smaller delay, but immediately face congestion. That may not be an issue, if the congestion (buffer-fill) to be detected reliably adds latency in a range significantly above the monitored physical delay.
<snip>
> Regards,
>
> Ruediger
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Sebastian Moeller <moeller0@gmx.de>
> Gesendet: Mittwoch, 2. November 2022 22:41
> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm
> <rpm@lists.bufferbloat.net>; ippm@ietf.org
> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and
> buffering
>
> Dear Ruediger,
>
> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow.
> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
>
> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
>
>
>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>>
>> Bob, Sebastian,
>>
>> not being active on your topic, just to add what I observed on congestion:
>
> [SM] I will try to explain how/if we could exploit your observations
> for our controller
>
>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
>
> [SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
>
>> - buffer fill reaches a "steady state", called bufferbloat on access
>> I think
>
> [SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
>
>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
>
> [SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
>
>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
>
> [SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
>
>> - a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill.
>
> [SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
>
>
>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
>
> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
>
> Many Thanks & Kind Regards
> Sebastian
>
>
>>
>> Regards,
>>
>> Ruediger
>>
>>
>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>>
>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>
>> [SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
>> Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>>
>>
>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>
>> [SM] Will reconsider. I started with start at zero, end then
>> switched to an x-range that starts with the delay corresponding to
>> 0.01% for the reflector/condition with the lowest such value and
>> stops at 97.5% for the reflector/condition with the highest delay
>> value. My rationale is that the base delay/path delay of each
>> reflector is not all that
>> informative* (and it can still be learned from reading the x-axis),
>> the long tail > 50% however is where I expect most differences so I
>> want to emphasize this and finally I wanted to avoid that the actual
>> "curvy" part gets compressed so much that all lines more or less
>> coincide. As I said, I will reconsider this
>>
>>
>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>>
>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>>
>>
>>>
>>> Bob
>>>> For about 2 years now the cake w-adaptive bandwidth project has
>>>> been exploring techniques to lightweightedly sense bandwidth and
>>>> buffering problems. One of my favorites was their discovery that
>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>> They've also explored leveraging ntp and multiple other methods,
>>>> and have scripts available that do a good job of compensating for
>>>> 5g and starlink's misbehaviors.
>>>> They've also pioneered a whole bunch of new graphing techniques,
>>>> which I do wish were used more than single number summaries
>>>> especially in analyzing the behaviors of new metrics like rpm,
>>>> samknows, ookla, and
>>>> RFC9097 - to see what is being missed.
>>>> There are thousands of posts about this research topic, a new post
>>>> on OWD just went by here.
>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>> and of course, I love flent's enormous graphing toolset for
>>>> simulating and analyzing complex network behaviors.
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>>
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://www.ietf.org/mailman/listinfo/ippm
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering
2022-11-03 11:25 ` Ruediger.Geib
@ 2022-11-03 11:48 ` Sebastian Moeller
0 siblings, 0 replies; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-03 11:48 UTC (permalink / raw)
To: Ruediger.Geib; +Cc: rjmcmahon, Rpm, IETF IPPM WG, aesomerville
Hi Ruediger,
many thanks for the interesting discussion.
> On Nov 3, 2022, at 12:25, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>
> Hi Sebastian,
>
> Quick feedback [RG2]
>
> -----------------------------------------
> Hi Ruediger,
>
>> On Nov 3, 2022, at 09:20, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>>
>> Hi Sebastian,
>>
>> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect.
>>
>> RG: I agree. And I appreciate "well enough" - many factors may impact delays along a measurement path, causing e.g. temporal noise or permanent change.
>
> [SM] Exactly! We do try to account for some of these factors to some degree:
> a) we use a set of diverse reflectors by default and use a voting mechanism and declare "congested_state" only if W out of the last X delay samples (from Y reflectors) were above threshold Z (all of W, X, Y, and Z are configurable to tune the code for specific links, but the defaults seem to already improve perceived latency-under-load noticeably for many/most users that reported back to us).
> [RG2] Allright, that approach is reasonable.
>
> b) we keep individual "baseline" values for each reflector that iintend to model the path-dependent delay component, we adjust the baseline with two rules:
> A) if a sample was larger that baseline, we feed it into an EWMA to update/increase the baseline slightly (which will allow the baseline to slowly grow to larger values if e.g. a path changed and now is longer); the assumption here is that underestimating the "true" baseline will make the controller more "trigger-happy" which will keep latency low at the expense of potential throughput, and we prefer that transient loss in potential throughput over a similarly transient decrease in responsiveness.
> B) if a sample is smaller than the baseline we immediately update the baseline to that value. (One rationale for that is that during prolonged congestion epochs the EWMA method in A) will have artificially increased our base line estimate so we want to be quick to correct it again if the congestion relaxes even for a short duration; the other is to quickly adapt in case a path change results in a shorter path delay; again the principle is that we value responsiveness over throughput).
>
> [RG2] That's a tricky and absolutely required part, differentiating a path change from congestion. A comment on B) - please keep in mind, that the baseline path may have a smaller delay, but immediately face congestion. That may not be an issue, if the congestion (buffer-fill) to be detected reliably adds latency in a range significantly above the monitored physical delay.
[SM2] Yes! We have to deal with a set of known unknowns here and hence we need to employ a few heuristics. About the min-delay being inflated already by persistent congestion, yes we are blind to that no ifs not buts. Rule B) tries to keep such errors as short as possible in that we reduce our baseline estimate immediately after we have evidence that we over-estimated it. That does as you observed help little if congestion stays persistently high. Personally I am lucky enough that I rarelely encountered something like this on my access link, but that is just that luck.
Our main goal is to control a known bottleneck (like an access link) and not the full internet path to a specific target location so most relevant traffic should be self generated so we have some rough estimate on the relative load of our link (but the available capacity share might still depend on other users in a shared segment so our load estimate is only approximate otherwise our job would be done already, if we know what fraction of our available capacity we currentky use and our current traffic rate we could directly calculate the shaper limit to keep responsiveness high ;) ).
Regards
Sebastian
>
>
> <snip>
>
>> Regards,
>>
>> Ruediger
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Sebastian Moeller <moeller0@gmx.de>
>> Gesendet: Mittwoch, 2. November 2022 22:41
>> An: Geib, Rüdiger <Ruediger.Geib@telekom.de>
>> Cc: rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm
>> <rpm@lists.bufferbloat.net>; ippm@ietf.org
>> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and
>> buffering
>>
>> Dear Ruediger,
>>
>> thank you very much for your helpful information. I will chew over this and see how/if I can exploit these "development of congestion observations" somehow.
>> The goal of these plots is not primarily to detect congestion* (that would be the core of autorate's functionality, detect increases in delay and respond in reducing the shaper rate to counter act them), but more to show how well this works (the current rationale is that compared to a situation without traffic shaping the difference in high versus low-load CDFs should be noticeably** smaller).
>>
>> *) autorate will be in control of an artificial bottleneck and we do measure the achieved throughput per direction, so we can reason about "congestion" based on throughput and delay; the loading is organic in that we simply measure the traffic volume per time of what travels over the relevant interfaces, the delay measurements however are active, which has its pros and cons...
>> **) Maybe even run a few statistical tests, like Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly smaller". I feel a parametric t-test might not be in order here, with delay PDFs decidedly non-normal in shape (then again they likely are mono-modal, so t-test would still work okayish in spite of its core assumption being violated).
>>
>>
>>> On Nov 2, 2022, at 10:41, <Ruediger.Geib@telekom.de> <Ruediger.Geib@telekom.de> wrote:
>>>
>>> Bob, Sebastian,
>>>
>>> not being active on your topic, just to add what I observed on congestion:
>>
>> [SM] I will try to explain how/if we could exploit your observations
>> for our controller
>>
>>> - starts with an increase of jitter, but measured minimum delays still remain constant. Technically, a queue builds up some of the time, but it isn't present permanently.
>>
>> [SM] So in that phase we would expect CDFs to have different slopes, higher variance should result in shallower slope? As for using this insight for the actual controller, I am not sure how that would work; maybe maintaining a "jitter" base line per reflector and test whether each new sample deviates significantly from that base line? That is similar to the approach we are currently taking with delay/RTT.
>>
>>> - buffer fill reaches a "steady state", called bufferbloat on access
>>> I think
>>
>> [SM] I would call it buffer bloat if that steady-state results in too high delays increases (which to a degree is a subjective judgement). Although in accordance with the Nichols/Jacobsen analogy of buffers/queues as shock absorbers a queue with with acceptable steady-state induced delay might not work too well to even out occasional bursts?
>>
>>> ; technically, OWD increases also for the minimum delays, jitter now decreases (what you've described that as "the delay magnitude" decreases or "minimum CDF shift" respectively, if I'm correct).
>>
>> [SM] That is somewhat unfortunate as it is harder to detect quickly than something that simply increases and stays high (like RTT).
>>
>>> I'd expect packet loss to occur, once the buffer fill is on steady state, but loss might be randomly distributed and could be of a low percentage.
>>
>> [SM] Loss is mostly invisible to our controller (it would need to affect our relatively thin active delay measurement traffic we have no insight into the rest of the traffic), but more than that the controller's goal is to avoid this situation so hopefully it will be rare and transient.
>>
>>> - a sudden rather long load burst may cause a jump-start to "steady-state" buffer fill.
>>
>> [SM] As would a rather steep drop in available capacity with traffic in-flight sized to the previous larger capacity. This is e.g. what can be observed over shared media like docsis/cable and GSM successors.
>>
>>
>>> The above holds for a slow but steady load increase (where the measurement frequency determines the timescale qualifying "slow").
>>> - in the end, max-min delay or delay distribution/jitter likely isn't an easy to handle single metric to identify congestion.
>>
>> [SM] Pragmatically we work with delay increase over baseline, which seems to work well enough to be useful, while it is unlikely to be perfect. The CDFs I plotted are really just for making sense post hoc out of the logged data... (cake-autorate is currently designed to maintain a "flight-recorder" log buffer that can be extracted after noticeable events, and I am trying to come up with how to slice and dice the data to help explain "noticeable events" from the limited log data we have).
>>
>> Many Thanks & Kind Regards
>> Sebastian
>>
>>
>>>
>>> Regards,
>>>
>>> Ruediger
>>>
>>>
>>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm <rpm@lists.bufferbloat.net> wrote:
>>>>
>>>> Bufferbloat shifts the minimum of the latency or OWD CDF.
>>>
>>> [SM] Thank you for spelling this out explicitly, I only worked on a vage implicit assumption along those lines. However what I want to avoid is using delay magnitude itself as classifier between high and low load condition as that seems statistically uncouth to then show that the delay differs between the two classes;).
>>> Yet, your comment convinced me that my current load threshold (at least for the high load condition) probably is too small, exactly because the "base" of the high-load CDFs coincides with the base of the low-load CDFs implying that the high-load class contains too many samples with decent delay (which after all is one of the goals of the whole autorate endeavor).
>>>
>>>
>>>> A suggestion is to disable x-axis auto-scaling and start from zero.
>>>
>>> [SM] Will reconsider. I started with start at zero, end then
>>> switched to an x-range that starts with the delay corresponding to
>>> 0.01% for the reflector/condition with the lowest such value and
>>> stops at 97.5% for the reflector/condition with the highest delay
>>> value. My rationale is that the base delay/path delay of each
>>> reflector is not all that
>>> informative* (and it can still be learned from reading the x-axis),
>>> the long tail > 50% however is where I expect most differences so I
>>> want to emphasize this and finally I wanted to avoid that the actual
>>> "curvy" part gets compressed so much that all lines more or less
>>> coincide. As I said, I will reconsider this
>>>
>>>
>>> *) We also maintain individual baselines per reflector, so I could just plot the differences from baseline, but that would essentially equalize all reflectors, and I think having a plot that easily shows reflectors with outlying base delay can be informative when selecting reflector candidates. However once we actually switch to OWDs baseline correction might be required anyways, as due to colck differences ICMP type 13/14 data can have massive offsets that are mostly indicative of un synched clocks**.
>>>
>>> **) This is whyI would prefer to use NTP servers as reflectors with NTP requests, my expectation is all of these should be reasonably synced by default so that offsets should be in the sane range....
>>>
>>>
>>>>
>>>> Bob
>>>>> For about 2 years now the cake w-adaptive bandwidth project has
>>>>> been exploring techniques to lightweightedly sense bandwidth and
>>>>> buffering problems. One of my favorites was their discovery that
>>>>> ICMP type 13 got them working OWD from millions of ipv4 devices!
>>>>> They've also explored leveraging ntp and multiple other methods,
>>>>> and have scripts available that do a good job of compensating for
>>>>> 5g and starlink's misbehaviors.
>>>>> They've also pioneered a whole bunch of new graphing techniques,
>>>>> which I do wish were used more than single number summaries
>>>>> especially in analyzing the behaviors of new metrics like rpm,
>>>>> samknows, ookla, and
>>>>> RFC9097 - to see what is being missed.
>>>>> There are thousands of posts about this research topic, a new post
>>>>> on OWD just went by here.
>>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793
>>>>> and of course, I love flent's enormous graphing toolset for
>>>>> simulating and analyzing complex network behaviors.
>>>> _______________________________________________
>>>> Rpm mailing list
>>>> Rpm@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/rpm
>>>
>>> _______________________________________________
>>> ippm mailing list
>>> ippm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/ippm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-01 14:51 ` MORTON JR., AL
@ 2022-11-04 17:14 ` MORTON JR., AL
2022-11-04 18:12 ` rjmcmahon
2022-11-04 19:10 ` Sebastian Moeller
0 siblings, 2 replies; 29+ messages in thread
From: MORTON JR., AL @ 2022-11-04 17:14 UTC (permalink / raw)
To: MORTON JR., AL, Dave Taht, rjmcmahon; +Cc: ippm, Rpm
Hi all,
I have been working through the threads on misery metrics and lightweight sensing of bw and buffering, and with those insights and gold-nuggets in mind, I'm iterating through testing the combinations of tools that Dave That and Bob McMahon suggested.
earlier this week, Dave wrote:
> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
goresponsiveness installed flawlessly - very nice instructions and getting-started info.
Comparison of NetQual (networkQuality -vs), UDPST -A c, gores/goresponsiveness
Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-stream service
(capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
NetQual UDPST (RFC 9097) gores
DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap RPM DelLD
882 788 m 76ms 8ms 967 28 0-16 127 (1382) 43ms
892 1036 h 58ms 8 966 27 0-18 128 (2124) 28ms
887 1304 h 46ms 6 969 27 0-18 130 (1478) 41ms
885 1008 h 60ms 8 967 28 0-22 127 (1490) 40ms
894 1383 h 43ms 11 967 28 0-15 133 (2731) 22ms
NetQual UDPST (RFC 9097) gores
UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap RPM DelLD
21 327 m 183ms 8ms 22 (51) 28 0-253 12
21 413 m 145ms 8 22 (43) 28 0-255 15
22 273 m 220ms 6 23 (53) 28 0-259 10
21 377 m 159ms 8 23 (51) 28 0-250 10
22 281 m 214ms 11 23 (52) 28 0-250 6
These tests were conducted in a round-robin fashion to minimize the possibility of network variations between measurements:
NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
NetQual indicates the same reduced capacity in Downstream when compared to UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer capacity, dep. on VLAN tag).
Upstream capacities are not very different (a factor that made TCP methods more viable many years ago when most consumer access speeds were limited to 10's of Megabits).
gores reports significantly lower capacity in both downstream and upstream measurements, a factor of 7 less than NetQual for downstream. Interestingly, the reduced capacity (taken as the working load) results in higher responsiveness: RPM meas are higher and loaded delays are lower for downstream.
The comparison of networkQuality and goresponsiveness is somewhat confounded by the need to use the Apple server infrastructure for both these methods (the documentation provides this option - thanks!). I don't have admin access to our server at the moment. But the measured differences are large despite the confounding factor.
goresponsiveness has its own, very different output format than networkQuality. There isn't a comparable "-v" option other than -debug (which is extremely detailed). gores only reports RPM for the downstream.
I hope that these results will prompt more of the principle evangelists and coders to weigh-in.
It's also worth noting that the RTTrange reported by UDPST is the range above the minimum RTT, and represents the entire 10 second test. The consistency of the maximum of the range (~255ms) seems to indicate that UDPST has characterized the length of the upstream buffer during measurements.
I'm sure there is more to observe that is prompted by these measurements; comments welcome!
Al
> -----Original Message-----
> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> Sent: Tuesday, November 1, 2022 10:51 AM
> To: Dave Taht <dave.taht@gmail.com>
> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
>
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
>
> Hi Dave,
> Thanks for trying UDPST (RFC 9097)!
>
> Something you might try with starlink:
> use the -X option and UDPST will generate random payloads.
>
> The measurements with -X will reflect the uncompressed that are possible.
> I tried this on a ship-board Internet access: uncompressed rate was 100kbps.
>
> A few more quick replies below,
> Al
>
> > -----Original Message-----
> > From: Dave Taht <dave.taht@gmail.com>
> > Sent: Tuesday, November 1, 2022 12:22 AM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> > metrics
> >
> > Dear Al:
> >
> > OK, I took your udpst tool for a spin.
> >
> > NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> > good feature set, VERY fast,
> [acm]
> Len Ciavattone (my partner in crime on several measurement projects) is the
> lead coder: he has implemented many measurement tools extremely efficiently,
> this one in C-lang.
>
> > and in very brief testing, seemed
> > to be accurate in the starlink case, though it's hard to tell with
> > them as the rate changes every 15s.
> [acm]
> Great! Our guiding principle developing UDPST has been to test the accuracy of
> measurements against a ground-truth. It pays-off.
>
> >
> > I filed a couple bug reports on trivial stuff:
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > SswH_opYmEw$
> [acm]
> much appreciated... We have an OB-UDPST project meeting this Friday, can
> discuss then.
>
> >
> > (Adding diffserv and ecn washing or marking detection would be a nice
> > feature to have)
> >
> > Aside from the sheer joy coming from the "it compiles! and runs!"
> > phase I haven't looked much further.
> >
> > I left a copy running on one of my starlink testbeds -
> > fremont.starlink.taht.net - if anyone wants to try it. It's
> > instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> > version from bob, but close), and now udpst, and good to about a gbit.
> >
> > nice tool!
> [acm]
> Thanks again!
>
> >
> > Has anyone here played with crusader? (
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$ )
> >
> > On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> > >
> > > > > have you tried irtt?
> >
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$ )
> > > > I have not. Seems like a reasonable tool for UDP testing. The feature I
> > didn't like in my scan of the documentation is the use of Inter-packet delay
> > variation (IPDV) instead of packet delay variation (PDV): variation from the
> > minimum (or reference) delay. The morbidly curious can find my analysis in
> RFC
> > 5481:
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > T7QSlc$
> > >
> > > irtt was meant to simulate high speed voip and one day
> > > videoconferencing. Please inspect the json output
> > > for other metrics. Due to OS limits it is typically only accurate to a
> > > 3ms interval. One thing it does admirably is begin to expose the
> > > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > > technologies, as well as request/grant systems like cable and gpon,
> > > especially when otherwise idle.
> > >
> > > Here is a highres plot of starlink's behaviors from last year:
> > > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> > bandwidth-
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > >
> > > clearly showing them "optimizing for bandwidth" and changing next sat
> > > hop, and about a 40ms interval of buffering between these switches.
> > > I'd published elsewhere, if anyone cares, a preliminary study of what
> > > starlink's default behaviors did to cubic and BBR...
> > >
> > > >
> > > > irtt's use of IPDV means that the results won’t compare with UDPST, and
> > possibly networkQuality. But I may give it a try anyway...
> > >
> > > The more the merrier! Someday the "right" metrics will arrive.
> > >
> > > As a side note, this paper focuses on RAN uplink latency
> > >
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHgvqerjg$ which I think
> > > is a major barrier to most forms of 5G actually achieving good
> > > performance in a FPS game, if it is true for more RANs. I'd like more
> > > to be testing uplink latencies idle and with load, on all
> > > technologies.
> > >
> > > >
> > > > thanks again, Dave.
> > > > Al
> > > >
> > > > > -----Original Message-----
> > > > > From: Dave Taht <dave.taht@gmail.com>
> > > > > Sent: Monday, October 31, 2022 12:52 PM
> > > > > To: MORTON JR., AL <acmorton@att.com>
> > > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > > metrics
> > > > >
> > > > > Thank you very much for the steer to RFC9097. I'd completely missed
> > that.
> > > > >
> > > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> wrote:
> > > > > >
> > > > > > (astute readers may have guessed that I pressed "send" too soon on
> > previous
> > > > > message...)
> > > > > >
> > > > > > I also conducted upstream tests this time, here are the results:
> > > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High
> and
> > > > > Medium)
> > > > > >
> > > > > > Net Qual UDPST (RFC9097)
> Ookla
> > > > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
> UpCap
> > > > > Ping(no load)
> > > > > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
> > 8
> > > > > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22
> > 8
> > > > > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22
> > 9
> > > > > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22
> > 7
> > > > > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22
> > 9
> > > > > >
> > > > > > The Upstream capacity measurements reflect an interesting feature
> that
> > we
> > > > > can reliably and repeatably measure with UDPST. The first ~3 seconds
> of
> > > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> this
> > > > > behavior in its 1 second sub-interval measurements and has a bimodal
> > reporting
> > > > > option that divides the complete measurement interval in two time
> > intervals to
> > > > > report an initial (turbo) max capacity and a steady-state max capacity
> > for the
> > > > > later intervals. The UDPST capacity results present both measurements:
> > steady-
> > > > > state first.
> > > > >
> > > > > Certainly we can expect bi-model distributions from many ISPs, as, for
> > > > > one thing, the "speedboost" concept remains popular, except that it's
> > > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > > > > it is often configured "sneakily", in that it doesn't kick in for the
> > > > > typical observed duration of the test, for some, they cut the
> > > > > available bandwidth about 20s in, others, 1 or 5 minutes.
> > > > >
> > > > > One of my biggest issues with the rpm spec so far is that it should,
> > > > > at least, sometimes, run randomly longer than the overly short
> > > > > interval it runs for and the tools also allow for manual override of
> > length.
> > > > >
> > > > > we caught a lot of tomfoolery with flent's rrul test running by
> default
> > for
> > > > > 1m.
> > > > >
> > > > > Also, AQMs on the path can take a while to find the optimal drop or
> mark
> > rate.
> > > > >
> > > > > >
> > > > > > The capacity processing in networkQuality and Ookla appear to report
> > the
> > > > > steady-state result.
> > > > >
> > > > > Ookla used to basically report the last result. Also it's not a good
> > > > > indicator of web traffic behavior at all, watching the curve
> > > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > > > > (40ms)....
> > > > >
> > > > > So adding another mode - how quickly is peak bandwidth actually
> > > > > reached, would be nice.
> > > > >
> > > > > I haven't poked into the current iteration of the goresponsiveness
> > > > > test at all: https://urldefense.com/v3/__https://github.com/network-
> > > > >
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > > > > would be good to try collecting more statistics and histograms and
> > > > > methods of analyzing the data in that libre-source version.
> > > > >
> > > > > How does networkQuality compare vs a vs your tool vs a vs
> > goresponsiveness?
> > > > >
> > > > > >I watched the upstream capacity measurements on the Ookla app, and
> > could
> > > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> > most of
> > > > > the test which determined the final result.
> > > > >
> > > > > I tend to get upset when I see ookla's new test flash a peak result in
> > > > > the seconds and then settle on some lower number somehow.
> > > > > So far as I know they are only sampling the latency every 250ms.
> > > > >
> > > > > >
> > > > > > The working latency is about 200ms in networkQuality and about 280ms
> > as
> > > > > measured by UDPST (RFC9097). Note that the networkQuality minimum
> delay
> > is
> > > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> > difference
> > > > > in working latency. Also, we used the very dynamic Type C load
> > > > > adjustment/search algorithm in UDPST during all of this testing, which
> > could
> > > > > explain the higher working latency to some degree.
> > > > > >
> > > > > > So, it's worth noting that the measurements needed for assessing
> > working
> > > > > latency/responsiveness are available in the UDPST utility, and that
> the
> > UDPST
> > > > > measurements are conducted on UDP transport (used by a growing
> fraction
> > of
> > > > > Internet traffic).
> > > > >
> > > > > Thx, didn't know of this work til now!
> > > > >
> > > > > have you tried irtt?
> > > > >
> > > > > >
> > > > > > comments welcome of course,
> > > > > > Al
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > Sent: Sunday, October 30, 2022 8:09 PM
> > > > > > > To: ippm@ietf.org
> > > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > > > Latency"
> > > > > > > metrics
> > > > > > >
> > > > > > >
> > > > > > > Hi again RPM friends and IPPM'ers,
> > > > > > >
> > > > > > > As promised, I repeated the tests shared last week, this time
> using
> > both
> > > > > the
> > > > > > > verbose (-v) and sequential (-s) dwn/up test options of
> > networkQuality. I
> > > > > > > followed Sebastian's calculations as well.
> > > > > > >
> > > > > > > Working Latency & Capacity Summary
> > > > > > >
> > > > > > > Net Qual UDPST
> > Ookla
> > > > > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
> > DnCap
> > > > > > > Ping(no load)
> > > > > > > 885 916 m 66ms 8ms 970 28 0-20
> 940
> > 8
> > > > > > > 888 1355 h 44ms 8ms 966 28 0-23
> 940
> > 8
> > > > > > > 891 1109 h 54ms 8ms 968 27 0-19
> 940
> > 9
> > > > > > > 887 1141 h 53ms 11ms 966 27 0-18
> 937
> > 7
> > > > > > > 884 1151 h 52ms 9ms 968 28 0-20
> 937
> > 9
> > > > > > >
> > > > > > > With the sequential test option, I noticed that networkQuality
> > achieved
> > > > > nearly
> > > > > > > the maximum capacity reported almost immediately at the start of a
> > test.
> > > > > > > However, the reported capacities are low by about 60Mbps,
> especially
> > when
> > > > > > > compared to the Ookla TCP measurements.
> > > > > > >
> > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> high
> > end of
> > > > > the
> > > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
> > > > > networkQuality
> > > > > > > RPM measurements were categorized as "High". There doesn't seem to
> > be much
> > > > > > > buffering in the downstream direction.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > > > > > > Sent: Monday, October 24, 2022 6:36 PM
> > > > > > > > To: ippm@ietf.org
> > > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > > > > > > metrics
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi RPM friends and IPPM'ers,
> > > > > > > >
> > > > > > > > I was wondering what a comparison of some of the "working
> latency"
> > > > > metrics
> > > > > > > > would look like, so I ran some tests using a service on DOCSIS
> > 3.1, with
> > > > > the
> > > > > > > > downlink provisioned for 1Gbps.
> > > > > > > >
> > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
> > Ookla
> > > > > > > Speedtest
> > > > > > > > with as similar connectivity as possible (but we know that the
> > traffic
> > > > > will
> > > > > > > > diverge to different servers and we can't change that aspect).
> > > > > > > >
> > > > > > > > Here's a quick summary of yesterday's results:
> > > > > > > >
> > > > > > > > Working Latency & Capacity Summary
> > > > > > > >
> > > > > > > > Net Qual UDPST Ookla
> > > > > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap
> > Ping(no
> > > > > load)
> > > > > > > > 878 62 970 28 0-19 941 6
> > > > > > > > 891 92 970 27 0-20 940 7
> > > > > > > > 891 120 966 28 0-22 937 9
> > > > > > > > 890 112 970 28 0-21 940 8
> > > > > > > > 903 70 970 28 0-16 935 9
> > > > > > > >
> > > > > > > > Note: all RPM values were categorized as Low.
> > > > > > > >
> > > > > > > > networkQuality downstream capacities are always on the low side
> > compared
> > > > > to
> > > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
> > what
> > > > > Ookla
> > > > > > > > achieved. I think that a longer test duration might be needed to
> > achieve
> > > > > the
> > > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
> > observed
> > > > > were
> > > > > > > > certainly headed in the right direction. (I recently upgraded to
> > > > > Monterey
> > > > > > > 12.6
> > > > > > > > on my MacBook, so should have the latest version.)
> > > > > > > >
> > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
> > should
> > > > > have
> > > > > > > > run the tests with the -v option to help with comparisons. I'll
> > repeat
> > > > > this
> > > > > > > > test when I can make time.
> > > > > > > >
> > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during
> the
> > test)
> > > > > and
> > > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
> > very
> > > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
> > this
> > > > > access
> > > > > > > and
> > > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
> > Chicago
> > > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
> > measurements
> > > > > were
> > > > > > > run
> > > > > > > > on an Ubuntu VM in my MacBook.
> > > > > > > >
> > > > > > > > The big disappointment was that the Ookla desktop app I updated
> > over the
> > > > > > > > weekend did not include the new responsiveness metric! I
> included
> > the
> > > > > ping
> > > > > > > > results anyway, and it was clearly using a server in the nearby
> > area.
> > > > > > > >
> > > > > > > > So, I have some more work to do, but I hope this is interesting-
> > enough
> > > > > to
> > > > > > > > start some comparison discussions, and bring-out some
> suggestions.
> > > > > > > >
> > > > > > > > happy testing all,
> > > > > > > > Al
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > ippm mailing list
> > > > > > > > ippm@ietf.org
> > > > > > > >
> > > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > > >
> > > > > > >
> > > > >
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ippm mailing list
> > > > > > > ippm@ietf.org
> > > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > > > > T!g-
> > > > >
> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > > > >
> > > > > > _______________________________________________
> > > > > > ippm mailing list
> > > > > > ippm@ietf.org
> > > > > >
> > > > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > > >
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > > > KXU$
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > This song goes out to all the folk that thought Stadia would work:
> > > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > mushroom-
> > > > > song-activity-6981366665607352320-
> > > > >
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > > > c48iQFub34zz4iE$
> > > > > Dave Täht CEO, TekLibre, LLC
> > >
> > >
> > >
> > > --
> > > This song goes out to all the folk that thought Stadia would work:
> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > mushroom-song-activity-6981366665607352320-
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHLHDpSWs$
> > > Dave Täht CEO, TekLibre, LLC
> >
> >
> >
> > --
> > This song goes out to all the folk that thought Stadia would work:
> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> mushroom-
> > song-activity-6981366665607352320-
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > IEUcSswHLHDpSWs$
> > Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-04 17:14 ` MORTON JR., AL
@ 2022-11-04 18:12 ` rjmcmahon
2022-11-04 18:58 ` MORTON JR., AL
2022-11-04 19:10 ` Sebastian Moeller
1 sibling, 1 reply; 29+ messages in thread
From: rjmcmahon @ 2022-11-04 18:12 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: Dave Taht, ippm, Rpm
Any iperf 2 data? https://sourceforge.net/projects/iperf2/
Bob
> Hi all,
>
> I have been working through the threads on misery metrics and
> lightweight sensing of bw and buffering, and with those insights and
> gold-nuggets in mind, I'm iterating through testing the combinations
> of tools that Dave That and Bob McMahon suggested.
>
> earlier this week, Dave wrote:
>> How does networkQuality compare vs a vs your tool vs a vs
>> goresponsiveness?
>
> goresponsiveness installed flawlessly - very nice instructions and
> getting-started info.
>
> Comparison of NetQual (networkQuality -vs), UDPST -A c,
> gores/goresponsiveness
>
> Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps
> down-stream service
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
>
> NetQual UDPST (RFC 9097) gores
> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> RPM DelLD
> 882 788 m 76ms 8ms 967 28 0-16 127
> (1382) 43ms
> 892 1036 h 58ms 8 966 27 0-18 128
> (2124) 28ms
> 887 1304 h 46ms 6 969 27 0-18 130
> (1478) 41ms
> 885 1008 h 60ms 8 967 28 0-22 127
> (1490) 40ms
> 894 1383 h 43ms 11 967 28 0-15 133
> (2731) 22ms
>
> NetQual UDPST (RFC 9097) gores
> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> RPM DelLD
> 21 327 m 183ms 8ms 22 (51) 28 0-253 12
> 21 413 m 145ms 8 22 (43) 28 0-255 15
> 22 273 m 220ms 6 23 (53) 28 0-259 10
> 21 377 m 159ms 8 23 (51) 28 0-250 10
> 22 281 m 214ms 11 23 (52) 28 0-250 6
>
> These tests were conducted in a round-robin fashion to minimize the
> possibility of network variations between measurements:
> NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest -
> repeat
>
> NetQual indicates the same reduced capacity in Downstream when
> compared to UDPST (940Mbps is the max for TCP payloads, while 967-970
> is max for IP-layer capacity, dep. on VLAN tag).
> Upstream capacities are not very different (a factor that made TCP
> methods more viable many years ago when most consumer access speeds
> were limited to 10's of Megabits).
>
> gores reports significantly lower capacity in both downstream and
> upstream measurements, a factor of 7 less than NetQual for downstream.
> Interestingly, the reduced capacity (taken as the working load)
> results in higher responsiveness: RPM meas are higher and loaded
> delays are lower for downstream.
>
> The comparison of networkQuality and goresponsiveness is somewhat
> confounded by the need to use the Apple server infrastructure for both
> these methods (the documentation provides this option - thanks!). I
> don't have admin access to our server at the moment. But the measured
> differences are large despite the confounding factor.
>
> goresponsiveness has its own, very different output format than
> networkQuality. There isn't a comparable "-v" option other than -debug
> (which is extremely detailed). gores only reports RPM for the
> downstream.
>
> I hope that these results will prompt more of the principle
> evangelists and coders to weigh-in.
>
> It's also worth noting that the RTTrange reported by UDPST is the
> range above the minimum RTT, and represents the entire 10 second test.
> The consistency of the maximum of the range (~255ms) seems to indicate
> that UDPST has characterized the length of the upstream buffer during
> measurements.
>
> I'm sure there is more to observe that is prompted by these
> measurements; comments welcome!
>
> Al
>
>
>> -----Original Message-----
>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> Sent: Tuesday, November 1, 2022 10:51 AM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
>> Latency"
>> metrics
>>
>> *** Security Advisory: This Message Originated Outside of AT&T ***.
>> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
>> information.
>>
>> Hi Dave,
>> Thanks for trying UDPST (RFC 9097)!
>>
>> Something you might try with starlink:
>> use the -X option and UDPST will generate random payloads.
>>
>> The measurements with -X will reflect the uncompressed that are
>> possible.
>> I tried this on a ship-board Internet access: uncompressed rate was
>> 100kbps.
>>
>> A few more quick replies below,
>> Al
>>
>> > -----Original Message-----
>> > From: Dave Taht <dave.taht@gmail.com>
>> > Sent: Tuesday, November 1, 2022 12:22 AM
>> > To: MORTON JR., AL <acmorton@att.com>
>> > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> > metrics
>> >
>> > Dear Al:
>> >
>> > OK, I took your udpst tool for a spin.
>> >
>> > NICE! 120k binary (I STILL work on machines with only 4MB of flash),
>> > good feature set, VERY fast,
>> [acm]
>> Len Ciavattone (my partner in crime on several measurement projects)
>> is the
>> lead coder: he has implemented many measurement tools extremely
>> efficiently,
>> this one in C-lang.
>>
>> > and in very brief testing, seemed
>> > to be accurate in the starlink case, though it's hard to tell with
>> > them as the rate changes every 15s.
>> [acm]
>> Great! Our guiding principle developing UDPST has been to test the
>> accuracy of
>> measurements against a ground-truth. It pays-off.
>>
>> >
>> > I filed a couple bug reports on trivial stuff:
>> >
>> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
>> >
>> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
>> > SswH_opYmEw$
>> [acm]
>> much appreciated... We have an OB-UDPST project meeting this Friday,
>> can
>> discuss then.
>>
>> >
>> > (Adding diffserv and ecn washing or marking detection would be a nice
>> > feature to have)
>> >
>> > Aside from the sheer joy coming from the "it compiles! and runs!"
>> > phase I haven't looked much further.
>> >
>> > I left a copy running on one of my starlink testbeds -
>> > fremont.starlink.taht.net - if anyone wants to try it. It's
>> > instrumented with netperf, flent, irtt, iperf2 (not quite the latest
>> > version from bob, but close), and now udpst, and good to about a gbit.
>> >
>> > nice tool!
>> [acm]
>> Thanks again!
>>
>> >
>> > Has anyone here played with crusader? (
>> >
>> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
>> > oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$ )
>> >
>> > On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>> > >
>> > > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>> > >
>> > > > > have you tried irtt?
>> >
>> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
>> > H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$ )
>> > > > I have not. Seems like a reasonable tool for UDP testing. The feature I
>> > didn't like in my scan of the documentation is the use of Inter-packet delay
>> > variation (IPDV) instead of packet delay variation (PDV): variation from the
>> > minimum (or reference) delay. The morbidly curious can find my analysis in
>> RFC
>> > 5481:
>> >
>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
>> >
>> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
>> > T7QSlc$
>> > >
>> > > irtt was meant to simulate high speed voip and one day
>> > > videoconferencing. Please inspect the json output
>> > > for other metrics. Due to OS limits it is typically only accurate to a
>> > > 3ms interval. One thing it does admirably is begin to expose the
>> > > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
>> > > technologies, as well as request/grant systems like cable and gpon,
>> > > especially when otherwise idle.
>> > >
>> > > Here is a highres plot of starlink's behaviors from last year:
>> > > https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
>> > bandwidth-
>> >
>> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
>> > KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
>> > >
>> > > clearly showing them "optimizing for bandwidth" and changing next sat
>> > > hop, and about a 40ms interval of buffering between these switches.
>> > > I'd published elsewhere, if anyone cares, a preliminary study of what
>> > > starlink's default behaviors did to cubic and BBR...
>> > >
>> > > >
>> > > > irtt's use of IPDV means that the results won’t compare with UDPST, and
>> > possibly networkQuality. But I may give it a try anyway...
>> > >
>> > > The more the merrier! Someday the "right" metrics will arrive.
>> > >
>> > > As a side note, this paper focuses on RAN uplink latency
>> > >
>> >
>> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
>> >
>> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHgvqerjg$ which I think
>> > > is a major barrier to most forms of 5G actually achieving good
>> > > performance in a FPS game, if it is true for more RANs. I'd like more
>> > > to be testing uplink latencies idle and with load, on all
>> > > technologies.
>> > >
>> > > >
>> > > > thanks again, Dave.
>> > > > Al
>> > > >
>> > > > > -----Original Message-----
>> > > > > From: Dave Taht <dave.taht@gmail.com>
>> > > > > Sent: Monday, October 31, 2022 12:52 PM
>> > > > > To: MORTON JR., AL <acmorton@att.com>
>> > > > > Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
>> > Latency"
>> > > > > metrics
>> > > > >
>> > > > > Thank you very much for the steer to RFC9097. I'd completely missed
>> > that.
>> > > > >
>> > > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
>> wrote:
>> > > > > >
>> > > > > > (astute readers may have guessed that I pressed "send" too soon on
>> > previous
>> > > > > message...)
>> > > > > >
>> > > > > > I also conducted upstream tests this time, here are the results:
>> > > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High
>> and
>> > > > > Medium)
>> > > > > >
>> > > > > > Net Qual UDPST (RFC9097)
>> Ookla
>> > > > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
>> UpCap
>> > > > > Ping(no load)
>> > > > > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
>> > 8
>> > > > > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22
>> > 8
>> > > > > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22
>> > 9
>> > > > > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22
>> > 7
>> > > > > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22
>> > 9
>> > > > > >
>> > > > > > The Upstream capacity measurements reflect an interesting feature
>> that
>> > we
>> > > > > can reliably and repeatably measure with UDPST. The first ~3 seconds
>> of
>> > > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
>> this
>> > > > > behavior in its 1 second sub-interval measurements and has a bimodal
>> > reporting
>> > > > > option that divides the complete measurement interval in two time
>> > intervals to
>> > > > > report an initial (turbo) max capacity and a steady-state max capacity
>> > for the
>> > > > > later intervals. The UDPST capacity results present both measurements:
>> > steady-
>> > > > > state first.
>> > > > >
>> > > > > Certainly we can expect bi-model distributions from many ISPs, as, for
>> > > > > one thing, the "speedboost" concept remains popular, except that it's
>> > > > > misnamed, as it should be called speed-subtract or speed-lose. Worse,
>> > > > > it is often configured "sneakily", in that it doesn't kick in for the
>> > > > > typical observed duration of the test, for some, they cut the
>> > > > > available bandwidth about 20s in, others, 1 or 5 minutes.
>> > > > >
>> > > > > One of my biggest issues with the rpm spec so far is that it should,
>> > > > > at least, sometimes, run randomly longer than the overly short
>> > > > > interval it runs for and the tools also allow for manual override of
>> > length.
>> > > > >
>> > > > > we caught a lot of tomfoolery with flent's rrul test running by
>> default
>> > for
>> > > > > 1m.
>> > > > >
>> > > > > Also, AQMs on the path can take a while to find the optimal drop or
>> mark
>> > rate.
>> > > > >
>> > > > > >
>> > > > > > The capacity processing in networkQuality and Ookla appear to report
>> > the
>> > > > > steady-state result.
>> > > > >
>> > > > > Ookla used to basically report the last result. Also it's not a good
>> > > > > indicator of web traffic behavior at all, watching the curve
>> > > > > go up much more slowly in their test on say, fiber 2ms, vs starlink,
>> > > > > (40ms)....
>> > > > >
>> > > > > So adding another mode - how quickly is peak bandwidth actually
>> > > > > reached, would be nice.
>> > > > >
>> > > > > I haven't poked into the current iteration of the goresponsiveness
>> > > > > test at all: https://urldefense.com/v3/__https://github.com/network-
>> > > > >
>> >
>> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
>> > > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
>> > > > > would be good to try collecting more statistics and histograms and
>> > > > > methods of analyzing the data in that libre-source version.
>> > > > >
>> > > > > How does networkQuality compare vs a vs your tool vs a vs
>> > goresponsiveness?
>> > > > >
>> > > > > >I watched the upstream capacity measurements on the Ookla app, and
>> > could
>> > > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
>> > most of
>> > > > > the test which determined the final result.
>> > > > >
>> > > > > I tend to get upset when I see ookla's new test flash a peak result in
>> > > > > the seconds and then settle on some lower number somehow.
>> > > > > So far as I know they are only sampling the latency every 250ms.
>> > > > >
>> > > > > >
>> > > > > > The working latency is about 200ms in networkQuality and about 280ms
>> > as
>> > > > > measured by UDPST (RFC9097). Note that the networkQuality minimum
>> delay
>> > is
>> > > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the
>> > difference
>> > > > > in working latency. Also, we used the very dynamic Type C load
>> > > > > adjustment/search algorithm in UDPST during all of this testing, which
>> > could
>> > > > > explain the higher working latency to some degree.
>> > > > > >
>> > > > > > So, it's worth noting that the measurements needed for assessing
>> > working
>> > > > > latency/responsiveness are available in the UDPST utility, and that
>> the
>> > UDPST
>> > > > > measurements are conducted on UDP transport (used by a growing
>> fraction
>> > of
>> > > > > Internet traffic).
>> > > > >
>> > > > > Thx, didn't know of this work til now!
>> > > > >
>> > > > > have you tried irtt?
>> > > > >
>> > > > > >
>> > > > > > comments welcome of course,
>> > > > > > Al
>> > > > > >
>> > > > > > > -----Original Message-----
>> > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > > > > > Sent: Sunday, October 30, 2022 8:09 PM
>> > > > > > > To: ippm@ietf.org
>> > > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working
>> > > > > Latency"
>> > > > > > > metrics
>> > > > > > >
>> > > > > > >
>> > > > > > > Hi again RPM friends and IPPM'ers,
>> > > > > > >
>> > > > > > > As promised, I repeated the tests shared last week, this time
>> using
>> > both
>> > > > > the
>> > > > > > > verbose (-v) and sequential (-s) dwn/up test options of
>> > networkQuality. I
>> > > > > > > followed Sebastian's calculations as well.
>> > > > > > >
>> > > > > > > Working Latency & Capacity Summary
>> > > > > > >
>> > > > > > > Net Qual UDPST
>> > Ookla
>> > > > > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
>> > DnCap
>> > > > > > > Ping(no load)
>> > > > > > > 885 916 m 66ms 8ms 970 28 0-20
>> 940
>> > 8
>> > > > > > > 888 1355 h 44ms 8ms 966 28 0-23
>> 940
>> > 8
>> > > > > > > 891 1109 h 54ms 8ms 968 27 0-19
>> 940
>> > 9
>> > > > > > > 887 1141 h 53ms 11ms 966 27 0-18
>> 937
>> > 7
>> > > > > > > 884 1151 h 52ms 9ms 968 28 0-20
>> 937
>> > 9
>> > > > > > >
>> > > > > > > With the sequential test option, I noticed that networkQuality
>> > achieved
>> > > > > nearly
>> > > > > > > the maximum capacity reported almost immediately at the start of a
>> > test.
>> > > > > > > However, the reported capacities are low by about 60Mbps,
>> especially
>> > when
>> > > > > > > compared to the Ookla TCP measurements.
>> > > > > > >
>> > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
>> high
>> > end of
>> > > > > the
>> > > > > > > RTTrange), for example 54ms compared to (27+19=46). Most of the
>> > > > > networkQuality
>> > > > > > > RPM measurements were categorized as "High". There doesn't seem to
>> > be much
>> > > > > > > buffering in the downstream direction.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > -----Original Message-----
>> > > > > > > > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> > > > > > > > Sent: Monday, October 24, 2022 6:36 PM
>> > > > > > > > To: ippm@ietf.org
>> > > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working
>> > Latency"
>> > > > > > > > metrics
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Hi RPM friends and IPPM'ers,
>> > > > > > > >
>> > > > > > > > I was wondering what a comparison of some of the "working
>> latency"
>> > > > > metrics
>> > > > > > > > would look like, so I ran some tests using a service on DOCSIS
>> > 3.1, with
>> > > > > the
>> > > > > > > > downlink provisioned for 1Gbps.
>> > > > > > > >
>> > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and
>> > Ookla
>> > > > > > > Speedtest
>> > > > > > > > with as similar connectivity as possible (but we know that the
>> > traffic
>> > > > > will
>> > > > > > > > diverge to different servers and we can't change that aspect).
>> > > > > > > >
>> > > > > > > > Here's a quick summary of yesterday's results:
>> > > > > > > >
>> > > > > > > > Working Latency & Capacity Summary
>> > > > > > > >
>> > > > > > > > Net Qual UDPST Ookla
>> > > > > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap
>> > Ping(no
>> > > > > load)
>> > > > > > > > 878 62 970 28 0-19 941 6
>> > > > > > > > 891 92 970 27 0-20 940 7
>> > > > > > > > 891 120 966 28 0-22 937 9
>> > > > > > > > 890 112 970 28 0-21 940 8
>> > > > > > > > 903 70 970 28 0-16 935 9
>> > > > > > > >
>> > > > > > > > Note: all RPM values were categorized as Low.
>> > > > > > > >
>> > > > > > > > networkQuality downstream capacities are always on the low side
>> > compared
>> > > > > to
>> > > > > > > > others. We would expect about 940Mbps for TCP, and that's mostly
>> > what
>> > > > > Ookla
>> > > > > > > > achieved. I think that a longer test duration might be needed to
>> > achieve
>> > > > > the
>> > > > > > > > actual 1Gbps capacity with networkQuality; intermediate values
>> > observed
>> > > > > were
>> > > > > > > > certainly headed in the right direction. (I recently upgraded to
>> > > > > Monterey
>> > > > > > > 12.6
>> > > > > > > > on my MacBook, so should have the latest version.)
>> > > > > > > >
>> > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I
>> > should
>> > > > > have
>> > > > > > > > run the tests with the -v option to help with comparisons. I'll
>> > repeat
>> > > > > this
>> > > > > > > > test when I can make time.
>> > > > > > > >
>> > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during
>> the
>> > test)
>> > > > > and
>> > > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to
>> > very
>> > > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded
>> > this
>> > > > > access
>> > > > > > > and
>> > > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in
>> > Chicago
>> > > > > > > > conducting tests, so the minimum 28ms is typical. UDPST
>> > measurements
>> > > > > were
>> > > > > > > run
>> > > > > > > > on an Ubuntu VM in my MacBook.
>> > > > > > > >
>> > > > > > > > The big disappointment was that the Ookla desktop app I updated
>> > over the
>> > > > > > > > weekend did not include the new responsiveness metric! I
>> included
>> > the
>> > > > > ping
>> > > > > > > > results anyway, and it was clearly using a server in the nearby
>> > area.
>> > > > > > > >
>> > > > > > > > So, I have some more work to do, but I hope this is interesting-
>> > enough
>> > > > > to
>> > > > > > > > start some comparison discussions, and bring-out some
>> suggestions.
>> > > > > > > >
>> > > > > > > > happy testing all,
>> > > > > > > > Al
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > _______________________________________________
>> > > > > > > > ippm mailing list
>> > > > > > > > ippm@ietf.org
>> > > > > > > >
>> > > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > > > > >
>> > > > > > >
>> > > > >
>> >
>> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>> > > > > > >
>> > > > > > > _______________________________________________
>> > > > > > > ippm mailing list
>> > > > > > > ippm@ietf.org
>> > > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > > > > T!g-
>> > > > >
>> > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>> > > > > >
>> > > > > > _______________________________________________
>> > > > > > ippm mailing list
>> > > > > > ippm@ietf.org
>> > > > > >
>> > > > >
>> >
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> > > > >
>> >
>> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
>> > > > > KXU$
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > This song goes out to all the folk that thought Stadia would work:
>> > > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> > mushroom-
>> > > > > song-activity-6981366665607352320-
>> > > > >
>> >
>> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
>> > > > > c48iQFub34zz4iE$
>> > > > > Dave Täht CEO, TekLibre, LLC
>> > >
>> > >
>> > >
>> > > --
>> > > This song goes out to all the folk that thought Stadia would work:
>> > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> > mushroom-song-activity-6981366665607352320-
>> >
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHLHDpSWs$
>> > > Dave Täht CEO, TekLibre, LLC
>> >
>> >
>> >
>> > --
>> > This song goes out to all the folk that thought Stadia would work:
>> > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> mushroom-
>> > song-activity-6981366665607352320-
>> >
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>> > IEUcSswHLHDpSWs$
>> > Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-04 18:12 ` rjmcmahon
@ 2022-11-04 18:58 ` MORTON JR., AL
0 siblings, 0 replies; 29+ messages in thread
From: MORTON JR., AL @ 2022-11-04 18:58 UTC (permalink / raw)
To: rjmcmahon; +Cc: Dave Taht, ippm, Rpm
> -----Original Message-----
> From: rjmcmahon <rjmcmahon@rjmcmahon.com>
> Sent: Friday, November 4, 2022 2:12 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: Dave Taht <dave.taht@gmail.com>; ippm@ietf.org; Rpm
> <rpm@lists.bufferbloat.net>
> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> metrics
>
> Any iperf 2 data?
> https://urldefense.com/v3/__https://sourceforge.net/projects/iperf2/__;!!BhdT!
> jm0H0ICmGbrmtAj1pqCwuDPP8s6l40wS3hZvt7i5KLoRmNgT9UXRigC0rIfnQk6NjodSBgtUaBP7yL
> coVw$
>
[acm]
Not yet, I hope to make some progress this weekend with the other tools...
Al
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-04 17:14 ` MORTON JR., AL
2022-11-04 18:12 ` rjmcmahon
@ 2022-11-04 19:10 ` Sebastian Moeller
2022-11-05 19:36 ` MORTON JR., AL
1 sibling, 1 reply; 29+ messages in thread
From: Sebastian Moeller @ 2022-11-04 19:10 UTC (permalink / raw)
To: MORTON JR., AL; +Cc: Dave Täht, rjmcmahon, Rpm, ippm, Will Hawkins
Hi Al,
> On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm <rpm@lists.bufferbloat.net> wrote:
>
> Hi all,
>
> I have been working through the threads on misery metrics and lightweight sensing of bw and buffering, and with those insights and gold-nuggets in mind, I'm iterating through testing the combinations of tools that Dave That and Bob McMahon suggested.
>
> earlier this week, Dave wrote:
>> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
>
> goresponsiveness installed flawlessly - very nice instructions and getting-started info.
>
> Comparison of NetQual (networkQuality -vs), UDPST -A c, gores/goresponsiveness
>
> Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-stream service
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
>
> NetQual UDPST (RFC 9097) gores
> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap RPM DelLD
> 882 788 m 76ms 8ms 967 28 0-16 127 (1382) 43ms
> 892 1036 h 58ms 8 966 27 0-18 128 (2124) 28ms
> 887 1304 h 46ms 6 969 27 0-18 130 (1478) 41ms
> 885 1008 h 60ms 8 967 28 0-22 127 (1490) 40ms
> 894 1383 h 43ms 11 967 28 0-15 133 (2731) 22ms
>
> NetQual UDPST (RFC 9097) gores
> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap RPM DelLD
> 21 327 m 183ms 8ms 22 (51) 28 0-253 12
> 21 413 m 145ms 8 22 (43) 28 0-255 15
> 22 273 m 220ms 6 23 (53) 28 0-259 10
> 21 377 m 159ms 8 23 (51) 28 0-250 10
> 22 281 m 214ms 11 23 (52) 28 0-250 6
>
> These tests were conducted in a round-robin fashion to minimize the possibility of network variations between measurements:
> NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
>
> NetQual indicates the same reduced capacity in Downstream when compared to UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer capacity, dep. on VLAN tag).
[SM] DOCSIS ISPs traditionally provision higher peak rates than they advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the 1050 Mbps range. But typically gigabit ethernet limits the practically achievable throughput somewhat:
Ethernet payload rate: 1000 * ((1500)/(1500 + 38)) = 975.29 Mbps
Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38 + 4)) = 972.76 Mbps
IPv4 payload (ethernet+VLAN): 1000 * ((1500 - 20)/(1500 + 38 + 4)) = 959.79 Mbps
IPv6 payload (ethernet+VLAN): 1000 * ((1500 - 40)/(1500 + 38 + 4)) = 946.82 Mbps
IPv4/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 20)/(1500 + 38 + 4)) = 946.82 Mbps
IPv6/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 40)/(1500 + 38 + 4)) = 933.85 Mbps
IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 20)/(1500 + 38 + 4)) = 939.04 Mbps
IPv6/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 40)/(1500 + 38 + 4)) = 926.07 Mbps
Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS, but on-line speedtest almost never return the simple average rate over the measurement duration, but play "clever" tricks to exclude the start-up phase and to aggregate over multiple flows that invariably ending with results that tend to exceed the hard limits shown above...
> Upstream capacities are not very different (a factor that made TCP methods more viable many years ago when most consumer access speeds were limited to 10's of Megabits).
[SM] My take on this is that this is partly due to the goal of ramping up very quickly and get away with a short measurement duration. That causes imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I often ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and for timecourses to look at and reason about).
> gores reports significantly lower capacity in both downstream and upstream measurements, a factor of 7 less than NetQual for downstream. Interestingly, the reduced capacity (taken as the working load) results in higher responsiveness: RPM meas are higher and loaded delays are lower for downstream.
[SM] Yepp, if you only fill-up the queue partially you will harvest less queueing delay and hence retain more responsiveness, albeit this really just seems to be better interpreted as go-responsiveness failing to achieve "working-conditions".
I tend to run gores like this:
time ./networkQuality --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --sattimeout 60 --extended-stats --logger-filename go_networkQuality_$(date +%Y%m%d_%H%M%S)
--sattimeout 60 extends the time out for the saturation measurement somewhat (before I saw it simply failing on a 1 Gbps access link, it did give some diagnostic message though).
>
> The comparison of networkQuality and goresponsiveness is somewhat confounded by the need to use the Apple server infrastructure for both these methods (the documentation provides this option - thanks!). I don't have admin access to our server at the moment. But the measured differences are large despite the confounding factor.
[SM] Puzzled, I thought when comparing the two ntworkQuality variants having the same back-end sort of helps be reducing the differences to the clients? But comparison to UDPST suffers somewhat.
>
> goresponsiveness has its own, very different output format than networkQuality. There isn't a comparable "-v" option other than -debug (which is extremely detailed). gores only reports RPM for the downstream.
[SM] I agree it would be nice if gores would grow a sequential mode as well.
> I hope that these results will prompt more of the principle evangelists and coders to weigh-in.
[SM] No luck ;) but at least "amateur hour" (aka me) is ready to discuss.
>
> It's also worth noting that the RTTrange reported by UDPST is the range above the minimum RTT, and represents the entire 10 second test. The consistency of the maximum of the range (~255ms) seems to indicate that UDPST has characterized the length of the upstream buffer during measurements.
[SM] thanks for explaining that.
Regards
Sebastian
>
> I'm sure there is more to observe that is prompted by these measurements; comments welcome!
>
> Al
>
>
>> -----Original Message-----
>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>> Sent: Tuesday, November 1, 2022 10:51 AM
>> To: Dave Taht <dave.taht@gmail.com>
>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>> metrics
>>
>> *** Security Advisory: This Message Originated Outside of AT&T ***.
>> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
>>
>> Hi Dave,
>> Thanks for trying UDPST (RFC 9097)!
>>
>> Something you might try with starlink:
>> use the -X option and UDPST will generate random payloads.
>>
>> The measurements with -X will reflect the uncompressed that are possible.
>> I tried this on a ship-board Internet access: uncompressed rate was 100kbps.
>>
>> A few more quick replies below,
>> Al
>>
>>> -----Original Message-----
>>> From: Dave Taht <dave.taht@gmail.com>
>>> Sent: Tuesday, November 1, 2022 12:22 AM
>>> To: MORTON JR., AL <acmorton@att.com>
>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
>>> metrics
>>>
>>> Dear Al:
>>>
>>> OK, I took your udpst tool for a spin.
>>>
>>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
>>> good feature set, VERY fast,
>> [acm]
>> Len Ciavattone (my partner in crime on several measurement projects) is the
>> lead coder: he has implemented many measurement tools extremely efficiently,
>> this one in C-lang.
>>
>>> and in very brief testing, seemed
>>> to be accurate in the starlink case, though it's hard to tell with
>>> them as the rate changes every 15s.
>> [acm]
>> Great! Our guiding principle developing UDPST has been to test the accuracy of
>> measurements against a ground-truth. It pays-off.
>>
>>>
>>> I filed a couple bug reports on trivial stuff:
>>>
>> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
>>>
>> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
>>> SswH_opYmEw$
>> [acm]
>> much appreciated... We have an OB-UDPST project meeting this Friday, can
>> discuss then.
>>
>>>
>>> (Adding diffserv and ecn washing or marking detection would be a nice
>>> feature to have)
>>>
>>> Aside from the sheer joy coming from the "it compiles! and runs!"
>>> phase I haven't looked much further.
>>>
>>> I left a copy running on one of my starlink testbeds -
>>> fremont.starlink.taht.net - if anyone wants to try it. It's
>>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
>>> version from bob, but close), and now udpst, and good to about a gbit.
>>>
>>> nice tool!
>> [acm]
>> Thanks again!
>>
>>>
>>> Has anyone here played with crusader? (
>>>
>> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
>>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$ )
>>>
>>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
>>>>
>>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
>>>>
>>>>>> have you tried irtt?
>>>
>> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
>>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$ )
>>>>> I have not. Seems like a reasonable tool for UDP testing. The feature I
>>> didn't like in my scan of the documentation is the use of Inter-packet delay
>>> variation (IPDV) instead of packet delay variation (PDV): variation from the
>>> minimum (or reference) delay. The morbidly curious can find my analysis in
>> RFC
>>> 5481:
>>>
>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
>>>
>> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
>>> T7QSlc$
>>>>
>>>> irtt was meant to simulate high speed voip and one day
>>>> videoconferencing. Please inspect the json output
>>>> for other metrics. Due to OS limits it is typically only accurate to a
>>>> 3ms interval. One thing it does admirably is begin to expose the
>>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
>>>> technologies, as well as request/grant systems like cable and gpon,
>>>> especially when otherwise idle.
>>>>
>>>> Here is a highres plot of starlink's behaviors from last year:
>>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
>>> bandwidth-
>>>
>> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
>>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
>>>>
>>>> clearly showing them "optimizing for bandwidth" and changing next sat
>>>> hop, and about a 40ms interval of buffering between these switches.
>>>> I'd published elsewhere, if anyone cares, a preliminary study of what
>>>> starlink's default behaviors did to cubic and BBR...
>>>>
>>>>>
>>>>> irtt's use of IPDV means that the results won’t compare with UDPST, and
>>> possibly networkQuality. But I may give it a try anyway...
>>>>
>>>> The more the merrier! Someday the "right" metrics will arrive.
>>>>
>>>> As a side note, this paper focuses on RAN uplink latency
>>>>
>>>
>> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
>>>
>> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHgvqerjg$ which I think
>>>> is a major barrier to most forms of 5G actually achieving good
>>>> performance in a FPS game, if it is true for more RANs. I'd like more
>>>> to be testing uplink latencies idle and with load, on all
>>>> technologies.
>>>>
>>>>>
>>>>> thanks again, Dave.
>>>>> Al
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Dave Taht <dave.taht@gmail.com>
>>>>>> Sent: Monday, October 31, 2022 12:52 PM
>>>>>> To: MORTON JR., AL <acmorton@att.com>
>>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
>>> Latency"
>>>>>> metrics
>>>>>>
>>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
>>> that.
>>>>>>
>>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
>> wrote:
>>>>>>>
>>>>>>> (astute readers may have guessed that I pressed "send" too soon on
>>> previous
>>>>>> message...)
>>>>>>>
>>>>>>> I also conducted upstream tests this time, here are the results:
>>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
>> and
>>>>>> Medium)
>>>>>>>
>>>>>>> Net Qual UDPST (RFC9097)
>> Ookla
>>>>>>> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
>> UpCap
>>>>>> Ping(no load)
>>>>>>> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
>>> 8
>>>>>>> 22 281 m 214ms 8ms 27 (52) 25 5-248 22
>>> 8
>>>>>>> 22 290 m 207ms 8ms 27 (55) 28 0-253 22
>>> 9
>>>>>>> 21 330 m 182ms 11ms 23 (44) 28 0-255 22
>>> 7
>>>>>>> 22 334 m 180ms 9ms 33 (56) 25 0-255 22
>>> 9
>>>>>>>
>>>>>>> The Upstream capacity measurements reflect an interesting feature
>> that
>>> we
>>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
>> of
>>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
>> this
>>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
>>> reporting
>>>>>> option that divides the complete measurement interval in two time
>>> intervals to
>>>>>> report an initial (turbo) max capacity and a steady-state max capacity
>>> for the
>>>>>> later intervals. The UDPST capacity results present both measurements:
>>> steady-
>>>>>> state first.
>>>>>>
>>>>>> Certainly we can expect bi-model distributions from many ISPs, as, for
>>>>>> one thing, the "speedboost" concept remains popular, except that it's
>>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
>>>>>> it is often configured "sneakily", in that it doesn't kick in for the
>>>>>> typical observed duration of the test, for some, they cut the
>>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
>>>>>>
>>>>>> One of my biggest issues with the rpm spec so far is that it should,
>>>>>> at least, sometimes, run randomly longer than the overly short
>>>>>> interval it runs for and the tools also allow for manual override of
>>> length.
>>>>>>
>>>>>> we caught a lot of tomfoolery with flent's rrul test running by
>> default
>>> for
>>>>>> 1m.
>>>>>>
>>>>>> Also, AQMs on the path can take a while to find the optimal drop or
>> mark
>>> rate.
>>>>>>
>>>>>>>
>>>>>>> The capacity processing in networkQuality and Ookla appear to report
>>> the
>>>>>> steady-state result.
>>>>>>
>>>>>> Ookla used to basically report the last result. Also it's not a good
>>>>>> indicator of web traffic behavior at all, watching the curve
>>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
>>>>>> (40ms)....
>>>>>>
>>>>>> So adding another mode - how quickly is peak bandwidth actually
>>>>>> reached, would be nice.
>>>>>>
>>>>>> I haven't poked into the current iteration of the goresponsiveness
>>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
>>>>>>
>>>
>> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
>>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
>>>>>> would be good to try collecting more statistics and histograms and
>>>>>> methods of analyzing the data in that libre-source version.
>>>>>>
>>>>>> How does networkQuality compare vs a vs your tool vs a vs
>>> goresponsiveness?
>>>>>>
>>>>>>> I watched the upstream capacity measurements on the Ookla app, and
>>> could
>>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
>>> most of
>>>>>> the test which determined the final result.
>>>>>>
>>>>>> I tend to get upset when I see ookla's new test flash a peak result in
>>>>>> the seconds and then settle on some lower number somehow.
>>>>>> So far as I know they are only sampling the latency every 250ms.
>>>>>>
>>>>>>>
>>>>>>> The working latency is about 200ms in networkQuality and about 280ms
>>> as
>>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
>> delay
>>> is
>>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
>>> difference
>>>>>> in working latency. Also, we used the very dynamic Type C load
>>>>>> adjustment/search algorithm in UDPST during all of this testing, which
>>> could
>>>>>> explain the higher working latency to some degree.
>>>>>>>
>>>>>>> So, it's worth noting that the measurements needed for assessing
>>> working
>>>>>> latency/responsiveness are available in the UDPST utility, and that
>> the
>>> UDPST
>>>>>> measurements are conducted on UDP transport (used by a growing
>> fraction
>>> of
>>>>>> Internet traffic).
>>>>>>
>>>>>> Thx, didn't know of this work til now!
>>>>>>
>>>>>> have you tried irtt?
>>>>>>
>>>>>>>
>>>>>>> comments welcome of course,
>>>>>>> Al
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
>>>>>>>> To: ippm@ietf.org
>>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
>>>>>> Latency"
>>>>>>>> metrics
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi again RPM friends and IPPM'ers,
>>>>>>>>
>>>>>>>> As promised, I repeated the tests shared last week, this time
>> using
>>> both
>>>>>> the
>>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
>>> networkQuality. I
>>>>>>>> followed Sebastian's calculations as well.
>>>>>>>>
>>>>>>>> Working Latency & Capacity Summary
>>>>>>>>
>>>>>>>> Net Qual UDPST
>>> Ookla
>>>>>>>> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
>>> DnCap
>>>>>>>> Ping(no load)
>>>>>>>> 885 916 m 66ms 8ms 970 28 0-20
>> 940
>>> 8
>>>>>>>> 888 1355 h 44ms 8ms 966 28 0-23
>> 940
>>> 8
>>>>>>>> 891 1109 h 54ms 8ms 968 27 0-19
>> 940
>>> 9
>>>>>>>> 887 1141 h 53ms 11ms 966 27 0-18
>> 937
>>> 7
>>>>>>>> 884 1151 h 52ms 9ms 968 28 0-20
>> 937
>>> 9
>>>>>>>>
>>>>>>>> With the sequential test option, I noticed that networkQuality
>>> achieved
>>>>>> nearly
>>>>>>>> the maximum capacity reported almost immediately at the start of a
>>> test.
>>>>>>>> However, the reported capacities are low by about 60Mbps,
>> especially
>>> when
>>>>>>>> compared to the Ookla TCP measurements.
>>>>>>>>
>>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
>> high
>>> end of
>>>>>> the
>>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
>>>>>> networkQuality
>>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
>>> be much
>>>>>>>> buffering in the downstream direction.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
>>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
>>>>>>>>> To: ippm@ietf.org
>>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
>>> Latency"
>>>>>>>>> metrics
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi RPM friends and IPPM'ers,
>>>>>>>>>
>>>>>>>>> I was wondering what a comparison of some of the "working
>> latency"
>>>>>> metrics
>>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
>>> 3.1, with
>>>>>> the
>>>>>>>>> downlink provisioned for 1Gbps.
>>>>>>>>>
>>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
>>> Ookla
>>>>>>>> Speedtest
>>>>>>>>> with as similar connectivity as possible (but we know that the
>>> traffic
>>>>>> will
>>>>>>>>> diverge to different servers and we can't change that aspect).
>>>>>>>>>
>>>>>>>>> Here's a quick summary of yesterday's results:
>>>>>>>>>
>>>>>>>>> Working Latency & Capacity Summary
>>>>>>>>>
>>>>>>>>> Net Qual UDPST Ookla
>>>>>>>>> DnCap RPM DnCap RTTmin RTTVarRnge DnCap
>>> Ping(no
>>>>>> load)
>>>>>>>>> 878 62 970 28 0-19 941 6
>>>>>>>>> 891 92 970 27 0-20 940 7
>>>>>>>>> 891 120 966 28 0-22 937 9
>>>>>>>>> 890 112 970 28 0-21 940 8
>>>>>>>>> 903 70 970 28 0-16 935 9
>>>>>>>>>
>>>>>>>>> Note: all RPM values were categorized as Low.
>>>>>>>>>
>>>>>>>>> networkQuality downstream capacities are always on the low side
>>> compared
>>>>>> to
>>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
>>> what
>>>>>> Ookla
>>>>>>>>> achieved. I think that a longer test duration might be needed to
>>> achieve
>>>>>> the
>>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
>>> observed
>>>>>> were
>>>>>>>>> certainly headed in the right direction. (I recently upgraded to
>>>>>> Monterey
>>>>>>>> 12.6
>>>>>>>>> on my MacBook, so should have the latest version.)
>>>>>>>>>
>>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
>>> should
>>>>>> have
>>>>>>>>> run the tests with the -v option to help with comparisons. I'll
>>> repeat
>>>>>> this
>>>>>>>>> test when I can make time.
>>>>>>>>>
>>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
>> the
>>> test)
>>>>>> and
>>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
>>> very
>>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
>>> this
>>>>>> access
>>>>>>>> and
>>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
>>> Chicago
>>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
>>> measurements
>>>>>> were
>>>>>>>> run
>>>>>>>>> on an Ubuntu VM in my MacBook.
>>>>>>>>>
>>>>>>>>> The big disappointment was that the Ookla desktop app I updated
>>> over the
>>>>>>>>> weekend did not include the new responsiveness metric! I
>> included
>>> the
>>>>>> ping
>>>>>>>>> results anyway, and it was clearly using a server in the nearby
>>> area.
>>>>>>>>>
>>>>>>>>> So, I have some more work to do, but I hope this is interesting-
>>> enough
>>>>>> to
>>>>>>>>> start some comparison discussions, and bring-out some
>> suggestions.
>>>>>>>>>
>>>>>>>>> happy testing all,
>>>>>>>>> Al
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ippm mailing list
>>>>>>>>> ippm@ietf.org
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ippm mailing list
>>>>>>>> ippm@ietf.org
>>>>>>>>
>>>>>>
>>>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>>>> T!g-
>>>>>>
>>> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ippm mailing list
>>>>>>> ippm@ietf.org
>>>>>>>
>>>>>>
>>>
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>>>>>>
>>>
>> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
>>>>>> KXU$
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> This song goes out to all the folk that thought Stadia would work:
>>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>>> mushroom-
>>>>>> song-activity-6981366665607352320-
>>>>>>
>>>
>> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
>>>>>> c48iQFub34zz4iE$
>>>>>> Dave Täht CEO, TekLibre, LLC
>>>>
>>>>
>>>>
>>>> --
>>>> This song goes out to all the folk that thought Stadia would work:
>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>>> mushroom-song-activity-6981366665607352320-
>>>
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHLHDpSWs$
>>>> Dave Täht CEO, TekLibre, LLC
>>>
>>>
>>>
>>> --
>>> This song goes out to all the folk that thought Stadia would work:
>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
>> mushroom-
>>> song-activity-6981366665607352320-
>>>
>> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
>>> IEUcSswHLHDpSWs$
>>> Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> ippm mailing list
>> ippm@ietf.org
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
>> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-rDt9HLI$
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-04 19:10 ` Sebastian Moeller
@ 2022-11-05 19:36 ` MORTON JR., AL
2022-12-11 19:21 ` MORTON JR., AL
0 siblings, 1 reply; 29+ messages in thread
From: MORTON JR., AL @ 2022-11-05 19:36 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Dave Täht, rjmcmahon, Rpm, ippm, Will Hawkins
Hi Sebastian, thanks for your comments and information.
I'll reply to a few key points in this top-post.
Sebastian wrote:
> [SM] DOCSIS ISPs traditionally provision higher peak rates than they
> advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> 1050 Mbps range. But typically gigabit ethernet limits the practically
> achievable throughput somewhat:
[acm]
Right, my ISP's CPE has Gig-E ports so I won't see any benefit from over-rate provisioning.
Sebastian wrote:
...
> IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 20)/(1500 + 38 + 4)) = 939.04 Mbps
...
> Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS,
> but on-line speedtest almost never return the simple average rate over the
> measurement duration, but play "clever" tricks to exclude the start-up phase
> and to aggregate over multiple flows that invariably ending with results that
> tend to exceed the hard limits shown above...
[acm]
My measurements with the Ookla desktop app on MacOS (shared earlier this week) are very consistent at ~940Mbps Dn-stream, so timestamps calculation sounds right.
My ISP specifies their service speed at 940Mbps, as though they assume subscribers will measure using Ookla or other TCP tool. In fact, our team hasn't seen such consistent results from Ookla or any TCP-based test in the 1Gbps range, and this makes me wonder if there might be some test-recognition here.
FYI - UDPST uses a max packet size with is sub-MTU, to avoid fragmentation when various encapsulations are encountered. Also, the definition of IP-Layer Capacity in the various standards (e.g., RFC 9097) includes the bits in the IP header in the total Capacity.
So, instead of:
> Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38 + 4)) = 972.76 Mbps
We have
Ethernet payload rate +VLAN: 1000 * ((1222)/(1222 + 38 + 4)) = 966.77 Mbps
and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps
UDPST has an option to use datagrams for the traditional 1500 octet MTU (-T), but a user could cause a lot of fragmentation on links with encapsulations wit this option.
acm wrote:
> > The comparison of networkQuality and goresponsiveness is somewhat confounded
> > by the need to use the Apple server infrastructure for both these methods ...
Sebastian wrote:
> [SM] Puzzled, I thought when comparing the two networkQuality variants
> having the same back-end sort of helps be reducing the differences to the
> clients? But comparison to UDPST suffers somewhat.
[acm]
Yes, I was hoping to match-up the client and server implementations. Instead, this might be more of a Client-X and Server-Y interoperability test (and could explain some results?), but that was not to be.
Thanks for your suggested command line for gores.
IIRC one of your early messages, Sebastian, your MacOS indicates that 12.6 is the latest. It's the same for me: my sufficiently powerful MacBook cannot upgrade to Ventura for the latest in networkQuality versions. It would help us who are doing some testing if the latest version of networkQuality could be made installable for us, somehow...
thanks again and regards,
Al
> -----Original Message-----
> From: Sebastian Moeller <moeller0@gmx.de>
> Sent: Friday, November 4, 2022 3:10 PM
> To: MORTON JR., AL <acmorton@att.com>
> Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <rjmcmahon@rjmcmahon.com>; Rpm
> <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <hawkinsw@obs.cr>
> Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> Latency" metrics
>
> Hi Al,
>
>
> > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm <rpm@lists.bufferbloat.net>
> wrote:
> >
> > Hi all,
> >
> > I have been working through the threads on misery metrics and lightweight
> sensing of bw and buffering, and with those insights and gold-nuggets in mind,
> I'm iterating through testing the combinations of tools that Dave That and Bob
> McMahon suggested.
> >
> > earlier this week, Dave wrote:
> >> How does networkQuality compare vs a vs your tool vs a vs goresponsiveness?
> >
> > goresponsiveness installed flawlessly - very nice instructions and getting-
> started info.
> >
> > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> gores/goresponsiveness
> >
> > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-
> stream service
> > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
> >
> > NetQual UDPST (RFC 9097) gores
> > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap RPM
> DelLD
> > 882 788 m 76ms 8ms 967 28 0-16 127
> (1382) 43ms
> > 892 1036 h 58ms 8 966 27 0-18 128
> (2124) 28ms
> > 887 1304 h 46ms 6 969 27 0-18 130
> (1478) 41ms
> > 885 1008 h 60ms 8 967 28 0-22 127
> (1490) 40ms
> > 894 1383 h 43ms 11 967 28 0-15 133
> (2731) 22ms
> >
> > NetQual UDPST (RFC 9097) gores
> > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap RPM
> DelLD
> > 21 327 m 183ms 8ms 22 (51) 28 0-253 12
> > 21 413 m 145ms 8 22 (43) 28 0-255 15
> > 22 273 m 220ms 6 23 (53) 28 0-259 10
> > 21 377 m 159ms 8 23 (51) 28 0-250 10
> > 22 281 m 214ms 11 23 (52) 28 0-250 6
> >
> > These tests were conducted in a round-robin fashion to minimize the
> possibility of network variations between measurements:
> > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
> >
> > NetQual indicates the same reduced capacity in Downstream when compared to
> UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-layer
> capacity, dep. on VLAN tag).
>
> [SM] DOCSIS ISPs traditionally provision higher peak rates than they
> advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> 1050 Mbps range. But typically gigabit ethernet limits the practically
> achievable throughput somewhat:
>
> Ethernet payload rate: 1000 * ((1500)/(1500 + 38)) =
> 975.29 Mbps
> Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38 + 4)) =
> 972.76 Mbps
> IPv4 payload (ethernet+VLAN): 1000 * ((1500 - 20)/(1500 + 38 + 4)) =
> 959.79 Mbps
> IPv6 payload (ethernet+VLAN): 1000 * ((1500 - 40)/(1500 + 38 + 4)) =
> 946.82 Mbps
> IPv4/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 20)/(1500 + 38 +
> 4)) = 946.82 Mbps
> IPv6/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 40)/(1500 + 38 +
> 4)) = 933.85 Mbps
> IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 20)/(1500 +
> 38 + 4)) = 939.04 Mbps
> IPv6/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 40)/(1500 +
> 38 + 4)) = 926.07 Mbps
>
>
> Speedtests tend to report IPv4/TCP timestamps might be on depending on the OS,
> but on-line speedtest almost never return the simple average rate over the
> measurement duration, but play "clever" tricks to exclude the start-up phase
> and to aggregate over multiple flows that invariably ending with results that
> tend to exceed the hard limits shown above...
>
>
> > Upstream capacities are not very different (a factor that made TCP methods
> more viable many years ago when most consumer access speeds were limited to
> 10's of Megabits).
>
> [SM] My take on this is that this is partly due to the goal of ramping
> up very quickly and get away with a short measurement duration. That causes
> imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I often
> ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and for
> timecourses to look at and reason about).
>
> > gores reports significantly lower capacity in both downstream and upstream
> measurements, a factor of 7 less than NetQual for downstream. Interestingly,
> the reduced capacity (taken as the working load) results in higher
> responsiveness: RPM meas are higher and loaded delays are lower for
> downstream.
>
> [SM] Yepp, if you only fill-up the queue partially you will harvest less
> queueing delay and hence retain more responsiveness, albeit this really just
> seems to be better interpreted as go-responsiveness failing to achieve
> "working-conditions".
>
> I tend to run gores like this:
> time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> /api/v1/gm/config --sattimeout 60 --extended-stats --logger-filename
> go_networkQuality_$(date +%Y%m%d_%H%M%S)
>
> --sattimeout 60 extends the time out for the saturation measurement somewhat
> (before I saw it simply failing on a 1 Gbps access link, it did give some
> diagnostic message though).
>
>
> >
> > The comparison of networkQuality and goresponsiveness is somewhat confounded
> by the need to use the Apple server infrastructure for both these methods (the
> documentation provides this option - thanks!). I don't have admin access to
> our server at the moment. But the measured differences are large despite the
> confounding factor.
>
> [SM] Puzzled, I thought when comparing the two ntworkQuality variants
> having the same back-end sort of helps be reducing the differences to the
> clients? But comparison to UDPST suffers somewhat.
>
> >
> > goresponsiveness has its own, very different output format than
> networkQuality. There isn't a comparable "-v" option other than -debug (which
> is extremely detailed). gores only reports RPM for the downstream.
>
> [SM] I agree it would be nice if gores would grow a sequential mode as
> well.
>
>
> > I hope that these results will prompt more of the principle evangelists and
> coders to weigh-in.
>
> [SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> discuss.
>
> >
> > It's also worth noting that the RTTrange reported by UDPST is the range
> above the minimum RTT, and represents the entire 10 second test. The
> consistency of the maximum of the range (~255ms) seems to indicate that UDPST
> has characterized the length of the upstream buffer during measurements.
>
> [SM] thanks for explaining that.
>
> Regards
> Sebastian
>
> >
> > I'm sure there is more to observe that is prompted by these measurements;
> comments welcome!
> >
> > Al
> >
> >
> >> -----Original Message-----
> >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >> Sent: Tuesday, November 1, 2022 10:51 AM
> >> To: Dave Taht <dave.taht@gmail.com>
> >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >> Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency"
> >> metrics
> >>
> >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> >>
> >> Hi Dave,
> >> Thanks for trying UDPST (RFC 9097)!
> >>
> >> Something you might try with starlink:
> >> use the -X option and UDPST will generate random payloads.
> >>
> >> The measurements with -X will reflect the uncompressed that are possible.
> >> I tried this on a ship-board Internet access: uncompressed rate was
> 100kbps.
> >>
> >> A few more quick replies below,
> >> Al
> >>
> >>> -----Original Message-----
> >>> From: Dave Taht <dave.taht@gmail.com>
> >>> Sent: Tuesday, November 1, 2022 12:22 AM
> >>> To: MORTON JR., AL <acmorton@att.com>
> >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> >>> metrics
> >>>
> >>> Dear Al:
> >>>
> >>> OK, I took your udpst tool for a spin.
> >>>
> >>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> >>> good feature set, VERY fast,
> >> [acm]
> >> Len Ciavattone (my partner in crime on several measurement projects) is the
> >> lead coder: he has implemented many measurement tools extremely
> efficiently,
> >> this one in C-lang.
> >>
> >>> and in very brief testing, seemed
> >>> to be accurate in the starlink case, though it's hard to tell with
> >>> them as the rate changes every 15s.
> >> [acm]
> >> Great! Our guiding principle developing UDPST has been to test the accuracy
> of
> >> measurements against a ground-truth. It pays-off.
> >>
> >>>
> >>> I filed a couple bug reports on trivial stuff:
> >>>
> >>
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> >>>
> >>
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> >>> SswH_opYmEw$
> >> [acm]
> >> much appreciated... We have an OB-UDPST project meeting this Friday, can
> >> discuss then.
> >>
> >>>
> >>> (Adding diffserv and ecn washing or marking detection would be a nice
> >>> feature to have)
> >>>
> >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> >>> phase I haven't looked much further.
> >>>
> >>> I left a copy running on one of my starlink testbeds -
> >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> >>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> >>> version from bob, but close), and now udpst, and good to about a gbit.
> >>>
> >>> nice tool!
> >> [acm]
> >> Thanks again!
> >>
> >>>
> >>> Has anyone here played with crusader? (
> >>>
> >>
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> >>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> )
> >>>
> >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> >>>>
> >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com> wrote:
> >>>>
> >>>>>> have you tried irtt?
> >>>
> >>
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> >>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> )
> >>>>> I have not. Seems like a reasonable tool for UDP testing. The feature I
> >>> didn't like in my scan of the documentation is the use of Inter-packet
> delay
> >>> variation (IPDV) instead of packet delay variation (PDV): variation from
> the
> >>> minimum (or reference) delay. The morbidly curious can find my analysis in
> >> RFC
> >>> 5481:
> >>>
> >>
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> >>>
> >>
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> >>> T7QSlc$
> >>>>
> >>>> irtt was meant to simulate high speed voip and one day
> >>>> videoconferencing. Please inspect the json output
> >>>> for other metrics. Due to OS limits it is typically only accurate to a
> >>>> 3ms interval. One thing it does admirably is begin to expose the
> >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> >>>> technologies, as well as request/grant systems like cable and gpon,
> >>>> especially when otherwise idle.
> >>>>
> >>>> Here is a highres plot of starlink's behaviors from last year:
> >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-adaptive-
> >>> bandwidth-
> >>>
> >>
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> >>>>
> >>>> clearly showing them "optimizing for bandwidth" and changing next sat
> >>>> hop, and about a 40ms interval of buffering between these switches.
> >>>> I'd published elsewhere, if anyone cares, a preliminary study of what
> >>>> starlink's default behaviors did to cubic and BBR...
> >>>>
> >>>>>
> >>>>> irtt's use of IPDV means that the results won’t compare with UDPST, and
> >>> possibly networkQuality. But I may give it a try anyway...
> >>>>
> >>>> The more the merrier! Someday the "right" metrics will arrive.
> >>>>
> >>>> As a side note, this paper focuses on RAN uplink latency
> >>>>
> >>>
> >>
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> >>>
> >>
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHgvqerjg$ which I think
> >>>> is a major barrier to most forms of 5G actually achieving good
> >>>> performance in a FPS game, if it is true for more RANs. I'd like more
> >>>> to be testing uplink latencies idle and with load, on all
> >>>> technologies.
> >>>>
> >>>>>
> >>>>> thanks again, Dave.
> >>>>> Al
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Dave Taht <dave.taht@gmail.com>
> >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> >>>>>> To: MORTON JR., AL <acmorton@att.com>
> >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> >>> Latency"
> >>>>>> metrics
> >>>>>>
> >>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
> >>> that.
> >>>>>>
> >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> >> wrote:
> >>>>>>>
> >>>>>>> (astute readers may have guessed that I pressed "send" too soon on
> >>> previous
> >>>>>> message...)
> >>>>>>>
> >>>>>>> I also conducted upstream tests this time, here are the results:
> >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
> >> and
> >>>>>> Medium)
> >>>>>>>
> >>>>>>> Net Qual UDPST (RFC9097)
> >> Ookla
> >>>>>>> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
> >> UpCap
> >>>>>> Ping(no load)
> >>>>>>> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
> >>> 8
> >>>>>>> 22 281 m 214ms 8ms 27 (52) 25 5-248 22
> >>> 8
> >>>>>>> 22 290 m 207ms 8ms 27 (55) 28 0-253 22
> >>> 9
> >>>>>>> 21 330 m 182ms 11ms 23 (44) 28 0-255 22
> >>> 7
> >>>>>>> 22 334 m 180ms 9ms 33 (56) 25 0-255 22
> >>> 9
> >>>>>>>
> >>>>>>> The Upstream capacity measurements reflect an interesting feature
> >> that
> >>> we
> >>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
> >> of
> >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> >> this
> >>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
> >>> reporting
> >>>>>> option that divides the complete measurement interval in two time
> >>> intervals to
> >>>>>> report an initial (turbo) max capacity and a steady-state max capacity
> >>> for the
> >>>>>> later intervals. The UDPST capacity results present both measurements:
> >>> steady-
> >>>>>> state first.
> >>>>>>
> >>>>>> Certainly we can expect bi-model distributions from many ISPs, as, for
> >>>>>> one thing, the "speedboost" concept remains popular, except that it's
> >>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> >>>>>> it is often configured "sneakily", in that it doesn't kick in for the
> >>>>>> typical observed duration of the test, for some, they cut the
> >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> >>>>>>
> >>>>>> One of my biggest issues with the rpm spec so far is that it should,
> >>>>>> at least, sometimes, run randomly longer than the overly short
> >>>>>> interval it runs for and the tools also allow for manual override of
> >>> length.
> >>>>>>
> >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> >> default
> >>> for
> >>>>>> 1m.
> >>>>>>
> >>>>>> Also, AQMs on the path can take a while to find the optimal drop or
> >> mark
> >>> rate.
> >>>>>>
> >>>>>>>
> >>>>>>> The capacity processing in networkQuality and Ookla appear to report
> >>> the
> >>>>>> steady-state result.
> >>>>>>
> >>>>>> Ookla used to basically report the last result. Also it's not a good
> >>>>>> indicator of web traffic behavior at all, watching the curve
> >>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> >>>>>> (40ms)....
> >>>>>>
> >>>>>> So adding another mode - how quickly is peak bandwidth actually
> >>>>>> reached, would be nice.
> >>>>>>
> >>>>>> I haven't poked into the current iteration of the goresponsiveness
> >>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
> >>>>>>
> >>>
> >>
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> >>>>>> would be good to try collecting more statistics and histograms and
> >>>>>> methods of analyzing the data in that libre-source version.
> >>>>>>
> >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> >>> goresponsiveness?
> >>>>>>
> >>>>>>> I watched the upstream capacity measurements on the Ookla app, and
> >>> could
> >>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for
> >>> most of
> >>>>>> the test which determined the final result.
> >>>>>>
> >>>>>> I tend to get upset when I see ookla's new test flash a peak result in
> >>>>>> the seconds and then settle on some lower number somehow.
> >>>>>> So far as I know they are only sampling the latency every 250ms.
> >>>>>>
> >>>>>>>
> >>>>>>> The working latency is about 200ms in networkQuality and about 280ms
> >>> as
> >>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
> >> delay
> >>> is
> >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> >>> difference
> >>>>>> in working latency. Also, we used the very dynamic Type C load
> >>>>>> adjustment/search algorithm in UDPST during all of this testing, which
> >>> could
> >>>>>> explain the higher working latency to some degree.
> >>>>>>>
> >>>>>>> So, it's worth noting that the measurements needed for assessing
> >>> working
> >>>>>> latency/responsiveness are available in the UDPST utility, and that
> >> the
> >>> UDPST
> >>>>>> measurements are conducted on UDP transport (used by a growing
> >> fraction
> >>> of
> >>>>>> Internet traffic).
> >>>>>>
> >>>>>> Thx, didn't know of this work til now!
> >>>>>>
> >>>>>> have you tried irtt?
> >>>>>>
> >>>>>>>
> >>>>>>> comments welcome of course,
> >>>>>>> Al
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> >>>>>>>> To: ippm@ietf.org
> >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> >>>>>> Latency"
> >>>>>>>> metrics
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi again RPM friends and IPPM'ers,
> >>>>>>>>
> >>>>>>>> As promised, I repeated the tests shared last week, this time
> >> using
> >>> both
> >>>>>> the
> >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> >>> networkQuality. I
> >>>>>>>> followed Sebastian's calculations as well.
> >>>>>>>>
> >>>>>>>> Working Latency & Capacity Summary
> >>>>>>>>
> >>>>>>>> Net Qual UDPST
> >>> Ookla
> >>>>>>>> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
> >>> DnCap
> >>>>>>>> Ping(no load)
> >>>>>>>> 885 916 m 66ms 8ms 970 28 0-20
> >> 940
> >>> 8
> >>>>>>>> 888 1355 h 44ms 8ms 966 28 0-23
> >> 940
> >>> 8
> >>>>>>>> 891 1109 h 54ms 8ms 968 27 0-19
> >> 940
> >>> 9
> >>>>>>>> 887 1141 h 53ms 11ms 966 27 0-18
> >> 937
> >>> 7
> >>>>>>>> 884 1151 h 52ms 9ms 968 28 0-20
> >> 937
> >>> 9
> >>>>>>>>
> >>>>>>>> With the sequential test option, I noticed that networkQuality
> >>> achieved
> >>>>>> nearly
> >>>>>>>> the maximum capacity reported almost immediately at the start of a
> >>> test.
> >>>>>>>> However, the reported capacities are low by about 60Mbps,
> >> especially
> >>> when
> >>>>>>>> compared to the Ookla TCP measurements.
> >>>>>>>>
> >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> >> high
> >>> end of
> >>>>>> the
> >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
> >>>>>> networkQuality
> >>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
> >>> be much
> >>>>>>>> buffering in the downstream direction.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> >>>>>>>>> To: ippm@ietf.org
> >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
> >>> Latency"
> >>>>>>>>> metrics
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi RPM friends and IPPM'ers,
> >>>>>>>>>
> >>>>>>>>> I was wondering what a comparison of some of the "working
> >> latency"
> >>>>>> metrics
> >>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
> >>> 3.1, with
> >>>>>> the
> >>>>>>>>> downlink provisioned for 1Gbps.
> >>>>>>>>>
> >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
> >>> Ookla
> >>>>>>>> Speedtest
> >>>>>>>>> with as similar connectivity as possible (but we know that the
> >>> traffic
> >>>>>> will
> >>>>>>>>> diverge to different servers and we can't change that aspect).
> >>>>>>>>>
> >>>>>>>>> Here's a quick summary of yesterday's results:
> >>>>>>>>>
> >>>>>>>>> Working Latency & Capacity Summary
> >>>>>>>>>
> >>>>>>>>> Net Qual UDPST Ookla
> >>>>>>>>> DnCap RPM DnCap RTTmin RTTVarRnge DnCap
> >>> Ping(no
> >>>>>> load)
> >>>>>>>>> 878 62 970 28 0-19 941 6
> >>>>>>>>> 891 92 970 27 0-20 940 7
> >>>>>>>>> 891 120 966 28 0-22 937 9
> >>>>>>>>> 890 112 970 28 0-21 940 8
> >>>>>>>>> 903 70 970 28 0-16 935 9
> >>>>>>>>>
> >>>>>>>>> Note: all RPM values were categorized as Low.
> >>>>>>>>>
> >>>>>>>>> networkQuality downstream capacities are always on the low side
> >>> compared
> >>>>>> to
> >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
> >>> what
> >>>>>> Ookla
> >>>>>>>>> achieved. I think that a longer test duration might be needed to
> >>> achieve
> >>>>>> the
> >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
> >>> observed
> >>>>>> were
> >>>>>>>>> certainly headed in the right direction. (I recently upgraded to
> >>>>>> Monterey
> >>>>>>>> 12.6
> >>>>>>>>> on my MacBook, so should have the latest version.)
> >>>>>>>>>
> >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
> >>> should
> >>>>>> have
> >>>>>>>>> run the tests with the -v option to help with comparisons. I'll
> >>> repeat
> >>>>>> this
> >>>>>>>>> test when I can make time.
> >>>>>>>>>
> >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> >> the
> >>> test)
> >>>>>> and
> >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
> >>> very
> >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
> >>> this
> >>>>>> access
> >>>>>>>> and
> >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
> >>> Chicago
> >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> >>> measurements
> >>>>>> were
> >>>>>>>> run
> >>>>>>>>> on an Ubuntu VM in my MacBook.
> >>>>>>>>>
> >>>>>>>>> The big disappointment was that the Ookla desktop app I updated
> >>> over the
> >>>>>>>>> weekend did not include the new responsiveness metric! I
> >> included
> >>> the
> >>>>>> ping
> >>>>>>>>> results anyway, and it was clearly using a server in the nearby
> >>> area.
> >>>>>>>>>
> >>>>>>>>> So, I have some more work to do, but I hope this is interesting-
> >>> enough
> >>>>>> to
> >>>>>>>>> start some comparison discussions, and bring-out some
> >> suggestions.
> >>>>>>>>>
> >>>>>>>>> happy testing all,
> >>>>>>>>> Al
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> ippm mailing list
> >>>>>>>>> ippm@ietf.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> >>
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> ippm mailing list
> >>>>>>>> ippm@ietf.org
> >>>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>>> T!g-
> >>>>>>
> >>> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> ippm mailing list
> >>>>>>> ippm@ietf.org
> >>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >>>>>>
> >>>
> >>
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> >>>>>> KXU$
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> This song goes out to all the folk that thought Stadia would work:
> >>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >>> mushroom-
> >>>>>> song-activity-6981366665607352320-
> >>>>>>
> >>>
> >>
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> >>>>>> c48iQFub34zz4iE$
> >>>>>> Dave Täht CEO, TekLibre, LLC
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> This song goes out to all the folk that thought Stadia would work:
> >>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >>> mushroom-song-activity-6981366665607352320-
> >>>
> >>
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHLHDpSWs$
> >>>> Dave Täht CEO, TekLibre, LLC
> >>>
> >>>
> >>>
> >>> --
> >>> This song goes out to all the folk that thought Stadia would work:
> >>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> >> mushroom-
> >>> song-activity-6981366665607352320-
> >>>
> >>
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> >>> IEUcSswHLHDpSWs$
> >>> Dave Täht CEO, TekLibre, LLC
> >> _______________________________________________
> >> ippm mailing list
> >> ippm@ietf.org
> >>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> rDt9HLI$
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> mERYCyuCd9e9PY3aNLkSw$
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-11-05 19:36 ` MORTON JR., AL
@ 2022-12-11 19:21 ` MORTON JR., AL
2022-12-12 2:38 ` Dave Taht
0 siblings, 1 reply; 29+ messages in thread
From: MORTON JR., AL @ 2022-12-11 19:21 UTC (permalink / raw)
To: MORTON JR., AL, Sebastian Moeller, Dave Taht, Randall Meyer
Cc: Rpm, Will Hawkins, ippm
Hi IPPM,
Prior to IETF-115, I shared a series of measurements with the IPPM list. We're looking at responsiveness and working latency with various metrics and multiple testing utilities. This message continues the discussion with new input.
When I first published some measurements, Dave Taht added his assessment and included other relevant email lists in the discussion. I'm continuing to cross-post to all the lists in this thread.
Dave originally suggested that I try a tool called irtt; I've done that now and these are the results.
Bob McMahon: I queued your request to try your iperf2 tool behind the irtt measurements. I hope to make some more measurements this week...
-=-=-=-=-=-=-=-=-=-=-=-=-=-
We're testing a DOCIS 3.1 based service with 1Gbps down, nominally 22Mbps up. I used wired ETH connected to the DOCSIS modem's switch.
Dave Taht made his server available for the irtt measurements. I installed irtt on a VM in my Macbook, the same VM that runs UDPST. I ran quite a few tests to become familiar with irtt, so I'll just summarize the relevant ones here.
I ran irtt with its traffic at the maximum allowed on Dave's server. The test duration was 10sec, with packets spaced at 1ms intervals from my VM client. This is the complete output:
./irtt client -i 1ms -l 1250 -d 10s fremont.starlink.taht.net
Min Mean Median Max Stddev
--- ---- ------ --- ------
RTT 46.63ms 51.58ms 51.4ms 58ms 1.55ms
send delay 20.74ms 25.57ms 25.4ms 32.04ms 1.54ms
receive delay 25.8ms 26.01ms 25.96ms 30.48ms 219µs
IPDV (jitter) 1.03µs 1.15ms 1.02ms 6.87ms 793µs
send IPDV 176ns 1.13ms 994µs 6.41ms 776µs
receive IPDV 7ns 79.8µs 41.7µs 4.54ms 140µs
send call time 10.1µs 55.8µs 1.34ms 30.3µs
timer error 68ns 431µs 6.49ms 490µs
server proc. time 680ns 2.43µs 47.7µs 1.73µs
duration: 10.2s (wait 174ms)
packets sent/received: 7137/7131 (0.08% loss)
server packets received: 7131/7137 (0.08%/0.00% loss up/down)
bytes sent/received: 8921250/8913750
send/receive rate: 7.14 Mbps / 7.13 Mbps
packet length: 1250 bytes
timer stats: 2863/10000 (28.63%) missed, 43.10% error
acm@acm-ubuntu1804-1:~/goirtt/irtt$
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
irtt supplies lots of info/stats about the RTT distribution. In the lightly loaded (7.1Mbps) scenario above, the RTT range is about 12 ms. The Mean and the Median are approximately the same.
irtt also supplies Inter-Packet Delay Varition (IPDV), showing that the packets seem to be delayed more than the sending interval occasionally (Max of 6.8ms).
irtt measurements indicate very low packet loss: no congestion at 7Mbps.
For the opposite end of the congestion spectrum, I ran irtt with UDPST (RFC 9097) running in parallel (and using the Type C search algorithm). We pick-up a lot more RTT and wider RTT range in this scenario:
irtt with udpst using Type C search = max load:
Min Mean Median Max Stddev
--- ---- ------ --- ------
RTT 47.58ms 118ms 56.53ms 301.6ms 90.28ms
send delay 24.05ms 94.85ms 33.38ms 278.5ms 90.26ms
receive delay 22.99ms 23.17ms 23.13ms 25.42ms 156µs
IPDV (jitter) 162ns 1.04ms 733µs 6.36ms 1.02ms
send IPDV 3.81µs 1.01ms 697µs 6.24ms 1.02ms
receive IPDV 88ns 93µs 49.8µs 1.48ms 145µs
send call time 4.28µs 39.3µs 903µs 32.4µs
timer error 86ns 287µs 6.13ms 214µs
server proc. time 670ns 3.59µs 19.3µs 2.26µs
duration: 10.9s (wait 904.8ms)
packets sent/received: 8305/2408 (71.01% loss)
server packets received: 2408/8305 (71.01%/0.00% loss up/down)
bytes sent/received: 10381250/3010000
send/receive rate: 8.31 Mbps / 2.47 Mbps
packet length: 1250 bytes
timer stats: 1695/10000 (16.95%) missed, 28.75% error
acm@acm-ubuntu1804-1:~/goirtt/irtt$
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
The irtt measurement of RTT range is now about 250ms in this maximally-loaded scenario.
One key RTT difference is the Mean delay, which is now twice the Median with working load added by UDPST.
Congestion is evident in the irtt loss measurements = 71%. However, the competing traffic did not break irtt's process or measurements.
UDPST's measurements were ~22Mbps Capacity (steady-state), with high loss, and similar RTT range of 251 ms.
Additional tests included load from UDPST at fixed rates: 14Mbps and 20Mbps. We compare the RTT range for all four conditions in the table below:
Load with irtt irtt RTT range UDPST RTT range
==================================================
irtt alone 12ms ---
UDPST at 14Mbps 11ms 22ms
UDPST at 20Mbps 14ms 46ms
UDPST at MaxCap 250ms 251ms
The unexpected result with irtt measurements is that the RTT range did not increase with load, where the UDPST RTT range increases with load. We're assuming that the majority of delay increases occur in the DOCSIS upstream queue and both test streams should see similar delay range as they do with maximum load. Perhaps there are some differences owing to the periodic irtt stream and the bursty UDPST stream (both are UDP), but this is speculation.
To check that the test path was operating similarly with earlier tests, we ran a couple of NetworkQuality and UDPST tests as before:
Comparison of NQ-vs, UDPST -A c, same columns as defined earlier.
Working Latency & Capacity Summary (Upstream only)
(capacity in Mbps, delays in ms, h and m are RPM categories, High and Medium)
Net Qual UDPST (RFC 9097)
UpCap RPM DelLD DelMin UpCap(stable) RTTmin RTTrange
22 276 m 217ms 11ms 23 28 0-252
22 291 m 206ms 11ms ~22* 28* 0-251*
* UDPST test result with the ~7Mbps irtt stream present.
We found that uplink test results are similar to previous tests, and the new irtt results were collected under similar conditions.
In conclusion, irtt provides a clear summary of the RTT distribution. Minimum, Mean, Median and Max RTT are useful individually and in combinations/comparisons. The irtt measurements compare favorably to those collected during IP-Layer Capacity tests with the UDPST utility.
comments welcome,
Al
> -----Original Message-----
> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> Sent: Saturday, November 5, 2022 3:37 PM
> To: Sebastian Moeller <moeller0@gmx.de>
> Cc: Rpm <rpm@lists.bufferbloat.net>; Will Hawkins <hawkinsw@obs.cr>;
> ippm@ietf.org
> Subject: Re: [ippm] [Rpm] Preliminary measurement comparison of "Working
> Latency" metrics
>
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
>
> Hi Sebastian, thanks for your comments and information.
> I'll reply to a few key points in this top-post.
>
> Sebastian wrote:
> > [SM] DOCSIS ISPs traditionally provision higher peak rates than they
> > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > achievable throughput somewhat:
> [acm]
> Right, my ISP's CPE has Gig-E ports so I won't see any benefit from over-rate
> provisioning.
>
> Sebastian wrote:
> ...
> > IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 20)/(1500
> + 38 + 4)) = 939.04 Mbps
> ...
> > Speedtests tend to report IPv4/TCP timestamps might be on depending on the
> OS,
> > but on-line speedtest almost never return the simple average rate over the
> > measurement duration, but play "clever" tricks to exclude the start-up phase
> > and to aggregate over multiple flows that invariably ending with results
> that
> > tend to exceed the hard limits shown above...
> [acm]
> My measurements with the Ookla desktop app on MacOS (shared earlier this week)
> are very consistent at ~940Mbps Dn-stream, so timestamps calculation sounds
> right.
> My ISP specifies their service speed at 940Mbps, as though they assume
> subscribers will measure using Ookla or other TCP tool. In fact, our team
> hasn't seen such consistent results from Ookla or any TCP-based test in the
> 1Gbps range, and this makes me wonder if there might be some test-recognition
> here.
>
> FYI - UDPST uses a max packet size with is sub-MTU, to avoid fragmentation
> when various encapsulations are encountered. Also, the definition of IP-Layer
> Capacity in the various standards (e.g., RFC 9097) includes the bits in the IP
> header in the total Capacity.
>
> So, instead of:
> > Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38 + 4)) =
> 972.76 Mbps
> We have
> Ethernet payload rate +VLAN: 1000 * ((1222)/(1222 + 38 + 4)) = 966.77
> Mbps
> and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps
>
> UDPST has an option to use datagrams for the traditional 1500 octet MTU (-T),
> but a user could cause a lot of fragmentation on links with encapsulations wit
> this option.
>
> acm wrote:
> > > The comparison of networkQuality and goresponsiveness is somewhat
> confounded
> > > by the need to use the Apple server infrastructure for both these methods
> ...
> Sebastian wrote:
> > [SM] Puzzled, I thought when comparing the two networkQuality variants
> > having the same back-end sort of helps be reducing the differences to the
> > clients? But comparison to UDPST suffers somewhat.
> [acm]
> Yes, I was hoping to match-up the client and server implementations. Instead,
> this might be more of a Client-X and Server-Y interoperability test (and could
> explain some results?), but that was not to be.
>
> Thanks for your suggested command line for gores.
>
> IIRC one of your early messages, Sebastian, your MacOS indicates that 12.6 is
> the latest. It's the same for me: my sufficiently powerful MacBook cannot
> upgrade to Ventura for the latest in networkQuality versions. It would help us
> who are doing some testing if the latest version of networkQuality could be
> made installable for us, somehow...
>
> thanks again and regards,
> Al
>
>
> > -----Original Message-----
> > From: Sebastian Moeller <moeller0@gmx.de>
> > Sent: Friday, November 4, 2022 3:10 PM
> > To: MORTON JR., AL <acmorton@att.com>
> > Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <rjmcmahon@rjmcmahon.com>;
> Rpm
> > <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <hawkinsw@obs.cr>
> > Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working
> > Latency" metrics
> >
> > Hi Al,
> >
> >
> > > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm
> <rpm@lists.bufferbloat.net>
> > wrote:
> > >
> > > Hi all,
> > >
> > > I have been working through the threads on misery metrics and lightweight
> > sensing of bw and buffering, and with those insights and gold-nuggets in
> mind,
> > I'm iterating through testing the combinations of tools that Dave That and
> Bob
> > McMahon suggested.
> > >
> > > earlier this week, Dave wrote:
> > >> How does networkQuality compare vs a vs your tool vs a vs
> goresponsiveness?
> > >
> > > goresponsiveness installed flawlessly - very nice instructions and
> getting-
> > started info.
> > >
> > > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> > gores/goresponsiveness
> > >
> > > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps down-
> > stream service
> > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > Medium)
> > >
> > > NetQual UDPST (RFC 9097) gores
> > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> RPM
> > DelLD
> > > 882 788 m 76ms 8ms 967 28 0-16 127
> > (1382) 43ms
> > > 892 1036 h 58ms 8 966 27 0-18 128
> > (2124) 28ms
> > > 887 1304 h 46ms 6 969 27 0-18 130
> > (1478) 41ms
> > > 885 1008 h 60ms 8 967 28 0-22 127
> > (1490) 40ms
> > > 894 1383 h 43ms 11 967 28 0-15 133
> > (2731) 22ms
> > >
> > > NetQual UDPST (RFC 9097) gores
> > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> RPM
> > DelLD
> > > 21 327 m 183ms 8ms 22 (51) 28 0-253 12
> > > 21 413 m 145ms 8 22 (43) 28 0-255 15
> > > 22 273 m 220ms 6 23 (53) 28 0-259 10
> > > 21 377 m 159ms 8 23 (51) 28 0-250 10
> > > 22 281 m 214ms 11 23 (52) 28 0-250 6
> > >
> > > These tests were conducted in a round-robin fashion to minimize the
> > possibility of network variations between measurements:
> > > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest - repeat
> > >
> > > NetQual indicates the same reduced capacity in Downstream when compared to
> > UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for IP-
> layer
> > capacity, dep. on VLAN tag).
> >
> > [SM] DOCSIS ISPs traditionally provision higher peak rates than they
> > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even via
> > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput in the
> > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > achievable throughput somewhat:
> >
> > Ethernet payload rate: 1000 * ((1500)/(1500 + 38)) =
> > 975.29 Mbps
> > Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38 + 4)) =
> > 972.76 Mbps
> > IPv4 payload (ethernet+VLAN): 1000 * ((1500 - 20)/(1500 + 38 + 4))
> =
> > 959.79 Mbps
> > IPv6 payload (ethernet+VLAN): 1000 * ((1500 - 40)/(1500 + 38 + 4))
> =
> > 946.82 Mbps
> > IPv4/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 20)/(1500 + 38
> +
> > 4)) = 946.82 Mbps
> > IPv6/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 40)/(1500 + 38
> +
> > 4)) = 933.85 Mbps
> > IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 20)/(1500
> +
> > 38 + 4)) = 939.04 Mbps
> > IPv6/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 - 40)/(1500
> +
> > 38 + 4)) = 926.07 Mbps
> >
> >
> > Speedtests tend to report IPv4/TCP timestamps might be on depending on the
> OS,
> > but on-line speedtest almost never return the simple average rate over the
> > measurement duration, but play "clever" tricks to exclude the start-up phase
> > and to aggregate over multiple flows that invariably ending with results
> that
> > tend to exceed the hard limits shown above...
> >
> >
> > > Upstream capacities are not very different (a factor that made TCP methods
> > more viable many years ago when most consumer access speeds were limited to
> > 10's of Megabits).
> >
> > [SM] My take on this is that this is partly due to the goal of ramping
> > up very quickly and get away with a short measurement duration. That causes
> > imprecision. As Dave said flent's RRUL test defaults to 60 seconds and I
> often
> > ran/run it for 5 to 10 minutes to get somewhat more reliable numbers (and
> for
> > timecourses to look at and reason about).
> >
> > > gores reports significantly lower capacity in both downstream and upstream
> > measurements, a factor of 7 less than NetQual for downstream. Interestingly,
> > the reduced capacity (taken as the working load) results in higher
> > responsiveness: RPM meas are higher and loaded delays are lower for
> > downstream.
> >
> > [SM] Yepp, if you only fill-up the queue partially you will harvest less
> > queueing delay and hence retain more responsiveness, albeit this really just
> > seems to be better interpreted as go-responsiveness failing to achieve
> > "working-conditions".
> >
> > I tend to run gores like this:
> > time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> > /api/v1/gm/config --sattimeout 60 --extended-stats --logger-filename
> > go_networkQuality_$(date +%Y%m%d_%H%M%S)
> >
> > --sattimeout 60 extends the time out for the saturation measurement somewhat
> > (before I saw it simply failing on a 1 Gbps access link, it did give some
> > diagnostic message though).
> >
> >
> > >
> > > The comparison of networkQuality and goresponsiveness is somewhat
> confounded
> > by the need to use the Apple server infrastructure for both these methods
> (the
> > documentation provides this option - thanks!). I don't have admin access to
> > our server at the moment. But the measured differences are large despite the
> > confounding factor.
> >
> > [SM] Puzzled, I thought when comparing the two ntworkQuality variants
> > having the same back-end sort of helps be reducing the differences to the
> > clients? But comparison to UDPST suffers somewhat.
> >
> > >
> > > goresponsiveness has its own, very different output format than
> > networkQuality. There isn't a comparable "-v" option other than -debug
> (which
> > is extremely detailed). gores only reports RPM for the downstream.
> >
> > [SM] I agree it would be nice if gores would grow a sequential mode as
> > well.
> >
> >
> > > I hope that these results will prompt more of the principle evangelists
> and
> > coders to weigh-in.
> >
> > [SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> > discuss.
> >
> > >
> > > It's also worth noting that the RTTrange reported by UDPST is the range
> > above the minimum RTT, and represents the entire 10 second test. The
> > consistency of the maximum of the range (~255ms) seems to indicate that
> UDPST
> > has characterized the length of the upstream buffer during measurements.
> >
> > [SM] thanks for explaining that.
> >
> > Regards
> > Sebastian
> >
> > >
> > > I'm sure there is more to observe that is prompted by these measurements;
> > comments welcome!
> > >
> > > Al
> > >
> > >
> > >> -----Original Message-----
> > >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >> Sent: Tuesday, November 1, 2022 10:51 AM
> > >> To: Dave Taht <dave.taht@gmail.com>
> > >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> Latency"
> > >> metrics
> > >>
> > >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> > >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> > >>
> > >> Hi Dave,
> > >> Thanks for trying UDPST (RFC 9097)!
> > >>
> > >> Something you might try with starlink:
> > >> use the -X option and UDPST will generate random payloads.
> > >>
> > >> The measurements with -X will reflect the uncompressed that are possible.
> > >> I tried this on a ship-board Internet access: uncompressed rate was
> > 100kbps.
> > >>
> > >> A few more quick replies below,
> > >> Al
> > >>
> > >>> -----Original Message-----
> > >>> From: Dave Taht <dave.taht@gmail.com>
> > >>> Sent: Tuesday, November 1, 2022 12:22 AM
> > >>> To: MORTON JR., AL <acmorton@att.com>
> > >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > >>> metrics
> > >>>
> > >>> Dear Al:
> > >>>
> > >>> OK, I took your udpst tool for a spin.
> > >>>
> > >>> NICE! 120k binary (I STILL work on machines with only 4MB of flash),
> > >>> good feature set, VERY fast,
> > >> [acm]
> > >> Len Ciavattone (my partner in crime on several measurement projects) is
> the
> > >> lead coder: he has implemented many measurement tools extremely
> > efficiently,
> > >> this one in C-lang.
> > >>
> > >>> and in very brief testing, seemed
> > >>> to be accurate in the starlink case, though it's hard to tell with
> > >>> them as the rate changes every 15s.
> > >> [acm]
> > >> Great! Our guiding principle developing UDPST has been to test the
> accuracy
> > of
> > >> measurements against a ground-truth. It pays-off.
> > >>
> > >>>
> > >>> I filed a couple bug reports on trivial stuff:
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> > >>>
> > >>
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > >>> SswH_opYmEw$
> > >> [acm]
> > >> much appreciated... We have an OB-UDPST project meeting this Friday, can
> > >> discuss then.
> > >>
> > >>>
> > >>> (Adding diffserv and ecn washing or marking detection would be a nice
> > >>> feature to have)
> > >>>
> > >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> > >>> phase I haven't looked much further.
> > >>>
> > >>> I left a copy running on one of my starlink testbeds -
> > >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> > >>> instrumented with netperf, flent, irtt, iperf2 (not quite the latest
> > >>> version from bob, but close), and now udpst, and good to about a gbit.
> > >>>
> > >>> nice tool!
> > >> [acm]
> > >> Thanks again!
> > >>
> > >>>
> > >>> Has anyone here played with crusader? (
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > >>> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> > )
> > >>>
> > >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com> wrote:
> > >>>>
> > >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com>
> wrote:
> > >>>>
> > >>>>>> have you tried irtt?
> > >>>
> > >>
> >
> (https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > >>> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> > )
> > >>>>> I have not. Seems like a reasonable tool for UDP testing. The feature
> I
> > >>> didn't like in my scan of the documentation is the use of Inter-packet
> > delay
> > >>> variation (IPDV) instead of packet delay variation (PDV): variation from
> > the
> > >>> minimum (or reference) delay. The morbidly curious can find my analysis
> in
> > >> RFC
> > >>> 5481:
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;!!
> > >>>
> > >>
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > >>> T7QSlc$
> > >>>>
> > >>>> irtt was meant to simulate high speed voip and one day
> > >>>> videoconferencing. Please inspect the json output
> > >>>> for other metrics. Due to OS limits it is typically only accurate to a
> > >>>> 3ms interval. One thing it does admirably is begin to expose the
> > >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > >>>> technologies, as well as request/grant systems like cable and gpon,
> > >>>> especially when otherwise idle.
> > >>>>
> > >>>> Here is a highres plot of starlink's behaviors from last year:
> > >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-
> adaptive-
> > >>> bandwidth-
> > >>>
> > >>
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > >>>>
> > >>>> clearly showing them "optimizing for bandwidth" and changing next sat
> > >>>> hop, and about a 40ms interval of buffering between these switches.
> > >>>> I'd published elsewhere, if anyone cares, a preliminary study of what
> > >>>> starlink's default behaviors did to cubic and BBR...
> > >>>>
> > >>>>>
> > >>>>> irtt's use of IPDV means that the results won’t compare with UDPST,
> and
> > >>> possibly networkQuality. But I may give it a try anyway...
> > >>>>
> > >>>> The more the merrier! Someday the "right" metrics will arrive.
> > >>>>
> > >>>> As a side note, this paper focuses on RAN uplink latency
> > >>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> > >>>
> > >>
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHgvqerjg$ which I think
> > >>>> is a major barrier to most forms of 5G actually achieving good
> > >>>> performance in a FPS game, if it is true for more RANs. I'd like more
> > >>>> to be testing uplink latencies idle and with load, on all
> > >>>> technologies.
> > >>>>
> > >>>>>
> > >>>>> thanks again, Dave.
> > >>>>> Al
> > >>>>>
> > >>>>>> -----Original Message-----
> > >>>>>> From: Dave Taht <dave.taht@gmail.com>
> > >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> > >>>>>> To: MORTON JR., AL <acmorton@att.com>
> > >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > >>> Latency"
> > >>>>>> metrics
> > >>>>>>
> > >>>>>> Thank you very much for the steer to RFC9097. I'd completely missed
> > >>> that.
> > >>>>>>
> > >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <acmorton@att.com>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> (astute readers may have guessed that I pressed "send" too soon on
> > >>> previous
> > >>>>>> message...)
> > >>>>>>>
> > >>>>>>> I also conducted upstream tests this time, here are the results:
> > >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories, High
> > >> and
> > >>>>>> Medium)
> > >>>>>>>
> > >>>>>>> Net Qual UDPST (RFC9097)
> > >> Ookla
> > >>>>>>> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
> > >> UpCap
> > >>>>>> Ping(no load)
> > >>>>>>> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22
> > >>> 8
> > >>>>>>> 22 281 m 214ms 8ms 27 (52) 25 5-248 22
> > >>> 8
> > >>>>>>> 22 290 m 207ms 8ms 27 (55) 28 0-253 22
> > >>> 9
> > >>>>>>> 21 330 m 182ms 11ms 23 (44) 28 0-255 22
> > >>> 7
> > >>>>>>> 22 334 m 180ms 9ms 33 (56) 25 0-255 22
> > >>> 9
> > >>>>>>>
> > >>>>>>> The Upstream capacity measurements reflect an interesting feature
> > >> that
> > >>> we
> > >>>>>> can reliably and repeatably measure with UDPST. The first ~3 seconds
> > >> of
> > >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST displays
> > >> this
> > >>>>>> behavior in its 1 second sub-interval measurements and has a bimodal
> > >>> reporting
> > >>>>>> option that divides the complete measurement interval in two time
> > >>> intervals to
> > >>>>>> report an initial (turbo) max capacity and a steady-state max
> capacity
> > >>> for the
> > >>>>>> later intervals. The UDPST capacity results present both
> measurements:
> > >>> steady-
> > >>>>>> state first.
> > >>>>>>
> > >>>>>> Certainly we can expect bi-model distributions from many ISPs, as,
> for
> > >>>>>> one thing, the "speedboost" concept remains popular, except that it's
> > >>>>>> misnamed, as it should be called speed-subtract or speed-lose. Worse,
> > >>>>>> it is often configured "sneakily", in that it doesn't kick in for the
> > >>>>>> typical observed duration of the test, for some, they cut the
> > >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> > >>>>>>
> > >>>>>> One of my biggest issues with the rpm spec so far is that it should,
> > >>>>>> at least, sometimes, run randomly longer than the overly short
> > >>>>>> interval it runs for and the tools also allow for manual override of
> > >>> length.
> > >>>>>>
> > >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> > >> default
> > >>> for
> > >>>>>> 1m.
> > >>>>>>
> > >>>>>> Also, AQMs on the path can take a while to find the optimal drop or
> > >> mark
> > >>> rate.
> > >>>>>>
> > >>>>>>>
> > >>>>>>> The capacity processing in networkQuality and Ookla appear to report
> > >>> the
> > >>>>>> steady-state result.
> > >>>>>>
> > >>>>>> Ookla used to basically report the last result. Also it's not a good
> > >>>>>> indicator of web traffic behavior at all, watching the curve
> > >>>>>> go up much more slowly in their test on say, fiber 2ms, vs starlink,
> > >>>>>> (40ms)....
> > >>>>>>
> > >>>>>> So adding another mode - how quickly is peak bandwidth actually
> > >>>>>> reached, would be nice.
> > >>>>>>
> > >>>>>> I haven't poked into the current iteration of the goresponsiveness
> > >>>>>> test at all: https://urldefense.com/v3/__https://github.com/network-
> > >>>>>>
> > >>>
> > >>
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > >>>>>> would be good to try collecting more statistics and histograms and
> > >>>>>> methods of analyzing the data in that libre-source version.
> > >>>>>>
> > >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> > >>> goresponsiveness?
> > >>>>>>
> > >>>>>>> I watched the upstream capacity measurements on the Ookla app, and
> > >>> could
> > >>>>>> easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps
> for
> > >>> most of
> > >>>>>> the test which determined the final result.
> > >>>>>>
> > >>>>>> I tend to get upset when I see ookla's new test flash a peak result
> in
> > >>>>>> the seconds and then settle on some lower number somehow.
> > >>>>>> So far as I know they are only sampling the latency every 250ms.
> > >>>>>>
> > >>>>>>>
> > >>>>>>> The working latency is about 200ms in networkQuality and about 280ms
> > >>> as
> > >>>>>> measured by UDPST (RFC9097). Note that the networkQuality minimum
> > >> delay
> > >>> is
> > >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of the
> > >>> difference
> > >>>>>> in working latency. Also, we used the very dynamic Type C load
> > >>>>>> adjustment/search algorithm in UDPST during all of this testing,
> which
> > >>> could
> > >>>>>> explain the higher working latency to some degree.
> > >>>>>>>
> > >>>>>>> So, it's worth noting that the measurements needed for assessing
> > >>> working
> > >>>>>> latency/responsiveness are available in the UDPST utility, and that
> > >> the
> > >>> UDPST
> > >>>>>> measurements are conducted on UDP transport (used by a growing
> > >> fraction
> > >>> of
> > >>>>>> Internet traffic).
> > >>>>>>
> > >>>>>> Thx, didn't know of this work til now!
> > >>>>>>
> > >>>>>> have you tried irtt?
> > >>>>>>
> > >>>>>>>
> > >>>>>>> comments welcome of course,
> > >>>>>>> Al
> > >>>>>>>
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> > >>>>>>>> To: ippm@ietf.org
> > >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > >>>>>> Latency"
> > >>>>>>>> metrics
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Hi again RPM friends and IPPM'ers,
> > >>>>>>>>
> > >>>>>>>> As promised, I repeated the tests shared last week, this time
> > >> using
> > >>> both
> > >>>>>> the
> > >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> > >>> networkQuality. I
> > >>>>>>>> followed Sebastian's calculations as well.
> > >>>>>>>>
> > >>>>>>>> Working Latency & Capacity Summary
> > >>>>>>>>
> > >>>>>>>> Net Qual UDPST
> > >>> Ookla
> > >>>>>>>> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
> > >>> DnCap
> > >>>>>>>> Ping(no load)
> > >>>>>>>> 885 916 m 66ms 8ms 970 28 0-20
> > >> 940
> > >>> 8
> > >>>>>>>> 888 1355 h 44ms 8ms 966 28 0-23
> > >> 940
> > >>> 8
> > >>>>>>>> 891 1109 h 54ms 8ms 968 27 0-19
> > >> 940
> > >>> 9
> > >>>>>>>> 887 1141 h 53ms 11ms 966 27 0-18
> > >> 937
> > >>> 7
> > >>>>>>>> 884 1151 h 52ms 9ms 968 28 0-20
> > >> 937
> > >>> 9
> > >>>>>>>>
> > >>>>>>>> With the sequential test option, I noticed that networkQuality
> > >>> achieved
> > >>>>>> nearly
> > >>>>>>>> the maximum capacity reported almost immediately at the start of a
> > >>> test.
> > >>>>>>>> However, the reported capacities are low by about 60Mbps,
> > >> especially
> > >>> when
> > >>>>>>>> compared to the Ookla TCP measurements.
> > >>>>>>>>
> > >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> > >> high
> > >>> end of
> > >>>>>> the
> > >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of the
> > >>>>>> networkQuality
> > >>>>>>>> RPM measurements were categorized as "High". There doesn't seem to
> > >>> be much
> > >>>>>>>> buffering in the downstream direction.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> -----Original Message-----
> > >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> > >>>>>>>>> To: ippm@ietf.org
> > >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of "Working
> > >>> Latency"
> > >>>>>>>>> metrics
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Hi RPM friends and IPPM'ers,
> > >>>>>>>>>
> > >>>>>>>>> I was wondering what a comparison of some of the "working
> > >> latency"
> > >>>>>> metrics
> > >>>>>>>>> would look like, so I ran some tests using a service on DOCSIS
> > >>> 3.1, with
> > >>>>>> the
> > >>>>>>>>> downlink provisioned for 1Gbps.
> > >>>>>>>>>
> > >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097), and
> > >>> Ookla
> > >>>>>>>> Speedtest
> > >>>>>>>>> with as similar connectivity as possible (but we know that the
> > >>> traffic
> > >>>>>> will
> > >>>>>>>>> diverge to different servers and we can't change that aspect).
> > >>>>>>>>>
> > >>>>>>>>> Here's a quick summary of yesterday's results:
> > >>>>>>>>>
> > >>>>>>>>> Working Latency & Capacity Summary
> > >>>>>>>>>
> > >>>>>>>>> Net Qual UDPST Ookla
> > >>>>>>>>> DnCap RPM DnCap RTTmin RTTVarRnge DnCap
> > >>> Ping(no
> > >>>>>> load)
> > >>>>>>>>> 878 62 970 28 0-19 941 6
> > >>>>>>>>> 891 92 970 27 0-20 940 7
> > >>>>>>>>> 891 120 966 28 0-22 937 9
> > >>>>>>>>> 890 112 970 28 0-21 940 8
> > >>>>>>>>> 903 70 970 28 0-16 935 9
> > >>>>>>>>>
> > >>>>>>>>> Note: all RPM values were categorized as Low.
> > >>>>>>>>>
> > >>>>>>>>> networkQuality downstream capacities are always on the low side
> > >>> compared
> > >>>>>> to
> > >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's mostly
> > >>> what
> > >>>>>> Ookla
> > >>>>>>>>> achieved. I think that a longer test duration might be needed to
> > >>> achieve
> > >>>>>> the
> > >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate values
> > >>> observed
> > >>>>>> were
> > >>>>>>>>> certainly headed in the right direction. (I recently upgraded to
> > >>>>>> Monterey
> > >>>>>>>> 12.6
> > >>>>>>>>> on my MacBook, so should have the latest version.)
> > >>>>>>>>>
> > >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded me, I
> > >>> should
> > >>>>>> have
> > >>>>>>>>> run the tests with the -v option to help with comparisons. I'll
> > >>> repeat
> > >>>>>> this
> > >>>>>>>>> test when I can make time.
> > >>>>>>>>>
> > >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> > >> the
> > >>> test)
> > >>>>>> and
> > >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up to
> > >>> very
> > >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM graded
> > >>> this
> > >>>>>> access
> > >>>>>>>> and
> > >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm in
> > >>> Chicago
> > >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> > >>> measurements
> > >>>>>> were
> > >>>>>>>> run
> > >>>>>>>>> on an Ubuntu VM in my MacBook.
> > >>>>>>>>>
> > >>>>>>>>> The big disappointment was that the Ookla desktop app I updated
> > >>> over the
> > >>>>>>>>> weekend did not include the new responsiveness metric! I
> > >> included
> > >>> the
> > >>>>>> ping
> > >>>>>>>>> results anyway, and it was clearly using a server in the nearby
> > >>> area.
> > >>>>>>>>>
> > >>>>>>>>> So, I have some more work to do, but I hope this is interesting-
> > >>> enough
> > >>>>>> to
> > >>>>>>>>> start some comparison discussions, and bring-out some
> > >> suggestions.
> > >>>>>>>>>
> > >>>>>>>>> happy testing all,
> > >>>>>>>>> Al
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> ippm mailing list
> > >>>>>>>>> ippm@ietf.org
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > >>>>>>>>
> > >>>>>>>> _______________________________________________
> > >>>>>>>> ippm mailing list
> > >>>>>>>> ippm@ietf.org
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>>> T!g-
> > >>>>>>
> > >>>
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > >>>>>>>
> > >>>>>>> _______________________________________________
> > >>>>>>> ippm mailing list
> > >>>>>>> ippm@ietf.org
> > >>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >>>>>>
> > >>>
> > >>
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > >>>>>> KXU$
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> This song goes out to all the folk that thought Stadia would work:
> > >>>>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >>> mushroom-
> > >>>>>> song-activity-6981366665607352320-
> > >>>>>>
> > >>>
> > >>
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > >>>>>> c48iQFub34zz4iE$
> > >>>>>> Dave Täht CEO, TekLibre, LLC
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> This song goes out to all the folk that thought Stadia would work:
> > >>>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >>> mushroom-song-activity-6981366665607352320-
> > >>>
> > >>
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHLHDpSWs$
> > >>>> Dave Täht CEO, TekLibre, LLC
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> This song goes out to all the folk that thought Stadia would work:
> > >>> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > >> mushroom-
> > >>> song-activity-6981366665607352320-
> > >>>
> > >>
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > >>> IEUcSswHLHDpSWs$
> > >>> Dave Täht CEO, TekLibre, LLC
> > >> _______________________________________________
> > >> ippm mailing list
> > >> ippm@ietf.org
> > >>
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > >> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> > rDt9HLI$
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > >
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> > T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> > mERYCyuCd9e9PY3aNLkSw$
>
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!h-glaezufaCxidYk1xzTF48dbEz67JPIJjPA_nweL8YvDu6Z3TmG0A37k_DQ15FIzzwoeCLOEaw$
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics
2022-12-11 19:21 ` MORTON JR., AL
@ 2022-12-12 2:38 ` Dave Taht
0 siblings, 0 replies; 29+ messages in thread
From: Dave Taht @ 2022-12-12 2:38 UTC (permalink / raw)
To: MORTON JR., AL
Cc: Sebastian Moeller, Randall Meyer, Rpm, Will Hawkins,
IETF IPPM WG, Pete Heist
[-- Attachment #1: Type: text/plain, Size: 44205 bytes --]
Adding in the author of irtt.
You had a significant timer error rate in your 1ms testing. Modern
hardware, particularly vms have great difficulty doing that below 3ms.
Similarly the cloud can be highly noisy. I am presently testing a highly
sdn - and virtualized networking fabric in a major DC... And the results
are rather disturbing. I'd like to repeat your test suite inside that DC.
On Sun, Dec 11, 2022, 11:21 AM MORTON JR., AL <acmorton@att.com> wrote:
> Hi IPPM,
>
> Prior to IETF-115, I shared a series of measurements with the IPPM list.
> We're looking at responsiveness and working latency with various metrics
> and multiple testing utilities. This message continues the discussion with
> new input.
>
> When I first published some measurements, Dave Taht added his assessment
> and included other relevant email lists in the discussion. I'm continuing
> to cross-post to all the lists in this thread.
>
> Dave originally suggested that I try a tool called irtt; I've done that
> now and these are the results.
>
> Bob McMahon: I queued your request to try your iperf2 tool behind the irtt
> measurements. I hope to make some more measurements this week...
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-
>
> We're testing a DOCIS 3.1 based service with 1Gbps down, nominally 22Mbps
> up. I used wired ETH connected to the DOCSIS modem's switch.
>
> Dave Taht made his server available for the irtt measurements. I installed
> irtt on a VM in my Macbook, the same VM that runs UDPST. I ran quite a few
> tests to become familiar with irtt, so I'll just summarize the relevant
> ones here.
>
> I ran irtt with its traffic at the maximum allowed on Dave's server. The
> test duration was 10sec, with packets spaced at 1ms intervals from my VM
> client. This is the complete output:
>
> ./irtt client -i 1ms -l 1250 -d 10s fremont.starlink.taht.net
>
> Min Mean Median Max Stddev
> --- ---- ------ --- ------
> RTT 46.63ms 51.58ms 51.4ms 58ms 1.55ms
> send delay 20.74ms 25.57ms 25.4ms 32.04ms 1.54ms
> receive delay 25.8ms 26.01ms 25.96ms 30.48ms 219µs
>
> IPDV (jitter) 1.03µs 1.15ms 1.02ms 6.87ms 793µs
> send IPDV 176ns 1.13ms 994µs 6.41ms 776µs
> receive IPDV 7ns 79.8µs 41.7µs 4.54ms 140µs
>
> send call time 10.1µs 55.8µs 1.34ms 30.3µs
> timer error 68ns 431µs 6.49ms 490µs
> server proc. time 680ns 2.43µs 47.7µs 1.73µs
>
> duration: 10.2s (wait 174ms)
> packets sent/received: 7137/7131 (0.08% loss)
> server packets received: 7131/7137 (0.08%/0.00% loss up/down)
> bytes sent/received: 8921250/8913750
> send/receive rate: 7.14 Mbps / 7.13 Mbps
> packet length: 1250 bytes
> timer stats: 2863/10000 (28.63%) missed, 43.10% error
> acm@acm-ubuntu1804-1:~/goirtt/irtt$
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> irtt supplies lots of info/stats about the RTT distribution. In the
> lightly loaded (7.1Mbps) scenario above, the RTT range is about 12 ms. The
> Mean and the Median are approximately the same.
> irtt also supplies Inter-Packet Delay Varition (IPDV), showing that the
> packets seem to be delayed more than the sending interval occasionally (Max
> of 6.8ms).
> irtt measurements indicate very low packet loss: no congestion at 7Mbps.
>
> For the opposite end of the congestion spectrum, I ran irtt with UDPST
> (RFC 9097) running in parallel (and using the Type C search algorithm). We
> pick-up a lot more RTT and wider RTT range in this scenario:
>
> irtt with udpst using Type C search = max load:
>
> Min Mean Median Max Stddev
> --- ---- ------ --- ------
> RTT 47.58ms 118ms 56.53ms 301.6ms 90.28ms
> send delay 24.05ms 94.85ms 33.38ms 278.5ms 90.26ms
> receive delay 22.99ms 23.17ms 23.13ms 25.42ms 156µs
>
> IPDV (jitter) 162ns 1.04ms 733µs 6.36ms 1.02ms
> send IPDV 3.81µs 1.01ms 697µs 6.24ms 1.02ms
> receive IPDV 88ns 93µs 49.8µs 1.48ms 145µs
>
> send call time 4.28µs 39.3µs 903µs 32.4µs
> timer error 86ns 287µs 6.13ms 214µs
> server proc. time 670ns 3.59µs 19.3µs 2.26µs
>
> duration: 10.9s (wait 904.8ms)
> packets sent/received: 8305/2408 (71.01% loss)
> server packets received: 2408/8305 (71.01%/0.00% loss up/down)
> bytes sent/received: 10381250/3010000
> send/receive rate: 8.31 Mbps / 2.47 Mbps
> packet length: 1250 bytes
> timer stats: 1695/10000 (16.95%) missed, 28.75% error
> acm@acm-ubuntu1804-1:~/goirtt/irtt$
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> The irtt measurement of RTT range is now about 250ms in this
> maximally-loaded scenario.
> One key RTT difference is the Mean delay, which is now twice the Median
> with working load added by UDPST.
> Congestion is evident in the irtt loss measurements = 71%. However, the
> competing traffic did not break irtt's process or measurements.
> UDPST's measurements were ~22Mbps Capacity (steady-state), with high loss,
> and similar RTT range of 251 ms.
>
>
> Additional tests included load from UDPST at fixed rates: 14Mbps and
> 20Mbps. We compare the RTT range for all four conditions in the table below:
>
> Load with irtt irtt RTT range UDPST RTT range
> ==================================================
> irtt alone 12ms ---
> UDPST at 14Mbps 11ms 22ms
> UDPST at 20Mbps 14ms 46ms
> UDPST at MaxCap 250ms 251ms
>
> The unexpected result with irtt measurements is that the RTT range did not
> increase with load, where the UDPST RTT range increases with load. We're
> assuming that the majority of delay increases occur in the DOCSIS upstream
> queue and both test streams should see similar delay range as they do with
> maximum load. Perhaps there are some differences owing to the periodic irtt
> stream and the bursty UDPST stream (both are UDP), but this is speculation.
>
> To check that the test path was operating similarly with earlier tests, we
> ran a couple of NetworkQuality and UDPST tests as before:
>
> Comparison of NQ-vs, UDPST -A c, same columns as defined earlier.
> Working Latency & Capacity Summary (Upstream only)
> (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> Medium)
>
> Net Qual UDPST (RFC 9097)
> UpCap RPM DelLD DelMin UpCap(stable) RTTmin RTTrange
> 22 276 m 217ms 11ms 23 28 0-252
> 22 291 m 206ms 11ms ~22* 28* 0-251*
>
> * UDPST test result with the ~7Mbps irtt stream present.
>
> We found that uplink test results are similar to previous tests, and the
> new irtt results were collected under similar conditions.
>
> In conclusion, irtt provides a clear summary of the RTT distribution.
> Minimum, Mean, Median and Max RTT are useful individually and in
> combinations/comparisons. The irtt measurements compare favorably to those
> collected during IP-Layer Capacity tests with the UDPST utility.
>
> comments welcome,
> Al
>
> > -----Original Message-----
> > From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > Sent: Saturday, November 5, 2022 3:37 PM
> > To: Sebastian Moeller <moeller0@gmx.de>
> > Cc: Rpm <rpm@lists.bufferbloat.net>; Will Hawkins <hawkinsw@obs.cr>;
> > ippm@ietf.org
> > Subject: Re: [ippm] [Rpm] Preliminary measurement comparison of "Working
> > Latency" metrics
> >
> > *** Security Advisory: This Message Originated Outside of AT&T ***.
> > Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> >
> > Hi Sebastian, thanks for your comments and information.
> > I'll reply to a few key points in this top-post.
> >
> > Sebastian wrote:
> > > [SM] DOCSIS ISPs traditionally provision higher peak rates than
> they
> > > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even
> via
> > > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput
> in the
> > > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > > achievable throughput somewhat:
> > [acm]
> > Right, my ISP's CPE has Gig-E ports so I won't see any benefit from
> over-rate
> > provisioning.
> >
> > Sebastian wrote:
> > ...
> > > IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 -
> 20)/(1500
> > + 38 + 4)) = 939.04 Mbps
> > ...
> > > Speedtests tend to report IPv4/TCP timestamps might be on depending on
> the
> > OS,
> > > but on-line speedtest almost never return the simple average rate over
> the
> > > measurement duration, but play "clever" tricks to exclude the start-up
> phase
> > > and to aggregate over multiple flows that invariably ending with
> results
> > that
> > > tend to exceed the hard limits shown above...
> > [acm]
> > My measurements with the Ookla desktop app on MacOS (shared earlier this
> week)
> > are very consistent at ~940Mbps Dn-stream, so timestamps calculation
> sounds
> > right.
> > My ISP specifies their service speed at 940Mbps, as though they assume
> > subscribers will measure using Ookla or other TCP tool. In fact, our team
> > hasn't seen such consistent results from Ookla or any TCP-based test in
> the
> > 1Gbps range, and this makes me wonder if there might be some
> test-recognition
> > here.
> >
> > FYI - UDPST uses a max packet size with is sub-MTU, to avoid
> fragmentation
> > when various encapsulations are encountered. Also, the definition of
> IP-Layer
> > Capacity in the various standards (e.g., RFC 9097) includes the bits in
> the IP
> > header in the total Capacity.
> >
> > So, instead of:
> > > Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38
> + 4)) =
> > 972.76 Mbps
> > We have
> > Ethernet payload rate +VLAN: 1000 * ((1222)/(1222 + 38 + 4)) =
> 966.77
> > Mbps
> > and why our Maximum IP-Layer Capacity measurements are approx ~967 Mbps
> >
> > UDPST has an option to use datagrams for the traditional 1500 octet MTU
> (-T),
> > but a user could cause a lot of fragmentation on links with
> encapsulations wit
> > this option.
> >
> > acm wrote:
> > > > The comparison of networkQuality and goresponsiveness is somewhat
> > confounded
> > > > by the need to use the Apple server infrastructure for both these
> methods
> > ...
> > Sebastian wrote:
> > > [SM] Puzzled, I thought when comparing the two networkQuality
> variants
> > > having the same back-end sort of helps be reducing the differences to
> the
> > > clients? But comparison to UDPST suffers somewhat.
> > [acm]
> > Yes, I was hoping to match-up the client and server implementations.
> Instead,
> > this might be more of a Client-X and Server-Y interoperability test (and
> could
> > explain some results?), but that was not to be.
> >
> > Thanks for your suggested command line for gores.
> >
> > IIRC one of your early messages, Sebastian, your MacOS indicates that
> 12.6 is
> > the latest. It's the same for me: my sufficiently powerful MacBook cannot
> > upgrade to Ventura for the latest in networkQuality versions. It would
> help us
> > who are doing some testing if the latest version of networkQuality could
> be
> > made installable for us, somehow...
> >
> > thanks again and regards,
> > Al
> >
> >
> > > -----Original Message-----
> > > From: Sebastian Moeller <moeller0@gmx.de>
> > > Sent: Friday, November 4, 2022 3:10 PM
> > > To: MORTON JR., AL <acmorton@att.com>
> > > Cc: Dave Täht <dave.taht@gmail.com>; rjmcmahon <
> rjmcmahon@rjmcmahon.com>;
> > Rpm
> > > <rpm@lists.bufferbloat.net>; ippm@ietf.org; Will Hawkins <
> hawkinsw@obs.cr>
> > > Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of
> "Working
> > > Latency" metrics
> > >
> > > Hi Al,
> > >
> > >
> > > > On Nov 4, 2022, at 18:14, MORTON JR., AL via Rpm
> > <rpm@lists.bufferbloat.net>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I have been working through the threads on misery metrics and
> lightweight
> > > sensing of bw and buffering, and with those insights and gold-nuggets
> in
> > mind,
> > > I'm iterating through testing the combinations of tools that Dave That
> and
> > Bob
> > > McMahon suggested.
> > > >
> > > > earlier this week, Dave wrote:
> > > >> How does networkQuality compare vs a vs your tool vs a vs
> > goresponsiveness?
> > > >
> > > > goresponsiveness installed flawlessly - very nice instructions and
> > getting-
> > > started info.
> > > >
> > > > Comparison of NetQual (networkQuality -vs), UDPST -A c,
> > > gores/goresponsiveness
> > > >
> > > > Working Latency & Capacity Summary on DOCSIS 3.1 access with 1 Gbps
> down-
> > > stream service
> > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and
> > > Medium)
> > > >
> > > > NetQual UDPST (RFC 9097) gores
> > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap
> > RPM
> > > DelLD
> > > > 882 788 m 76ms 8ms 967 28 0-16 127
> > > (1382) 43ms
> > > > 892 1036 h 58ms 8 966 27 0-18 128
> > > (2124) 28ms
> > > > 887 1304 h 46ms 6 969 27 0-18 130
> > > (1478) 41ms
> > > > 885 1008 h 60ms 8 967 28 0-22 127
> > > (1490) 40ms
> > > > 894 1383 h 43ms 11 967 28 0-15 133
> > > (2731) 22ms
> > > >
> > > > NetQual UDPST (RFC 9097) gores
> > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap
> > RPM
> > > DelLD
> > > > 21 327 m 183ms 8ms 22 (51) 28 0-253 12
> > > > 21 413 m 145ms 8 22 (43) 28 0-255 15
> > > > 22 273 m 220ms 6 23 (53) 28 0-259 10
> > > > 21 377 m 159ms 8 23 (51) 28 0-250 10
> > > > 22 281 m 214ms 11 23 (52) 28 0-250 6
> > > >
> > > > These tests were conducted in a round-robin fashion to minimize the
> > > possibility of network variations between measurements:
> > > > NetQual - rest - UDPST-Dn - rest- UDPST-Up - rest - gores - rest -
> repeat
> > > >
> > > > NetQual indicates the same reduced capacity in Downstream when
> compared to
> > > UDPST (940Mbps is the max for TCP payloads, while 967-970 is max for
> IP-
> > layer
> > > capacity, dep. on VLAN tag).
> > >
> > > [SM] DOCSIS ISPs traditionally provision higher peak rates than
> they
> > > advertise, with a DOCSIS modem/router with > 1 Gbps LAN capacity (even
> via
> > > bonded LAG) people in Germany routinely measure TCP/IPv4/HTTP goodput
> in the
> > > 1050 Mbps range. But typically gigabit ethernet limits the practically
> > > achievable throughput somewhat:
> > >
> > > Ethernet payload rate: 1000 * ((1500)/(1500 +
> 38)) =
> > > 975.29 Mbps
> > > Ethernet payload rate +VLAN: 1000 * ((1500)/(1500 + 38
> + 4)) =
> > > 972.76 Mbps
> > > IPv4 payload (ethernet+VLAN): 1000 * ((1500 - 20)/(1500
> + 38 + 4))
> > =
> > > 959.79 Mbps
> > > IPv6 payload (ethernet+VLAN): 1000 * ((1500 - 40)/(1500
> + 38 + 4))
> > =
> > > 946.82 Mbps
> > > IPv4/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 20)/(1500 + 38
> > +
> > > 4)) = 946.82 Mbps
> > > IPv6/TCP payload (ethernet+VLAN): 1000 * ((1500 - 20 - 40)/(1500 + 38
> > +
> > > 4)) = 933.85 Mbps
> > > IPv4/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 -
> 20)/(1500
> > +
> > > 38 + 4)) = 939.04 Mbps
> > > IPv6/TCP/RFC1323 timestamps payload: 1000 * ((1500 - 12 - 20 -
> 40)/(1500
> > +
> > > 38 + 4)) = 926.07 Mbps
> > >
> > >
> > > Speedtests tend to report IPv4/TCP timestamps might be on depending on
> the
> > OS,
> > > but on-line speedtest almost never return the simple average rate over
> the
> > > measurement duration, but play "clever" tricks to exclude the start-up
> phase
> > > and to aggregate over multiple flows that invariably ending with
> results
> > that
> > > tend to exceed the hard limits shown above...
> > >
> > >
> > > > Upstream capacities are not very different (a factor that made TCP
> methods
> > > more viable many years ago when most consumer access speeds were
> limited to
> > > 10's of Megabits).
> > >
> > > [SM] My take on this is that this is partly due to the goal of
> ramping
> > > up very quickly and get away with a short measurement duration. That
> causes
> > > imprecision. As Dave said flent's RRUL test defaults to 60 seconds and
> I
> > often
> > > ran/run it for 5 to 10 minutes to get somewhat more reliable numbers
> (and
> > for
> > > timecourses to look at and reason about).
> > >
> > > > gores reports significantly lower capacity in both downstream and
> upstream
> > > measurements, a factor of 7 less than NetQual for downstream.
> Interestingly,
> > > the reduced capacity (taken as the working load) results in higher
> > > responsiveness: RPM meas are higher and loaded delays are lower for
> > > downstream.
> > >
> > > [SM] Yepp, if you only fill-up the queue partially you will
> harvest less
> > > queueing delay and hence retain more responsiveness, albeit this
> really just
> > > seems to be better interpreted as go-responsiveness failing to achieve
> > > "working-conditions".
> > >
> > > I tend to run gores like this:
> > > time ./networkQuality --config mensura.cdn-apple.com --port 443 --path
> > > /api/v1/gm/config --sattimeout 60 --extended-stats --logger-filename
> > > go_networkQuality_$(date +%Y%m%d_%H%M%S)
> > >
> > > --sattimeout 60 extends the time out for the saturation measurement
> somewhat
> > > (before I saw it simply failing on a 1 Gbps access link, it did give
> some
> > > diagnostic message though).
> > >
> > >
> > > >
> > > > The comparison of networkQuality and goresponsiveness is somewhat
> > confounded
> > > by the need to use the Apple server infrastructure for both these
> methods
> > (the
> > > documentation provides this option - thanks!). I don't have admin
> access to
> > > our server at the moment. But the measured differences are large
> despite the
> > > confounding factor.
> > >
> > > [SM] Puzzled, I thought when comparing the two ntworkQuality
> variants
> > > having the same back-end sort of helps be reducing the differences to
> the
> > > clients? But comparison to UDPST suffers somewhat.
> > >
> > > >
> > > > goresponsiveness has its own, very different output format than
> > > networkQuality. There isn't a comparable "-v" option other than -debug
> > (which
> > > is extremely detailed). gores only reports RPM for the downstream.
> > >
> > > [SM] I agree it would be nice if gores would grow a sequential
> mode as
> > > well.
> > >
> > >
> > > > I hope that these results will prompt more of the principle
> evangelists
> > and
> > > coders to weigh-in.
> > >
> > > [SM] No luck ;) but at least "amateur hour" (aka me) is ready to
> > > discuss.
> > >
> > > >
> > > > It's also worth noting that the RTTrange reported by UDPST is the
> range
> > > above the minimum RTT, and represents the entire 10 second test. The
> > > consistency of the maximum of the range (~255ms) seems to indicate that
> > UDPST
> > > has characterized the length of the upstream buffer during
> measurements.
> > >
> > > [SM] thanks for explaining that.
> > >
> > > Regards
> > > Sebastian
> > >
> > > >
> > > > I'm sure there is more to observe that is prompted by these
> measurements;
> > > comments welcome!
> > > >
> > > > Al
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR., AL
> > > >> Sent: Tuesday, November 1, 2022 10:51 AM
> > > >> To: Dave Taht <dave.taht@gmail.com>
> > > >> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > Latency"
> > > >> metrics
> > > >>
> > > >> *** Security Advisory: This Message Originated Outside of AT&T ***.
> > > >> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> > information.
> > > >>
> > > >> Hi Dave,
> > > >> Thanks for trying UDPST (RFC 9097)!
> > > >>
> > > >> Something you might try with starlink:
> > > >> use the -X option and UDPST will generate random payloads.
> > > >>
> > > >> The measurements with -X will reflect the uncompressed that are
> possible.
> > > >> I tried this on a ship-board Internet access: uncompressed rate was
> > > 100kbps.
> > > >>
> > > >> A few more quick replies below,
> > > >> Al
> > > >>
> > > >>> -----Original Message-----
> > > >>> From: Dave Taht <dave.taht@gmail.com>
> > > >>> Sent: Tuesday, November 1, 2022 12:22 AM
> > > >>> To: MORTON JR., AL <acmorton@att.com>
> > > >>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >>> Subject: Re: [ippm] Preliminary measurement comparison of "Working
> > > Latency"
> > > >>> metrics
> > > >>>
> > > >>> Dear Al:
> > > >>>
> > > >>> OK, I took your udpst tool for a spin.
> > > >>>
> > > >>> NICE! 120k binary (I STILL work on machines with only 4MB of
> flash),
> > > >>> good feature set, VERY fast,
> > > >> [acm]
> > > >> Len Ciavattone (my partner in crime on several measurement
> projects) is
> > the
> > > >> lead coder: he has implemented many measurement tools extremely
> > > efficiently,
> > > >> this one in C-lang.
> > > >>
> > > >>> and in very brief testing, seemed
> > > >>> to be accurate in the starlink case, though it's hard to tell with
> > > >>> them as the rate changes every 15s.
> > > >> [acm]
> > > >> Great! Our guiding principle developing UDPST has been to test the
> > accuracy
> > > of
> > > >> measurements against a ground-truth. It pays-off.
> > > >>
> > > >>>
> > > >>> I filed a couple bug reports on trivial stuff:
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/BroadbandForum/obudpst/issues/8
> > > >>>
> > > >>
> > >
> >
> __;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUc
> > > >>> SswH_opYmEw$
> > > >> [acm]
> > > >> much appreciated... We have an OB-UDPST project meeting this
> Friday, can
> > > >> discuss then.
> > > >>
> > > >>>
> > > >>> (Adding diffserv and ecn washing or marking detection would be a
> nice
> > > >>> feature to have)
> > > >>>
> > > >>> Aside from the sheer joy coming from the "it compiles! and runs!"
> > > >>> phase I haven't looked much further.
> > > >>>
> > > >>> I left a copy running on one of my starlink testbeds -
> > > >>> fremont.starlink.taht.net - if anyone wants to try it. It's
> > > >>> instrumented with netperf, flent, irtt, iperf2 (not quite the
> latest
> > > >>> version from bob, but close), and now udpst, and good to about a
> gbit.
> > > >>>
> > > >>> nice tool!
> > > >> [acm]
> > > >> Thanks again!
> > > >>
> > > >>>
> > > >>> Has anyone here played with crusader? (
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://github.com/Zoxc/crusader__;!!BhdT!iufMVqCy
> > > >>>
> oH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHm4Wtzjc$
> > > )
> > > >>>
> > > >>> On Mon, Oct 31, 2022 at 4:30 PM Dave Taht <dave.taht@gmail.com>
> wrote:
> > > >>>>
> > > >>>> On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL <acmorton@att.com>
> > wrote:
> > > >>>>
> > > >>>>>> have you tried irtt?
> > > >>>
> > > >>
> > >
> > (
> https://urldefense.com/v3/__https://github.com/heistp/irtt__;!!BhdT!iufMVqCyo
> > > >>>
> H_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHBSirIcE$
> > > )
> > > >>>>> I have not. Seems like a reasonable tool for UDP testing. The
> feature
> > I
> > > >>> didn't like in my scan of the documentation is the use of
> Inter-packet
> > > delay
> > > >>> variation (IPDV) instead of packet delay variation (PDV):
> variation from
> > > the
> > > >>> minimum (or reference) delay. The morbidly curious can find my
> analysis
> > in
> > > >> RFC
> > > >>> 5481:
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/rfc5481__;
> !!
> > > >>>
> > > >>
> > >
> >
> BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nOIEUcSswHt
> > > >>> T7QSlc$
> > > >>>>
> > > >>>> irtt was meant to simulate high speed voip and one day
> > > >>>> videoconferencing. Please inspect the json output
> > > >>>> for other metrics. Due to OS limits it is typically only accurate
> to a
> > > >>>> 3ms interval. One thing it does admirably is begin to expose the
> > > >>>> sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless
> > > >>>> technologies, as well as request/grant systems like cable and
> gpon,
> > > >>>> especially when otherwise idle.
> > > >>>>
> > > >>>> Here is a highres plot of starlink's behaviors from last year:
> > > >>>> https://urldefense.com/v3/__https://forum.openwrt.org/t/cake-w-
> > adaptive-
> > > >>> bandwidth-
> > > >>>
> > > >>
> > >
> >
> historic/108848/3238__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vle
> > > >>> KCIKpL3PBLwci5nOIEUcSswH_5Sms3w$
> > > >>>>
> > > >>>> clearly showing them "optimizing for bandwidth" and changing next
> sat
> > > >>>> hop, and about a 40ms interval of buffering between these
> switches.
> > > >>>> I'd published elsewhere, if anyone cares, a preliminary study of
> what
> > > >>>> starlink's default behaviors did to cubic and BBR...
> > > >>>>
> > > >>>>>
> > > >>>>> irtt's use of IPDV means that the results won’t compare with
> UDPST,
> > and
> > > >>> possibly networkQuality. But I may give it a try anyway...
> > > >>>>
> > > >>>> The more the merrier! Someday the "right" metrics will arrive.
> > > >>>>
> > > >>>> As a side note, this paper focuses on RAN uplink latency
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://dl.ifip.org/db/conf/itc/itc2021/1570740615
> > > >>>
> > > >>
> > >
> >
> .pdf__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHgvqerjg$ which I think
> > > >>>> is a major barrier to most forms of 5G actually achieving good
> > > >>>> performance in a FPS game, if it is true for more RANs. I'd like
> more
> > > >>>> to be testing uplink latencies idle and with load, on all
> > > >>>> technologies.
> > > >>>>
> > > >>>>>
> > > >>>>> thanks again, Dave.
> > > >>>>> Al
> > > >>>>>
> > > >>>>>> -----Original Message-----
> > > >>>>>> From: Dave Taht <dave.taht@gmail.com>
> > > >>>>>> Sent: Monday, October 31, 2022 12:52 PM
> > > >>>>>> To: MORTON JR., AL <acmorton@att.com>
> > > >>>>>> Cc: ippm@ietf.org; Rpm <rpm@lists.bufferbloat.net>
> > > >>>>>> Subject: Re: [ippm] Preliminary measurement comparison of
> "Working
> > > >>> Latency"
> > > >>>>>> metrics
> > > >>>>>>
> > > >>>>>> Thank you very much for the steer to RFC9097. I'd completely
> missed
> > > >>> that.
> > > >>>>>>
> > > >>>>>> On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL <
> acmorton@att.com>
> > > >> wrote:
> > > >>>>>>>
> > > >>>>>>> (astute readers may have guessed that I pressed "send" too
> soon on
> > > >>> previous
> > > >>>>>> message...)
> > > >>>>>>>
> > > >>>>>>> I also conducted upstream tests this time, here are the
> results:
> > > >>>>>>> (capacity in Mbps, delays in ms, h and m are RPM categories,
> High
> > > >> and
> > > >>>>>> Medium)
> > > >>>>>>>
> > > >>>>>>> Net Qual UDPST (RFC9097)
> > > >> Ookla
> > > >>>>>>> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange
> > > >> UpCap
> > > >>>>>> Ping(no load)
> > > >>>>>>> 34 1821 h 33ms 11ms 23 (42) 28 0-252
> 22
> > > >>> 8
> > > >>>>>>> 22 281 m 214ms 8ms 27 (52) 25 5-248
> 22
> > > >>> 8
> > > >>>>>>> 22 290 m 207ms 8ms 27 (55) 28 0-253
> 22
> > > >>> 9
> > > >>>>>>> 21 330 m 182ms 11ms 23 (44) 28 0-255
> 22
> > > >>> 7
> > > >>>>>>> 22 334 m 180ms 9ms 33 (56) 25 0-255
> 22
> > > >>> 9
> > > >>>>>>>
> > > >>>>>>> The Upstream capacity measurements reflect an interesting
> feature
> > > >> that
> > > >>> we
> > > >>>>>> can reliably and repeatably measure with UDPST. The first ~3
> seconds
> > > >> of
> > > >>>>>> upstream data experience a "turbo mode" of ~50Mbps. UDPST
> displays
> > > >> this
> > > >>>>>> behavior in its 1 second sub-interval measurements and has a
> bimodal
> > > >>> reporting
> > > >>>>>> option that divides the complete measurement interval in two
> time
> > > >>> intervals to
> > > >>>>>> report an initial (turbo) max capacity and a steady-state max
> > capacity
> > > >>> for the
> > > >>>>>> later intervals. The UDPST capacity results present both
> > measurements:
> > > >>> steady-
> > > >>>>>> state first.
> > > >>>>>>
> > > >>>>>> Certainly we can expect bi-model distributions from many ISPs,
> as,
> > for
> > > >>>>>> one thing, the "speedboost" concept remains popular, except
> that it's
> > > >>>>>> misnamed, as it should be called speed-subtract or speed-lose.
> Worse,
> > > >>>>>> it is often configured "sneakily", in that it doesn't kick in
> for the
> > > >>>>>> typical observed duration of the test, for some, they cut the
> > > >>>>>> available bandwidth about 20s in, others, 1 or 5 minutes.
> > > >>>>>>
> > > >>>>>> One of my biggest issues with the rpm spec so far is that it
> should,
> > > >>>>>> at least, sometimes, run randomly longer than the overly short
> > > >>>>>> interval it runs for and the tools also allow for manual
> override of
> > > >>> length.
> > > >>>>>>
> > > >>>>>> we caught a lot of tomfoolery with flent's rrul test running by
> > > >> default
> > > >>> for
> > > >>>>>> 1m.
> > > >>>>>>
> > > >>>>>> Also, AQMs on the path can take a while to find the optimal
> drop or
> > > >> mark
> > > >>> rate.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> The capacity processing in networkQuality and Ookla appear to
> report
> > > >>> the
> > > >>>>>> steady-state result.
> > > >>>>>>
> > > >>>>>> Ookla used to basically report the last result. Also it's not a
> good
> > > >>>>>> indicator of web traffic behavior at all, watching the curve
> > > >>>>>> go up much more slowly in their test on say, fiber 2ms, vs
> starlink,
> > > >>>>>> (40ms)....
> > > >>>>>>
> > > >>>>>> So adding another mode - how quickly is peak bandwidth actually
> > > >>>>>> reached, would be nice.
> > > >>>>>>
> > > >>>>>> I haven't poked into the current iteration of the
> goresponsiveness
> > > >>>>>> test at all:
> https://urldefense.com/v3/__https://github.com/network-
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvb
> > > >>>>>> K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it
> > > >>>>>> would be good to try collecting more statistics and histograms
> and
> > > >>>>>> methods of analyzing the data in that libre-source version.
> > > >>>>>>
> > > >>>>>> How does networkQuality compare vs a vs your tool vs a vs
> > > >>> goresponsiveness?
> > > >>>>>>
> > > >>>>>>> I watched the upstream capacity measurements on the Ookla app,
> and
> > > >>> could
> > > >>>>>> easily see the initial rise to 40-50Mbps, then the drop to
> ~22Mbps
> > for
> > > >>> most of
> > > >>>>>> the test which determined the final result.
> > > >>>>>>
> > > >>>>>> I tend to get upset when I see ookla's new test flash a peak
> result
> > in
> > > >>>>>> the seconds and then settle on some lower number somehow.
> > > >>>>>> So far as I know they are only sampling the latency every 250ms.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> The working latency is about 200ms in networkQuality and about
> 280ms
> > > >>> as
> > > >>>>>> measured by UDPST (RFC9097). Note that the networkQuality
> minimum
> > > >> delay
> > > >>> is
> > > >>>>>> ~20ms lower than the UDPST RTTmin, so this accounts for some of
> the
> > > >>> difference
> > > >>>>>> in working latency. Also, we used the very dynamic Type C load
> > > >>>>>> adjustment/search algorithm in UDPST during all of this testing,
> > which
> > > >>> could
> > > >>>>>> explain the higher working latency to some degree.
> > > >>>>>>>
> > > >>>>>>> So, it's worth noting that the measurements needed for
> assessing
> > > >>> working
> > > >>>>>> latency/responsiveness are available in the UDPST utility, and
> that
> > > >> the
> > > >>> UDPST
> > > >>>>>> measurements are conducted on UDP transport (used by a growing
> > > >> fraction
> > > >>> of
> > > >>>>>> Internet traffic).
> > > >>>>>>
> > > >>>>>> Thx, didn't know of this work til now!
> > > >>>>>>
> > > >>>>>> have you tried irtt?
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> comments welcome of course,
> > > >>>>>>> Al
> > > >>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR.,
> AL
> > > >>>>>>>> Sent: Sunday, October 30, 2022 8:09 PM
> > > >>>>>>>> To: ippm@ietf.org
> > > >>>>>>>> Subject: Re: [ippm] Preliminary measurement comparison of
> "Working
> > > >>>>>> Latency"
> > > >>>>>>>> metrics
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Hi again RPM friends and IPPM'ers,
> > > >>>>>>>>
> > > >>>>>>>> As promised, I repeated the tests shared last week, this time
> > > >> using
> > > >>> both
> > > >>>>>> the
> > > >>>>>>>> verbose (-v) and sequential (-s) dwn/up test options of
> > > >>> networkQuality. I
> > > >>>>>>>> followed Sebastian's calculations as well.
> > > >>>>>>>>
> > > >>>>>>>> Working Latency & Capacity Summary
> > > >>>>>>>>
> > > >>>>>>>> Net Qual UDPST
> > > >>> Ookla
> > > >>>>>>>> DnCap RPM DelLD DelMin DnCap RTTmin RTTrange
> > > >>> DnCap
> > > >>>>>>>> Ping(no load)
> > > >>>>>>>> 885 916 m 66ms 8ms 970 28 0-20
> > > >> 940
> > > >>> 8
> > > >>>>>>>> 888 1355 h 44ms 8ms 966 28 0-23
> > > >> 940
> > > >>> 8
> > > >>>>>>>> 891 1109 h 54ms 8ms 968 27 0-19
> > > >> 940
> > > >>> 9
> > > >>>>>>>> 887 1141 h 53ms 11ms 966 27 0-18
> > > >> 937
> > > >>> 7
> > > >>>>>>>> 884 1151 h 52ms 9ms 968 28 0-20
> > > >> 937
> > > >>> 9
> > > >>>>>>>>
> > > >>>>>>>> With the sequential test option, I noticed that networkQuality
> > > >>> achieved
> > > >>>>>> nearly
> > > >>>>>>>> the maximum capacity reported almost immediately at the start
> of a
> > > >>> test.
> > > >>>>>>>> However, the reported capacities are low by about 60Mbps,
> > > >> especially
> > > >>> when
> > > >>>>>>>> compared to the Ookla TCP measurements.
> > > >>>>>>>>
> > > >>>>>>>> The loaded delay (DelLD) is similar to the UDPST RTTmin + (the
> > > >> high
> > > >>> end of
> > > >>>>>> the
> > > >>>>>>>> RTTrange), for example 54ms compared to (27+19=46). Most of
> the
> > > >>>>>> networkQuality
> > > >>>>>>>> RPM measurements were categorized as "High". There doesn't
> seem to
> > > >>> be much
> > > >>>>>>>> buffering in the downstream direction.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> -----Original Message-----
> > > >>>>>>>>> From: ippm <ippm-bounces@ietf.org> On Behalf Of MORTON JR.,
> AL
> > > >>>>>>>>> Sent: Monday, October 24, 2022 6:36 PM
> > > >>>>>>>>> To: ippm@ietf.org
> > > >>>>>>>>> Subject: [ippm] Preliminary measurement comparison of
> "Working
> > > >>> Latency"
> > > >>>>>>>>> metrics
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Hi RPM friends and IPPM'ers,
> > > >>>>>>>>>
> > > >>>>>>>>> I was wondering what a comparison of some of the "working
> > > >> latency"
> > > >>>>>> metrics
> > > >>>>>>>>> would look like, so I ran some tests using a service on
> DOCSIS
> > > >>> 3.1, with
> > > >>>>>> the
> > > >>>>>>>>> downlink provisioned for 1Gbps.
> > > >>>>>>>>>
> > > >>>>>>>>> I intended to run apple's networkQuality, UDPST (RFC9097),
> and
> > > >>> Ookla
> > > >>>>>>>> Speedtest
> > > >>>>>>>>> with as similar connectivity as possible (but we know that
> the
> > > >>> traffic
> > > >>>>>> will
> > > >>>>>>>>> diverge to different servers and we can't change that
> aspect).
> > > >>>>>>>>>
> > > >>>>>>>>> Here's a quick summary of yesterday's results:
> > > >>>>>>>>>
> > > >>>>>>>>> Working Latency & Capacity Summary
> > > >>>>>>>>>
> > > >>>>>>>>> Net Qual UDPST Ookla
> > > >>>>>>>>> DnCap RPM DnCap RTTmin RTTVarRnge DnCap
> > > >>> Ping(no
> > > >>>>>> load)
> > > >>>>>>>>> 878 62 970 28 0-19 941
> 6
> > > >>>>>>>>> 891 92 970 27 0-20 940
> 7
> > > >>>>>>>>> 891 120 966 28 0-22 937
> 9
> > > >>>>>>>>> 890 112 970 28 0-21 940
> 8
> > > >>>>>>>>> 903 70 970 28 0-16 935
> 9
> > > >>>>>>>>>
> > > >>>>>>>>> Note: all RPM values were categorized as Low.
> > > >>>>>>>>>
> > > >>>>>>>>> networkQuality downstream capacities are always on the low
> side
> > > >>> compared
> > > >>>>>> to
> > > >>>>>>>>> others. We would expect about 940Mbps for TCP, and that's
> mostly
> > > >>> what
> > > >>>>>> Ookla
> > > >>>>>>>>> achieved. I think that a longer test duration might be
> needed to
> > > >>> achieve
> > > >>>>>> the
> > > >>>>>>>>> actual 1Gbps capacity with networkQuality; intermediate
> values
> > > >>> observed
> > > >>>>>> were
> > > >>>>>>>>> certainly headed in the right direction. (I recently
> upgraded to
> > > >>>>>> Monterey
> > > >>>>>>>> 12.6
> > > >>>>>>>>> on my MacBook, so should have the latest version.)
> > > >>>>>>>>>
> > > >>>>>>>>> Also, as Sebastian Moeller's message to the list reminded
> me, I
> > > >>> should
> > > >>>>>> have
> > > >>>>>>>>> run the tests with the -v option to help with comparisons.
> I'll
> > > >>> repeat
> > > >>>>>> this
> > > >>>>>>>>> test when I can make time.
> > > >>>>>>>>>
> > > >>>>>>>>> The UDPST measurements of RTTmin (minimum RTT observed during
> > > >> the
> > > >>> test)
> > > >>>>>> and
> > > >>>>>>>>> the range of variation above the minimum (RTTVarRnge) add-up
> to
> > > >>> very
> > > >>>>>>>>> reasonable responsiveness IMO, so I'm not clear why RPM
> graded
> > > >>> this
> > > >>>>>> access
> > > >>>>>>>> and
> > > >>>>>>>>> path as "Low". The UDPST server I'm using is in NJ, and I'm
> in
> > > >>> Chicago
> > > >>>>>>>>> conducting tests, so the minimum 28ms is typical. UDPST
> > > >>> measurements
> > > >>>>>> were
> > > >>>>>>>> run
> > > >>>>>>>>> on an Ubuntu VM in my MacBook.
> > > >>>>>>>>>
> > > >>>>>>>>> The big disappointment was that the Ookla desktop app I
> updated
> > > >>> over the
> > > >>>>>>>>> weekend did not include the new responsiveness metric! I
> > > >> included
> > > >>> the
> > > >>>>>> ping
> > > >>>>>>>>> results anyway, and it was clearly using a server in the
> nearby
> > > >>> area.
> > > >>>>>>>>>
> > > >>>>>>>>> So, I have some more work to do, but I hope this is
> interesting-
> > > >>> enough
> > > >>>>>> to
> > > >>>>>>>>> start some comparison discussions, and bring-out some
> > > >> suggestions.
> > > >>>>>>>>>
> > > >>>>>>>>> happy testing all,
> > > >>>>>>>>> Al
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> _______________________________________________
> > > >>>>>>>>> ippm mailing list
> > > >>>>>>>>> ippm@ietf.org
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$
> > > >>>>>>>>
> > > >>>>>>>> _______________________________________________
> > > >>>>>>>> ippm mailing list
> > > >>>>>>>> ippm@ietf.org
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>>> T!g-
> > > >>>>>>
> > > >>>
> >
> FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$
> > > >>>>>>>
> > > >>>>>>> _______________________________________________
> > > >>>>>>> ippm mailing list
> > > >>>>>>> ippm@ietf.org
> > > >>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFub_gMs
> > > >>>>>> KXU$
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> This song goes out to all the folk that thought Stadia would
> work:
> > > >>>>>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >>> mushroom-
> > > >>>>>> song-activity-6981366665607352320-
> > > >>>>>>
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxT
> > > >>>>>> c48iQFub34zz4iE$
> > > >>>>>> Dave Täht CEO, TekLibre, LLC
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> This song goes out to all the folk that thought Stadia would work:
> > > >>>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >>> mushroom-song-activity-6981366665607352320-
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHLHDpSWs$
> > > >>>> Dave Täht CEO, TekLibre, LLC
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> This song goes out to all the folk that thought Stadia would work:
> > > >>>
> https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-
> > > >> mushroom-
> > > >>> song-activity-6981366665607352320-
> > > >>>
> > > >>
> > >
> >
> FXtz__;!!BhdT!iufMVqCyoH_CQ2AXdNJV1QSjl_7srzb_IznWE87U6E583vleKCIKpL3PBLwci5nO
> > > >>> IEUcSswHLHDpSWs$
> > > >>> Dave Täht CEO, TekLibre, LLC
> > > >> _______________________________________________
> > > >> ippm mailing list
> > > >> ippm@ietf.org
> > > >>
> > >
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> > > >>
> T!jOVBx7DlKXbDMiZqaYSUhBtkSdUvfGpYUyGvLerdLsLBJZPMzEGcbhC9ZSzsZOd1dYC-
> > > rDt9HLI$
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > >
> > >
> >
> https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/rpm__;!!Bhd
> > > T!h8K1vAtpaGSUHpuVMl5sZgi7k-f64BEaV91ypoUokPjn57v_79iCnp7W-
> > > mERYCyuCd9e9PY3aNLkSw$
> >
> > _______________________________________________
> > ippm mailing list
> > ippm@ietf.org
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> >
> T!h-glaezufaCxidYk1xzTF48dbEz67JPIJjPA_nweL8YvDu6Z3TmG0A37k_DQ15FIzzwoeCLOEaw$
>
[-- Attachment #2: Type: text/html, Size: 66594 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2022-12-12 2:39 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CH0PR02MB79808E2508E6AED66DC7657AD32E9@CH0PR02MB7980.namprd02.prod.outlook.com>
[not found] ` <CH0PR02MB7980DFB52D45F2458782430FD3379@CH0PR02MB7980.namprd02.prod.outlook.com>
[not found] ` <CH0PR02MB7980D3036BF700A074D902A1D3379@CH0PR02MB7980.namprd02.prod.outlook.com>
2022-10-31 16:52 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics Dave Taht
2022-10-31 18:52 ` rjmcmahon
2022-10-31 22:08 ` MORTON JR., AL
2022-10-31 22:44 ` rjmcmahon
2022-11-01 20:15 ` [Rpm] lightweight active sensing of bandwidth and buffering Dave Taht
2022-11-01 23:39 ` rjmcmahon
2022-11-02 8:23 ` Sebastian Moeller
2022-11-02 9:41 ` [Rpm] [ippm] " Ruediger.Geib
2022-11-02 19:29 ` rjmcmahon
2022-11-02 19:44 ` Dave Taht
2022-11-02 20:37 ` rjmcmahon
2022-11-02 21:13 ` Sebastian Moeller
2022-11-02 21:41 ` Sebastian Moeller
2022-11-03 8:20 ` Ruediger.Geib
2022-11-03 8:57 ` Sebastian Moeller
2022-11-03 11:25 ` Ruediger.Geib
2022-11-03 11:48 ` Sebastian Moeller
2022-11-02 19:21 ` [Rpm] " rjmcmahon
2022-10-31 20:40 ` [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics MORTON JR., AL
2022-10-31 23:30 ` Dave Taht
2022-11-01 4:21 ` Dave Taht
2022-11-01 14:51 ` MORTON JR., AL
2022-11-04 17:14 ` MORTON JR., AL
2022-11-04 18:12 ` rjmcmahon
2022-11-04 18:58 ` MORTON JR., AL
2022-11-04 19:10 ` Sebastian Moeller
2022-11-05 19:36 ` MORTON JR., AL
2022-12-11 19:21 ` MORTON JR., AL
2022-12-12 2:38 ` Dave Taht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox