[Bloat] An ACM Queue article about bloat and LibraQoS

[Bloat] An ACM Queue article about bloat and LibraQoS

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

I have a new article on ACM Queue, at 
https://dl.acm.org/doi/pdf/10.1145/3674953 
<https://dl.acm.org/doi/pdf/10.1145/3674953>

"If you're an ISP and all your customers hate you, take heart. This is 
now a solvable problem."

It's part of a tongue-in-cheek series called "You Don't Know Jack", and 
it's about Bufferbloat and the LibreQoS project.

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net            |              -- Mark Twain

[-- Attachment #2: Type: text/html, Size: 1877 bytes --]

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[Sebastian Moeller]

Hi Dave,

nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.

Regards
	Sebastian


> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953
> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project. 
> --dave
> -- 
> David Collier-Brown, | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net | -- Mark Twain
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[David Collier-Brown]

On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote:
> Hi Dave,
>
> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.
>
> Regards
> 	Sebastian

That was a deliberate action on my part: the "after" status shows much 
smaller throughput numbers, leading observant readers to ask if 
improving latency didn't destroy one's bandwidth.

I suspect that the throughputs "with bloat" are greatly exaggerated... 
from counting the extra data that fills the bloated buffers (;-))

--dave


>
>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>
>> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953
>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project.
>> --dave
>> -- 
>> David Collier-Brown, | Always do right. This will gratify
>> System Programmer and Author | some people and astonish the rest
>> davecb@spamcop.net | -- Mark Twain
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |              -- Mark Twain

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[Sebastian Moeller]

Hi Dave,


> On 15. Jul 2024, at 12:59, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote:
>> Hi Dave,
>> 
>> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.
>> 
>> Regards
>> Sebastian
> 
> That was a deliberate action on my part: the "after" status shows much smaller throughput numbers, leading observant readers to ask if improving latency didn't destroy one's bandwidth.

	[SM] Ah, I see. But just for our discussion here, how much (potential) capacity did you need to treade in for decent responsiveness?

> I suspect that the throughputs "with bloat" are greatly exaggerated... from counting the extra data that fills the bloated buffers (;-))

	[SM] Good point... especially for upload tests, I occasionally see numbers which I know for a fact to be impossible (abovre the gross capacity of my access link) and I wonder whether all tests actually do proper reporting of the grand average capacity numbers. This should not be rocket science: take the start time of the first maesasurement flow and the end time of the last measurement flow and divide the total number of payload bytes trasferred by the total time this took. 

> 
> --dave
> 
> 
>> 
>>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>> 
>>> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953
>>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
>>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project.
>>> --dave
>>> -- 
>>> David Collier-Brown, | Always do right. This will gratify
>>> System Programmer and Author | some people and astonish the rest
>>> davecb@spamcop.net | -- Mark Twain
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> -- 
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net           |              -- Mark Twain
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 3201 bytes --]

Let's see... Found them!

score 	A+ 	D 	%
up 	9.1 	31.8 	29%
down 	163 	456.2 	36%

A+ figures were only 20-30% of the bloated D figure (:-()

I showed the article with both graphs to an outside reader and he 
spotted it instantly, and asked why I was only getting a tiny fraction 
of the "bloated" throughput.

--dave



On 2024-07-15 09:12, Sebastian Moeller wrote:
> Hi Dave,
>
>
>> On 15. Jul 2024, at 12:59, David Collier-Brown via Bloat<bloat@lists.bufferbloat.net>  wrote:
>>
>> On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote:
>>> Hi Dave,
>>>
>>> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.
>>>
>>> Regards
>>> Sebastian
>> That was a deliberate action on my part: the "after" status shows much smaller throughput numbers, leading observant readers to ask if improving latency didn't destroy one's bandwidth.
> 	[SM] Ah, I see. But just for our discussion here, how much (potential) capacity did you need to treade in for decent responsiveness?
>
>> I suspect that the throughputs "with bloat" are greatly exaggerated... from counting the extra data that fills the bloated buffers (;-))
> 	[SM] Good point... especially for upload tests, I occasionally see numbers which I know for a fact to be impossible (abovre the gross capacity of my access link) and I wonder whether all tests actually do proper reporting of the grand average capacity numbers. This should not be rocket science: take the start time of the first maesasurement flow and the end time of the last measurement flow and divide the total number of payload bytes trasferred by the total time this took.
>
>> --dave
>>
>>
>>>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat<bloat@lists.bufferbloat.net>  wrote:
>>>>
>>>> I have a new article on ACM Queue, athttps://dl.acm.org/doi/pdf/10.1145/3674953
>>>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
>>>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project.
>>>> --dave
>>>> -- 
>>>> David Collier-Brown, | Always do right. This will gratify
>>>> System Programmer and Author | some people and astonish the rest
>>>> davecb@spamcop.net  | -- Mark Twain
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> -- 
>> David Collier-Brown,         | Always do right. This will gratify
>> System Programmer and Author | some people and astonish the rest
>> davecb@spamcop.net            |              -- Mark Twain
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net            |              -- Mark Twain

[-- Attachment #2: Type: text/html, Size: 9147 bytes --]

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1062 bytes --]

Hi Dave,



> On 15. Jul 2024, at 16:26, David Collier-Brown <davec-b@rogers.com> wrote:
> 
> Let's see... Found them!
> score A+ D %
> up 9.1 31.8 29%
> down 163 456.2 36%
> A+ figures were only 20-30% of the bloated D figure (:-()
> I showed the article with both graphs to an outside reader and he spotted it instantly, and asked why I was only getting a tiny fraction of the "bloated" throughput. 
> --dave

[SM] OK, that clearly is a lot of throughput sacrifice... are you sure your link is as optimised as possible? If not I wozuld be happy to help.
Regarding other tools for measurements that include latency-under-load:

A) Ookla's speedtest.net <http://speedtest.net/> will also report the interquartil means for latency under load
Example: https://www.speedtest.net/result/16503399049

B) Cloudflare's speed.cloudflare.com <http://speed.cloudflare.com/> will also report some scores
Here is an article delving into how these scores are derived from the measured data
https://blog.cloudflare.com/aim-database-for-internet-quality

[-- Attachment #2: Screenshot 2024-07-16 at 15.06.20.png --]
[-- Type: image/png, Size: 856606 bytes --]

[-- Attachment #3: Type: text/plain, Size: 9882 bytes --]



c) Apple's networkQuality tool (on macos/ios) or the open source go version thereof: https://github.com/network-quality/goresponsiveness
user@laptop goresponsiveness % networkQuality -v -s
==== Verbose Results ====
---
Capacity:
---
   Uplink capacity: 39.688 Mbps
      Accuracy: High
      Uplink bytes transferred: 106.702 MB
      Uplink Flow count: 16
   Downlink capacity: 74.621 Mbps
      Accuracy: High
      Downlink bytes transferred: 50.644 MB
      Downlink Flow count: 6
---
Latency:
---
   Idle Latency:
      2618 RPM (22.917 milliseconds)
         Transport: 3178 RPM (18.875 milliseconds)
         Security: 1832 RPM (32.750 milliseconds)
         HTTP: 3503 RPM (17.125 milliseconds)
      Accuracy: High
   Uplink Responsiveness: Medium
      296 RPM (202.171 milliseconds)
         Transport: 3390 RPM (17.699 milliseconds)
         Security: 2260 RPM (26.538 milliseconds)
         HTTP: 3276 RPM (18.312 milliseconds)
         HTTP loaded: 155 RPM (386.579 milliseconds)
      Accuracy: High
   Downlink Responsiveness: Medium
      919 RPM (65.240 milliseconds)
         Transport: 3564 RPM (16.833 milliseconds)
         Security: 2488 RPM (24.115 milliseconds)
         HTTP: 3440 RPM (17.438 milliseconds)
         HTTP loaded: 430 RPM (139.362 milliseconds)
      Accuracy: High
---
Protocols Used:
---
    HTTP/2: 100%
---
Transport-layer info:
---
    ECN Disabled: 100%
---
Other Info:
---
   Test Endpoint: defra1-edge-bx-006.aaplimg.com
   Interface: en7
   Start: 2024-07-16 15:10:23.240
   End: 2024-07-16 15:10:51.921
   OS Version: Version 14.5 (Build 23F79)

Here the RPM for Uplink and Downlink HTTP are most relevant (these measure inter-flow latency increases that is how much the latency of flows stayiung below their capacity share is effected by parallel capacity seeking flows, HTTP loaded measures the self congestion of the load bearing flows)

Here an invocation of the opensource version (using apple's servers):

./networkQuality --detailed --relative-rpm --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --rpm.timeout 120 --extended-stats --rpm.mnp 32 --logger-filename go_networkQuality_20240716_151413
07-16-2024 13:14:13 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Unbounded Baseline RPM:  2857 (P90)
Unbounded Baseline RPM:  3322 (Single-Sided 5% Trimmed Mean)
Baseline RPM:  2827 (P90)
Baseline RPM:  3327 (Single-Sided 5% Trimmed Mean)
Results:
========
Download:
Throughput: 68.989 Mbps (8.624 MBps), using 26 parallel connections.
Extended Statistics:
Maximum Segment Size: 1208
Total Bytes Retransmitted: 0
Retransmission Ratio: 0.00%
Total Bytes Reordered: 140449506
Average RTT: 26.85185185185185
Unbounded RPM Calculation Statistics:
Total Self Probes:            1922
Total Foreign Probes:         5766
Trimmed Self Probes Count:    96 of 1922
Trimmed Foreign Probes Count: 288 of 5766
P90 Self RTT:                 0.635866s
P90 Foreign RTT:              0.020506s
P90 RTT:                      0.328186s
Trimmed Mean Self RTT:        0.166220s
Trimmed Mean Foreign RTT:     0.018372s
Trimmed Mean RTT:             0.092296s
RPM Calculation Statistics:
Total Self Probes:            399
Total Foreign Probes:         1197
Trimmed Self Probes Count:    19 of 399
Trimmed Foreign Probes Count: 59 of 1197
P90 Self RTT:                 0.806346s
P90 Foreign RTT:              0.020808s
P90 RTT:                      0.413577s
Trimmed Mean Self RTT:        0.329165s
Trimmed Mean Foreign RTT:     0.019207s
Trimmed Mean RTT:             0.174186s
RPM: 145 (P90)
RPM: 344 (Single-Sided 5% Trimmed Mean)
========
Upload:
Note: Test did not run to stability (responsiveness), these results are estimates.
Throughput: 38.249 Mbps (4.781 MBps), using 32 parallel connections.
Extended Statistics:
Maximum Segment Size: 1208
Total Bytes Retransmitted: 102005114
Retransmission Ratio: 15.48%
Total Bytes Reordered: 0
Average RTT: 43.75
Unbounded RPM Calculation Statistics:
Total Self Probes:            11091
Total Foreign Probes:         33273
Trimmed Self Probes Count:    554 of 11091
Trimmed Foreign Probes Count: 1663 of 33273
P90 Self RTT:                 1.564135s
P90 Foreign RTT:              0.222845s
P90 RTT:                      0.893490s
Trimmed Mean Self RTT:        0.111610s
Trimmed Mean Foreign RTT:     0.018731s
Trimmed Mean RTT:             0.065171s
RPM Calculation Statistics:
Total Self Probes:            381
Total Foreign Probes:         1143
Trimmed Self Probes Count:    19 of 381
Trimmed Foreign Probes Count: 57 of 1143
P90 Self RTT:                 1.131356s
P90 Foreign RTT:              0.270268s
P90 RTT:                      0.700812s
Trimmed Mean Self RTT:        0.139588s
Trimmed Mean Foreign RTT:     0.020490s
Trimmed Mean RTT:             0.080039s
RPM: 86 (P90)
RPM: 750 (Single-Sided 5% Trimmed Mean)
========
Unbounded Final RPM Calculation stats:
Total Self Probes:            13013
Total Foreign Probes:         39039
Trimmed Self Probes Count:    650 of 13013
Trimmed Foreign Probes Count: 1951 of 39039
P90 Self RTT:                 1.372974s
P90 Foreign RTT:              0.154186s
P90 RTT:                      0.763580s
Trimmed Mean Self RTT:        0.117897s
Trimmed Mean Foreign RTT:     0.018549s
Trimmed Mean RTT:             0.068223s

Unbounded Final RPM: 79 (P90)
Unbounded Final RPM: 879 (Single-Sided 5% Trimmed Mean)

Final RPM Calculation stats:
Total Self Probes:            11490
Total Foreign Probes:         34470
Trimmed Self Probes Count:    574 of 11490
Trimmed Foreign Probes Count: 1723 of 34470
P90 Self RTT:                 1.517121s
P90 Foreign RTT:              0.208764s
P90 RTT:                      0.862942s
Trimmed Mean Self RTT:        0.113263s
Trimmed Mean Foreign RTT:     0.018745s
Trimmed Mean RTT:             0.066004s

Final RPM: 70 (P90) RTT: 0.862942s
Final RPM: 909 (Single-Sided 5% Trimmed Mean) RTT: 0.066004s
Final RPM (Self Only): 40 (P90) RTT: 1.517121s
Final RPM (Self Only): 530 (Single-Sided 5% Trimmed Mean) RTT: 0.113263s
Final RPM (Foreign Only): 287 (P90) RTT: 0.208764s
Final RPM (Foreign Only): 3201 (Single-Sided 5% Trimmed Mean) RTT: 0.018745s
Working-Conditions Effect: Final RPM is     2% of baseline RPM (P90)
Working-Conditions Effect: Final RPM is    27% of baseline RPM (Single-Sided 5% Trimmed Mean)

real 2m56.763s
user 0m16.186s
sys 0m26.759s

Here Foreign means inter-flow measurements, Self meand intra-flow latency.



And of course if you have your own server in the cloud there is always flent's RRUL test...

Regards
	Sebastian

P.S.: For an article about libreqos, maybe the beofre and after of a link of one of the libreqos using ISPs might be the best test?


> 
> 
> On 2024-07-15 09:12, Sebastian Moeller wrote:
>> Hi Dave,
>> 
>> 
>> 
>>> On 15. Jul 2024, at 12:59, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>> 
>>> On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote:
>>> 
>>>> Hi Dave,
>>>> 
>>>> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.
>>>> 
>>>> Regards
>>>> Sebastian
>>>> 
>>> That was a deliberate action on my part: the "after" status shows much smaller throughput numbers, leading observant readers to ask if improving latency didn't destroy one's bandwidth.
>>> 
>> [SM] Ah, I see. But just for our discussion here, how much (potential) capacity did you need to treade in for decent responsiveness?
>> 
>> 
>>> I suspect that the throughputs "with bloat" are greatly exaggerated... from counting the extra data that fills the bloated buffers (;-))
>>> 
>> [SM] Good point... especially for upload tests, I occasionally see numbers which I know for a fact to be impossible (abovre the gross capacity of my access link) and I wonder whether all tests actually do proper reporting of the grand average capacity numbers. This should not be rocket science: take the start time of the first maesasurement flow and the end time of the last measurement flow and divide the total number of payload bytes trasferred by the total time this took. 
>> 
>> 
>>> --dave
>>> 
>>> 
>>> 
>>>> 
>>>>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>>>> 
>>>>> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953
>>>>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
>>>>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project.
>>>>> --dave
>>>>> -- 
>>>>> David Collier-Brown, | Always do right. This will gratify
>>>>> System Programmer and Author | some people and astonish the rest
>>>>> davecb@spamcop.net | -- Mark Twain
>>>>> _______________________________________________
>>>>> Bloat mailing list
>>>>> Bloat@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>>> 
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>> 
>>> -- 
>>> David Collier-Brown, | Always do right. This will gratify
>>> System Programmer and Author | some people and astonish the rest
>>> davecb@spamcop.net | -- Mark Twain
>>> 
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>> 
>> 
> -- 
> David Collier-Brown, | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net | -- Mark Twain

Re: [Bloat] An ACM Queue article about bloat and LibraQoS

[David Collier-Brown]

Many thanks!

--dave

On 2024-07-16 09:48, Sebastian Moeller via Bloat wrote:
> Hi Dave,
>
>
>
>> On 15. Jul 2024, at 16:26, David Collier-Brown <davec-b@rogers.com> wrote:
>>
>> Let's see... Found them!
>> score A+ D %
>> up 9.1 31.8 29%
>> down 163 456.2 36%
>> A+ figures were only 20-30% of the bloated D figure (:-()
>> I showed the article with both graphs to an outside reader and he spotted it instantly, and asked why I was only getting a tiny fraction of the "bloated" throughput.
>> --dave
> [SM] OK, that clearly is a lot of throughput sacrifice... are you sure your link is as optimised as possible? If not I wozuld be happy to help.
> Regarding other tools for measurements that include latency-under-load:
>
> A) Ookla's speedtest.net <http://speedtest.net/> will also report the interquartil means for latency under load
> Example: https://www.speedtest.net/result/16503399049
>
> B) Cloudflare's speed.cloudflare.com <http://speed.cloudflare.com/> will also report some scores
> Here is an article delving into how these scores are derived from the measured data
> https://blog.cloudflare.com/aim-database-for-internet-quality
>
>
> c) Apple's networkQuality tool (on macos/ios) or the open source go version thereof: https://github.com/network-quality/goresponsiveness
> user@laptop goresponsiveness % networkQuality -v -s
> ==== Verbose Results ====
> ---
> Capacity:
> ---
>     Uplink capacity: 39.688 Mbps
>        Accuracy: High
>        Uplink bytes transferred: 106.702 MB
>        Uplink Flow count: 16
>     Downlink capacity: 74.621 Mbps
>        Accuracy: High
>        Downlink bytes transferred: 50.644 MB
>        Downlink Flow count: 6
> ---
> Latency:
> ---
>     Idle Latency:
>        2618 RPM (22.917 milliseconds)
>           Transport: 3178 RPM (18.875 milliseconds)
>           Security: 1832 RPM (32.750 milliseconds)
>           HTTP: 3503 RPM (17.125 milliseconds)
>        Accuracy: High
>     Uplink Responsiveness: Medium
>        296 RPM (202.171 milliseconds)
>           Transport: 3390 RPM (17.699 milliseconds)
>           Security: 2260 RPM (26.538 milliseconds)
>           HTTP: 3276 RPM (18.312 milliseconds)
>           HTTP loaded: 155 RPM (386.579 milliseconds)
>        Accuracy: High
>     Downlink Responsiveness: Medium
>        919 RPM (65.240 milliseconds)
>           Transport: 3564 RPM (16.833 milliseconds)
>           Security: 2488 RPM (24.115 milliseconds)
>           HTTP: 3440 RPM (17.438 milliseconds)
>           HTTP loaded: 430 RPM (139.362 milliseconds)
>        Accuracy: High
> ---
> Protocols Used:
> ---
>      HTTP/2: 100%
> ---
> Transport-layer info:
> ---
>      ECN Disabled: 100%
> ---
> Other Info:
> ---
>     Test Endpoint: defra1-edge-bx-006.aaplimg.com
>     Interface: en7
>     Start: 2024-07-16 15:10:23.240
>     End: 2024-07-16 15:10:51.921
>     OS Version: Version 14.5 (Build 23F79)
>
> Here the RPM for Uplink and Downlink HTTP are most relevant (these measure inter-flow latency increases that is how much the latency of flows stayiung below their capacity share is effected by parallel capacity seeking flows, HTTP loaded measures the self congestion of the load bearing flows)
>
> Here an invocation of the opensource version (using apple's servers):
>
> ./networkQuality --detailed --relative-rpm --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --rpm.timeout 120 --extended-stats --rpm.mnp 32 --logger-filename go_networkQuality_20240716_151413
> 07-16-2024 13:14:13 UTC Go Responsiveness to mensura.cdn-apple.com:443...
> Unbounded Baseline RPM:  2857 (P90)
> Unbounded Baseline RPM:  3322 (Single-Sided 5% Trimmed Mean)
> Baseline RPM:  2827 (P90)
> Baseline RPM:  3327 (Single-Sided 5% Trimmed Mean)
> Results:
> ========
> Download:
> Throughput: 68.989 Mbps (8.624 MBps), using 26 parallel connections.
> Extended Statistics:
> Maximum Segment Size: 1208
> Total Bytes Retransmitted: 0
> Retransmission Ratio: 0.00%
> Total Bytes Reordered: 140449506
> Average RTT: 26.85185185185185
> Unbounded RPM Calculation Statistics:
> Total Self Probes:            1922
> Total Foreign Probes:         5766
> Trimmed Self Probes Count:    96 of 1922
> Trimmed Foreign Probes Count: 288 of 5766
> P90 Self RTT:                 0.635866s
> P90 Foreign RTT:              0.020506s
> P90 RTT:                      0.328186s
> Trimmed Mean Self RTT:        0.166220s
> Trimmed Mean Foreign RTT:     0.018372s
> Trimmed Mean RTT:             0.092296s
> RPM Calculation Statistics:
> Total Self Probes:            399
> Total Foreign Probes:         1197
> Trimmed Self Probes Count:    19 of 399
> Trimmed Foreign Probes Count: 59 of 1197
> P90 Self RTT:                 0.806346s
> P90 Foreign RTT:              0.020808s
> P90 RTT:                      0.413577s
> Trimmed Mean Self RTT:        0.329165s
> Trimmed Mean Foreign RTT:     0.019207s
> Trimmed Mean RTT:             0.174186s
> RPM: 145 (P90)
> RPM: 344 (Single-Sided 5% Trimmed Mean)
> ========
> Upload:
> Note: Test did not run to stability (responsiveness), these results are estimates.
> Throughput: 38.249 Mbps (4.781 MBps), using 32 parallel connections.
> Extended Statistics:
> Maximum Segment Size: 1208
> Total Bytes Retransmitted: 102005114
> Retransmission Ratio: 15.48%
> Total Bytes Reordered: 0
> Average RTT: 43.75
> Unbounded RPM Calculation Statistics:
> Total Self Probes:            11091
> Total Foreign Probes:         33273
> Trimmed Self Probes Count:    554 of 11091
> Trimmed Foreign Probes Count: 1663 of 33273
> P90 Self RTT:                 1.564135s
> P90 Foreign RTT:              0.222845s
> P90 RTT:                      0.893490s
> Trimmed Mean Self RTT:        0.111610s
> Trimmed Mean Foreign RTT:     0.018731s
> Trimmed Mean RTT:             0.065171s
> RPM Calculation Statistics:
> Total Self Probes:            381
> Total Foreign Probes:         1143
> Trimmed Self Probes Count:    19 of 381
> Trimmed Foreign Probes Count: 57 of 1143
> P90 Self RTT:                 1.131356s
> P90 Foreign RTT:              0.270268s
> P90 RTT:                      0.700812s
> Trimmed Mean Self RTT:        0.139588s
> Trimmed Mean Foreign RTT:     0.020490s
> Trimmed Mean RTT:             0.080039s
> RPM: 86 (P90)
> RPM: 750 (Single-Sided 5% Trimmed Mean)
> ========
> Unbounded Final RPM Calculation stats:
> Total Self Probes:            13013
> Total Foreign Probes:         39039
> Trimmed Self Probes Count:    650 of 13013
> Trimmed Foreign Probes Count: 1951 of 39039
> P90 Self RTT:                 1.372974s
> P90 Foreign RTT:              0.154186s
> P90 RTT:                      0.763580s
> Trimmed Mean Self RTT:        0.117897s
> Trimmed Mean Foreign RTT:     0.018549s
> Trimmed Mean RTT:             0.068223s
>
> Unbounded Final RPM: 79 (P90)
> Unbounded Final RPM: 879 (Single-Sided 5% Trimmed Mean)
>
> Final RPM Calculation stats:
> Total Self Probes:            11490
> Total Foreign Probes:         34470
> Trimmed Self Probes Count:    574 of 11490
> Trimmed Foreign Probes Count: 1723 of 34470
> P90 Self RTT:                 1.517121s
> P90 Foreign RTT:              0.208764s
> P90 RTT:                      0.862942s
> Trimmed Mean Self RTT:        0.113263s
> Trimmed Mean Foreign RTT:     0.018745s
> Trimmed Mean RTT:             0.066004s
>
> Final RPM: 70 (P90) RTT: 0.862942s
> Final RPM: 909 (Single-Sided 5% Trimmed Mean) RTT: 0.066004s
> Final RPM (Self Only): 40 (P90) RTT: 1.517121s
> Final RPM (Self Only): 530 (Single-Sided 5% Trimmed Mean) RTT: 0.113263s
> Final RPM (Foreign Only): 287 (P90) RTT: 0.208764s
> Final RPM (Foreign Only): 3201 (Single-Sided 5% Trimmed Mean) RTT: 0.018745s
> Working-Conditions Effect: Final RPM is     2% of baseline RPM (P90)
> Working-Conditions Effect: Final RPM is    27% of baseline RPM (Single-Sided 5% Trimmed Mean)
>
> real 2m56.763s
> user 0m16.186s
> sys 0m26.759s
>
> Here Foreign means inter-flow measurements, Self meand intra-flow latency.
>
>
>
> And of course if you have your own server in the cloud there is always flent's RRUL test...
>
> Regards
> 	Sebastian
>
> P.S.: For an article about libreqos, maybe the beofre and after of a link of one of the libreqos using ISPs might be the best test?
>
>
>>
>> On 2024-07-15 09:12, Sebastian Moeller wrote:
>>> Hi Dave,
>>>
>>>
>>>
>>>> On 15. Jul 2024, at 12:59, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>>>
>>>> On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote:
>>>>
>>>>> Hi Dave,
>>>>>
>>>>> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only.
>>>>>
>>>>> Regards
>>>>> Sebastian
>>>>>
>>>> That was a deliberate action on my part: the "after" status shows much smaller throughput numbers, leading observant readers to ask if improving latency didn't destroy one's bandwidth.
>>>>
>>> [SM] Ah, I see. But just for our discussion here, how much (potential) capacity did you need to treade in for decent responsiveness?
>>>
>>>
>>>> I suspect that the throughputs "with bloat" are greatly exaggerated... from counting the extra data that fills the bloated buffers (;-))
>>>>
>>> [SM] Good point... especially for upload tests, I occasionally see numbers which I know for a fact to be impossible (abovre the gross capacity of my access link) and I wonder whether all tests actually do proper reporting of the grand average capacity numbers. This should not be rocket science: take the start time of the first maesasurement flow and the end time of the last measurement flow and divide the total number of payload bytes trasferred by the total time this took.
>>>
>>>
>>>> --dave
>>>>
>>>>
>>>>
>>>>>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat <bloat@lists.bufferbloat.net> wrote:
>>>>>>
>>>>>> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953
>>>>>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem."
>>>>>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project.
>>>>>> --dave
>>>>>> -- 
>>>>>> David Collier-Brown, | Always do right. This will gratify
>>>>>> System Programmer and Author | some people and astonish the rest
>>>>>> davecb@spamcop.net | -- Mark Twain
>>>>>> _______________________________________________
>>>>>> Bloat mailing list
>>>>>> Bloat@lists.bufferbloat.net
>>>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>>>>
>>>>> _______________________________________________
>>>>> Bloat mailing list
>>>>> Bloat@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>>>
>>>> -- 
>>>> David Collier-Brown, | Always do right. This will gratify
>>>> System Programmer and Author | some people and astonish the rest
>>>> davecb@spamcop.net | -- Mark Twain
>>>>
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>>
>> -- 
>> David Collier-Brown, | Always do right. This will gratify
>> System Programmer and Author | some people and astonish the rest
>> davecb@spamcop.net | -- Mark Twain
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |              -- Mark Twain