From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sonic309-16.consmr.mail.bf2.yahoo.com (sonic309-16.consmr.mail.bf2.yahoo.com [74.6.129.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EE4563B2A4 for ; Tue, 16 Jul 2024 10:25:58 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rogers.com; s=s2048; t=1721139958; bh=ED9O5+DzSgOyLdGCPuGdwfAII4BBo/9pcZx06S6fkhg=; h=Date:Subject:To:References:From:In-Reply-To:From:Subject:Reply-To; b=Rff0mrp5Q3nK6pcuwqsepL+fslocenBiryQOLVYNFuzApXS7AzQYhxNdcJWGSMM+3eMADS3pS+wRn5phcsPS2kDqJSWYO1DNMr0DsxMfZCHofq840QfM423GPQOFI1ZYK2Aoh8tgvdMRWGVjTu1ULoFEJR3kmQoJcr1p7vlb/R7pat4p+1kdNL2l4ATwqrQBuhiuAdrug2x3Z/yn/iT7/rWCMRRrNQ0P5RnApjAIUdjm7i2/yUadtaG2entcKMni8A65QroQ972sfapxSL9cIktmrJprGDkNJscp1FPawbISJwczMyo5oNd+8AlJJZSV/UJ6Z8XxnIqDQWLQRSbOVA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1721139958; bh=b4qsbwqD3x1mULUcgMyNOOGXWAkcGZeVt7pGEGXdK/Z=; h=X-Sonic-MF:Date:Subject:To:From:From:Subject; b=Ry3/mM3XNFMcmp/RFVQPYLdRht3a5GSlsgtLuzL6L9R1n0346FnyfPPttmfvkSlUj41ba4+Fdo6PcKf7Qwk5+7dus/NHyq58bPBU/QZoUrGMcjy56DOQ27V6CuTEkpLuwOm3EojNikfepmejCVmUEWMuoy+TjPGIUS3cVOleU25nRFTKUxbh9Xenbo0JpfdJ3is0TYQJLEZlUW0mfOUEWFXbdEEpWsT4wDLwvNyzmAIliroI8E7JWyeNWMlG7n4N2pBs0h1J0iAiJIQoeyFVLSteTpHz3TnnDCSpFC4QhpYyQRHsKCGGoM1dxfqu/fgGnJzLYIIdwojIt1wtPvpvJA== X-YMail-OSG: .vyEFZ8VM1k7SPWD0EEIylFsi9TrHOJpoc6wY5d3Yb.ydaOtv4LautHo0woc8OH PG0xVAbVlAhkrfNksNsPOnIYvQVIdva9niivxfNh53lW8Gs3sUtkM2EuvJWV8r6R8Cp6xsCglfWs kRnGERkhVdouybfGRowOyvP2K8aF9BV3fU.9_gluYuEQYVDG_4qaajNYsW19843S36mzwSjbXvLP wgcIQ54LV5EALHb2GSoA5QdhcnFfUa3f2vWk1iCMdM2JQw511yNTJmszNjOWpOFc7VYRi90oLRsM xMX_SwBNx02U3js53uaL2nDI_xfELJIZsRYA3wl1r0UpuTuM3QtpD1F0cjSoY_kyG5MlsLNqw1Fb dxpRYeAPIhYY4_6aNlqMQAcmqGdcu4yKfKNu4sBfN7Hp32FvC6y4IWipbBjSG7LRaoD0EqufBevR nUIPIfq4P5ooxdWgIXKH8vNmZgi8md7ZcnIEtcnOiXa73sw4eaGOI0ikjCjbJ_wzQXAhjKvFbr_a P8p77FHtnYsj2lcSV6v5w1HK8I8YDfRB3o13sLTbmxj0oV2fqinA2WBnFHD6gbImCzu0iEJdRPNP NIjvxAitCKkF6PByEFu7VCN.o1z.GBs6ZoZSIcCX476guMls4NEeQeU0KTIFBVMwQ2PQSIqlutlv 7JJWX4.dZfifVTgepG4B_uEBB9W9iF.Aw7H30y9glM.RePgQej1eiBWs2hveJocApPZK0a2BOiYC tHzzOeXaRsCu.NxjzW4.lc_KyrFRf4PMfOBp3DgTJn5mAH2o_yVzrXQO6JmmQ5hhBoFPaTfA724g J4y.Kuj6Pcyemf2EbQg1YSYRJ0cs90jUWwwM1V.6F_JcoGXwucQzDaCDWDQM8xT3aeNHwZGBqB0g 1ErYWQYDZEdlLm6FaIx_U1MYeopD9m86Gnq5efeDWyGXUaAb6vPgj2S6Mpv59kogUd0z1nS1zTK5 4Q0fiCyEbGnLn4KZbtg1lIg6s9AxgSzQiOPAthAaSZn_rnfgGt5b0JSyTXqcqhoubxghc_Wg3YtT XBINnFdjhKHF93YPVXLcQnwqGD.JwQCHBhl9x9w9QZMlvhtjUefyzD_aRNLb5K5ckkKLsYMAbju2 Fk_sgJL1wZVtLYURT1wn5EFaFirsBCZfuP.TtJI6tG4Qxc4hWoSFXJ440giGsGHLXt2czmIwwPSq 7OhmWbFi2yvpO1QzIkYAWz_9ctnzX80RcxfIF03TG.guXtulzzu3MJS04kSO5gRRWYpshw6tk8o9 YF.7NM.xu2WGWXc_LjUvzTF9gm52fkmzzOXjgmjKlhrgjCoqultNk0dG_ergM0poZKtBPTm4_Z9D UNm7gVAF8D7uoVxGAW7trOMbPdccMmVwYfkQhNA1EoimeD6eiRAHFjQNhn0E_GbHalQbC0t67yrl fD3eiqMIAgyEJ0flQXEk5wqxI1fQf.LDtiLvxNMSOu1o7S79Ktmyk6HKQNiBmuF58lLi45.6sRkQ J2C8teTj0.pBB4oQpYh7nh1YybQl7o31i0HAFNvX5Kc9wisAqhBNKtO9A357yxBpwcT3rGD_DH5v sozm9ZqCQ4OVYNDNGt9M74iwXGVH12bJivQo7vjy4566qI2.WJhPe7ylIBv_AE5NaQgRYRMO_KDi 7jWIIsPlvgDNcf3SmfgSmsrgVFkJdcZ83F1yfZAs40G51bLlPH1SOBKl57d5887C33RJ.kJeD58a .LyNPke5stBNOXz5J2su4Qf.RFPJfdbHKgahNFxKYEbuoPzAygNtm8th29fRD1SQKu1mUgD8RURo o9rPNpM1r_zpLf6IYUB5FIjEdH_.AH1lKylVrY9y_zC_UC2CHhlm.0FB7ARfyANcxv8GpFsuJi5_ HCl59bhcU8v2UnaKM5EDentP1NKKUDeYCHRJQqkeM4yNV6_53Z5cPZT1OdJEQe6dejcTFQ7ulaCr 6fZUjzL.fAtfehVJo4zCp_p6bTYsS4jD6zBiFf9zJrLWuQeyoiiVBb8tgyd2DCD210SCehSA6YE0 b4FShq4v3Z_hlz0b.ID.cMVs4Fx__LSRKuPt700Ku0n3paqBmWLkT3wp5SySIP6k._3srcp1T9uZ 1Bv4nJRstnh948m.Fw6lWwRciSn3RaP4cvMtljT4Aj1HUolnGT8o.a7hGk.rdXYYIM0Gb3sba8bq IlSYErCcEWunQX6mCB2n6xuibwFia3H1emnqA6G2NCIQZqYlzabe..omR_mPGEFYKgNKLCe4Fksz zE.y1WTGm_gtmFSCl7WboK6OTVTysD7jREvoBAtHA.csNCG2MNH5J3zWu_gCHs7_1eY2mV.nPMg- - X-Sonic-MF: X-Sonic-ID: 37e44cd2-e9a2-4f47-b961-74a0526833fd Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.bf2.yahoo.com with HTTP; Tue, 16 Jul 2024 14:25:58 +0000 Received: by hermes--production-bf1-7bd4f49c5c-2s5sw (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 6432c71a2724e86343687ab73b5005c9; Tue, 16 Jul 2024 14:25:55 +0000 (UTC) Message-ID: Date: Tue, 16 Jul 2024 10:25:54 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: bloat@lists.bufferbloat.net References: <9B529996-9FF0-4487-8F74-B01230E6FCA1@viagenie.ca> <11567.1720044376@obiwan.sandelman.ca> <8d89d7ec-558e-4498-9457-3c6dd90eb03f@rogers.com> <8FBDAF2F-DF18-4FFD-A589-EF2323B6820A@gmx.de> <868EB2E2-3E7F-4B04-8EB0-D81D22A73CD6@gmx.de> Content-Language: en-US From: David Collier-Brown In-Reply-To: <868EB2E2-3E7F-4B04-8EB0-D81D22A73CD6@gmx.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailer: WebService/1.1.22501 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo Subject: Re: [Bloat] An ACM Queue article about bloat and LibraQoS X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2024 14:25:59 -0000 Many thanks! --dave On 2024-07-16 09:48, Sebastian Moeller via Bloat wrote: > Hi Dave, > > > >> On 15. Jul 2024, at 16:26, David Collier-Brown wrote: >> >> Let's see... Found them! >> score A+ D % >> up 9.1 31.8 29% >> down 163 456.2 36% >> A+ figures were only 20-30% of the bloated D figure (:-() >> I showed the article with both graphs to an outside reader and he spotted it instantly, and asked why I was only getting a tiny fraction of the "bloated" throughput. >> --dave > [SM] OK, that clearly is a lot of throughput sacrifice... are you sure your link is as optimised as possible? If not I wozuld be happy to help. > Regarding other tools for measurements that include latency-under-load: > > A) Ookla's speedtest.net will also report the interquartil means for latency under load > Example: https://www.speedtest.net/result/16503399049 > > B) Cloudflare's speed.cloudflare.com will also report some scores > Here is an article delving into how these scores are derived from the measured data > https://blog.cloudflare.com/aim-database-for-internet-quality > > > c) Apple's networkQuality tool (on macos/ios) or the open source go version thereof: https://github.com/network-quality/goresponsiveness > user@laptop goresponsiveness % networkQuality -v -s > ==== Verbose Results ==== > --- > Capacity: > --- > Uplink capacity: 39.688 Mbps > Accuracy: High > Uplink bytes transferred: 106.702 MB > Uplink Flow count: 16 > Downlink capacity: 74.621 Mbps > Accuracy: High > Downlink bytes transferred: 50.644 MB > Downlink Flow count: 6 > --- > Latency: > --- > Idle Latency: > 2618 RPM (22.917 milliseconds) > Transport: 3178 RPM (18.875 milliseconds) > Security: 1832 RPM (32.750 milliseconds) > HTTP: 3503 RPM (17.125 milliseconds) > Accuracy: High > Uplink Responsiveness: Medium > 296 RPM (202.171 milliseconds) > Transport: 3390 RPM (17.699 milliseconds) > Security: 2260 RPM (26.538 milliseconds) > HTTP: 3276 RPM (18.312 milliseconds) > HTTP loaded: 155 RPM (386.579 milliseconds) > Accuracy: High > Downlink Responsiveness: Medium > 919 RPM (65.240 milliseconds) > Transport: 3564 RPM (16.833 milliseconds) > Security: 2488 RPM (24.115 milliseconds) > HTTP: 3440 RPM (17.438 milliseconds) > HTTP loaded: 430 RPM (139.362 milliseconds) > Accuracy: High > --- > Protocols Used: > --- > HTTP/2: 100% > --- > Transport-layer info: > --- > ECN Disabled: 100% > --- > Other Info: > --- > Test Endpoint: defra1-edge-bx-006.aaplimg.com > Interface: en7 > Start: 2024-07-16 15:10:23.240 > End: 2024-07-16 15:10:51.921 > OS Version: Version 14.5 (Build 23F79) > > Here the RPM for Uplink and Downlink HTTP are most relevant (these measure inter-flow latency increases that is how much the latency of flows stayiung below their capacity share is effected by parallel capacity seeking flows, HTTP loaded measures the self congestion of the load bearing flows) > > Here an invocation of the opensource version (using apple's servers): > > ./networkQuality --detailed --relative-rpm --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --rpm.timeout 120 --extended-stats --rpm.mnp 32 --logger-filename go_networkQuality_20240716_151413 > 07-16-2024 13:14:13 UTC Go Responsiveness to mensura.cdn-apple.com:443... > Unbounded Baseline RPM: 2857 (P90) > Unbounded Baseline RPM: 3322 (Single-Sided 5% Trimmed Mean) > Baseline RPM: 2827 (P90) > Baseline RPM: 3327 (Single-Sided 5% Trimmed Mean) > Results: > ======== > Download: > Throughput: 68.989 Mbps (8.624 MBps), using 26 parallel connections. > Extended Statistics: > Maximum Segment Size: 1208 > Total Bytes Retransmitted: 0 > Retransmission Ratio: 0.00% > Total Bytes Reordered: 140449506 > Average RTT: 26.85185185185185 > Unbounded RPM Calculation Statistics: > Total Self Probes: 1922 > Total Foreign Probes: 5766 > Trimmed Self Probes Count: 96 of 1922 > Trimmed Foreign Probes Count: 288 of 5766 > P90 Self RTT: 0.635866s > P90 Foreign RTT: 0.020506s > P90 RTT: 0.328186s > Trimmed Mean Self RTT: 0.166220s > Trimmed Mean Foreign RTT: 0.018372s > Trimmed Mean RTT: 0.092296s > RPM Calculation Statistics: > Total Self Probes: 399 > Total Foreign Probes: 1197 > Trimmed Self Probes Count: 19 of 399 > Trimmed Foreign Probes Count: 59 of 1197 > P90 Self RTT: 0.806346s > P90 Foreign RTT: 0.020808s > P90 RTT: 0.413577s > Trimmed Mean Self RTT: 0.329165s > Trimmed Mean Foreign RTT: 0.019207s > Trimmed Mean RTT: 0.174186s > RPM: 145 (P90) > RPM: 344 (Single-Sided 5% Trimmed Mean) > ======== > Upload: > Note: Test did not run to stability (responsiveness), these results are estimates. > Throughput: 38.249 Mbps (4.781 MBps), using 32 parallel connections. > Extended Statistics: > Maximum Segment Size: 1208 > Total Bytes Retransmitted: 102005114 > Retransmission Ratio: 15.48% > Total Bytes Reordered: 0 > Average RTT: 43.75 > Unbounded RPM Calculation Statistics: > Total Self Probes: 11091 > Total Foreign Probes: 33273 > Trimmed Self Probes Count: 554 of 11091 > Trimmed Foreign Probes Count: 1663 of 33273 > P90 Self RTT: 1.564135s > P90 Foreign RTT: 0.222845s > P90 RTT: 0.893490s > Trimmed Mean Self RTT: 0.111610s > Trimmed Mean Foreign RTT: 0.018731s > Trimmed Mean RTT: 0.065171s > RPM Calculation Statistics: > Total Self Probes: 381 > Total Foreign Probes: 1143 > Trimmed Self Probes Count: 19 of 381 > Trimmed Foreign Probes Count: 57 of 1143 > P90 Self RTT: 1.131356s > P90 Foreign RTT: 0.270268s > P90 RTT: 0.700812s > Trimmed Mean Self RTT: 0.139588s > Trimmed Mean Foreign RTT: 0.020490s > Trimmed Mean RTT: 0.080039s > RPM: 86 (P90) > RPM: 750 (Single-Sided 5% Trimmed Mean) > ======== > Unbounded Final RPM Calculation stats: > Total Self Probes: 13013 > Total Foreign Probes: 39039 > Trimmed Self Probes Count: 650 of 13013 > Trimmed Foreign Probes Count: 1951 of 39039 > P90 Self RTT: 1.372974s > P90 Foreign RTT: 0.154186s > P90 RTT: 0.763580s > Trimmed Mean Self RTT: 0.117897s > Trimmed Mean Foreign RTT: 0.018549s > Trimmed Mean RTT: 0.068223s > > Unbounded Final RPM: 79 (P90) > Unbounded Final RPM: 879 (Single-Sided 5% Trimmed Mean) > > Final RPM Calculation stats: > Total Self Probes: 11490 > Total Foreign Probes: 34470 > Trimmed Self Probes Count: 574 of 11490 > Trimmed Foreign Probes Count: 1723 of 34470 > P90 Self RTT: 1.517121s > P90 Foreign RTT: 0.208764s > P90 RTT: 0.862942s > Trimmed Mean Self RTT: 0.113263s > Trimmed Mean Foreign RTT: 0.018745s > Trimmed Mean RTT: 0.066004s > > Final RPM: 70 (P90) RTT: 0.862942s > Final RPM: 909 (Single-Sided 5% Trimmed Mean) RTT: 0.066004s > Final RPM (Self Only): 40 (P90) RTT: 1.517121s > Final RPM (Self Only): 530 (Single-Sided 5% Trimmed Mean) RTT: 0.113263s > Final RPM (Foreign Only): 287 (P90) RTT: 0.208764s > Final RPM (Foreign Only): 3201 (Single-Sided 5% Trimmed Mean) RTT: 0.018745s > Working-Conditions Effect: Final RPM is 2% of baseline RPM (P90) > Working-Conditions Effect: Final RPM is 27% of baseline RPM (Single-Sided 5% Trimmed Mean) > > real 2m56.763s > user 0m16.186s > sys 0m26.759s > > Here Foreign means inter-flow measurements, Self meand intra-flow latency. > > > > And of course if you have your own server in the cloud there is always flent's RRUL test... > > Regards > Sebastian > > P.S.: For an article about libreqos, maybe the beofre and after of a link of one of the libreqos using ISPs might be the best test? > > >> >> On 2024-07-15 09:12, Sebastian Moeller wrote: >>> Hi Dave, >>> >>> >>> >>>> On 15. Jul 2024, at 12:59, David Collier-Brown via Bloat wrote: >>>> >>>> On 2024-07-15 05:18, Sebastian Moeller via Bloat wrote: >>>> >>>>> Hi Dave, >>>>> >>>>> nice! I only would have wished for the A-grade waveform result as well, it is a bit of a downer that the last figure show the unpleasant 'before' status only. >>>>> >>>>> Regards >>>>> Sebastian >>>>> >>>> That was a deliberate action on my part: the "after" status shows much smaller throughput numbers, leading observant readers to ask if improving latency didn't destroy one's bandwidth. >>>> >>> [SM] Ah, I see. But just for our discussion here, how much (potential) capacity did you need to treade in for decent responsiveness? >>> >>> >>>> I suspect that the throughputs "with bloat" are greatly exaggerated... from counting the extra data that fills the bloated buffers (;-)) >>>> >>> [SM] Good point... especially for upload tests, I occasionally see numbers which I know for a fact to be impossible (abovre the gross capacity of my access link) and I wonder whether all tests actually do proper reporting of the grand average capacity numbers. This should not be rocket science: take the start time of the first maesasurement flow and the end time of the last measurement flow and divide the total number of payload bytes trasferred by the total time this took. >>> >>> >>>> --dave >>>> >>>> >>>> >>>>>> On 14. Jul 2024, at 23:30, David Collier-Brown via Bloat wrote: >>>>>> >>>>>> I have a new article on ACM Queue, at https://dl.acm.org/doi/pdf/10.1145/3674953 >>>>>> "If you're an ISP and all your customers hate you, take heart. This is now a solvable problem." >>>>>> It's part of a tongue-in-cheek series called "You Don't Know Jack", and it's about Bufferbloat and the LibreQoS project. >>>>>> --dave >>>>>> -- >>>>>> David Collier-Brown, | Always do right. This will gratify >>>>>> System Programmer and Author | some people and astonish the rest >>>>>> davecb@spamcop.net | -- Mark Twain >>>>>> _______________________________________________ >>>>>> Bloat mailing list >>>>>> Bloat@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/bloat >>>>>> >>>>> _______________________________________________ >>>>> Bloat mailing list >>>>> Bloat@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/bloat >>>>> >>>> -- >>>> David Collier-Brown, | Always do right. This will gratify >>>> System Programmer and Author | some people and astonish the rest >>>> davecb@spamcop.net | -- Mark Twain >>>> >>>> _______________________________________________ >>>> Bloat mailing list >>>> Bloat@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/bloat >>>> >> -- >> David Collier-Brown, | Always do right. This will gratify >> System Programmer and Author | some people and astonish the rest >> davecb@spamcop.net | -- Mark Twain > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain