From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4DF413B29E; Mon, 24 Oct 2022 20:12:40 -0400 (EDT) Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id 8B7C01B252; Mon, 24 Oct 2022 17:12:39 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com 8B7C01B252 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1666656759; bh=T/V8At4FlcISbusw5pjT7lWklm2yPFGKcc3IcVYhxss=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=RzpoRUFu5ajcfff/mqjE6zjTrYT7D/A9YTQhSfpZiho26Wq48dvpc/l+0R2K4XPvH UBuQkxx15+KJAIcKct9bt7kIuGYGCfjvAhfiDQkDQ/hIlVuuYPnwncf/2Nx3rtae5B BZBoQ6KeWoX9pcGwbFjxabj8iv/2CisqZM3zcWCc= MIME-Version: 1.0 Date: Mon, 24 Oct 2022 17:12:39 -0700 From: rjmcmahon To: rjmcmahon Cc: Christoph Paasch , Rpm , tsvwg IETF list , IETF IPPM WG , Dave Taht via Starlink , Glenn Fishbine , Measurement Analysis and Tools Working Group , discuss In-Reply-To: References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> <26F033E5-2490-414E-8DFF-4ACD27B74075@apple.com> Message-ID: <4cd5e118b9d260fea3d694a83f08aed3@rjmcmahon.com> X-Sender: rjmcmahon@rjmcmahon.com Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Rpm] [Starlink] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2022 00:12:40 -0000 Sorry, the RPi4 rebooted and this test was over wired ethernet. But the point is the same. Bob > Be careful about assuming network loads always worsen latency over > networks. Below is an example over WiFi with a rasberry pi over a > Netgear Nighthawk RAXE500 to a 1G wired linux host, without a load > then with an upstream load. I've noticed similar with some hardware > forwarding planes where a busier AP outperforms, in terms of latency, > a lightly loaded one. (Note: Iperf 2 uses responses per second (RPS) > as the SI Units of time is the second.) > > rjmcmahon@ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i > 1 --bounceback > ------------------------------------------------------------ > Client connecting to 192.168.1.69, TCP port 5001 with pid 4148 (1 > flows) > Write buffer size: 100 Byte > Bursting: 100 Byte writes 10 times every 1.00 second(s) > Bounce-back test (size= 100 Byte) (server hold req=0 usecs & > tcp_quickack) > TOS set to 0x0 and nodelay (Nagle off) > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 1] local 192.168.1.40%eth0 port 53750 connected with 192.168.1.69 > port 5001 (bb w/quickack len/hold=100/0) (sock=3) > (icwnd/mss/irtt=14/1448/341) (ct=0.44 ms) on 2022-10-25 00:00:48 (UTC) > [ ID] Interval Transfer Bandwidth BB > cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS > [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.340/0.206/0.852/0.188 ms 0 14K/220 us 2941 rps > [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.481/0.362/0.572/0.057 ms 0 14K/327 us 2078 rps > [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.471/0.344/0.694/0.089 ms 0 14K/340 us 2123 rps > [ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.406/0.330/0.595/0.072 ms 0 14K/318 us 2465 rps > [ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.471/0.405/0.603/0.057 ms 0 14K/348 us 2124 rps > [ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.428/0.355/0.641/0.079 ms 0 14K/324 us 2337 rps > [ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.429/0.329/0.616/0.086 ms 0 14K/306 us 2329 rps > [ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.445/0.325/0.673/0.092 ms 0 14K/321 us 2248 rps > [ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.423/0.348/0.604/0.074 ms 0 14K/299 us 2366 rps > [ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.463/0.369/0.729/0.108 ms 0 14K/306 us 2159 rps > [ 1] 0.00-10.01 sec 19.7 KBytes 16.1 Kbits/sec > 101=0.438/0.206/0.852/0.102 ms 0 14K/1192 us 2285 rps > [ 1] 0.00-10.01 sec BB8-PDF: > bin(w=100us):cnt(101)=3:5,4:30,5:48,6:9,7:7,8:1,9:1 > (5.00/95.00/99.7%=4/7/9,Outliers=0,obl/obu=0/0) > > > rjmcmahon@ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i > 1 --bounceback --bounceback-congest=up,1 > ------------------------------------------------------------ > Client connecting to 192.168.1.69, TCP port 5001 with pid 4152 (1 > flows) > Write buffer size: 100 Byte > Bursting: 100 Byte writes 10 times every 1.00 second(s) > Bounce-back test (size= 100 Byte) (server hold req=0 usecs & > tcp_quickack) > TOS set to 0x0 and nodelay (Nagle off) > TCP window size: 16.0 KByte (default) > ------------------------------------------------------------ > [ 2] local 192.168.1.40%eth0 port 50462 connected with 192.168.1.69 > port 5001 (sock=4) (qack) (icwnd/mss/irtt=14/1448/475) (ct=0.63 ms) on > 2022-10-25 00:01:07 (UTC) > [ 1] local 192.168.1.40%eth0 port 50472 connected with 192.168.1.69 > port 5001 (bb w/quickack len/hold=100/0) (sock=3) > (icwnd/mss/irtt=14/1448/375) (ct=0.67 ms) on 2022-10-25 00:01:07 (UTC) > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT(var) NetPwr > [ 2] 0.00-1.00 sec 3.73 MBytes 31.3 Mbits/sec 39069/0 0 > 59K/133(1) us 29376 > [ ID] Interval Transfer Bandwidth BB > cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS > [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.282/0.178/0.887/0.214 ms 0 14K/191 us 3547 rps > [ 2] 1.00-2.00 sec 3.77 MBytes 31.6 Mbits/sec 39512/0 0 > 59K/133(1) us 29708 > [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.196/0.149/0.240/0.024 ms 0 14K/161 us 5115 rps > [ 2] 2.00-3.00 sec 3.77 MBytes 31.6 Mbits/sec 39558/0 0 > 59K/125(8) us 31646 > [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.163/0.134/0.192/0.018 ms 0 14K/136 us 6124 rps > [ 2] 3.00-4.00 sec 3.77 MBytes 31.6 Mbits/sec 39560/0 0 > 59K/133(1) us 29744 > [ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.171/0.149/0.216/0.019 ms 0 14K/133 us 5838 rps > [ 2] 4.00-5.00 sec 3.76 MBytes 31.6 Mbits/sec 39460/0 0 > 59K/131(2) us 30122 > [ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.188/0.131/0.242/0.029 ms 0 14K/143 us 5308 rps > [ 2] 5.00-6.00 sec 3.77 MBytes 31.6 Mbits/sec 39545/0 0 > 59K/133(0) us 29733 > [ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.197/0.147/0.255/0.031 ms 0 14K/149 us 5079 rps > [ 2] 6.00-7.00 sec 3.78 MBytes 31.7 Mbits/sec 39631/0 0 > 59K/133(1) us 29798 > [ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.196/0.146/0.229/0.025 ms 0 14K/151 us 5102 rps > [ 2] 7.00-8.00 sec 3.77 MBytes 31.6 Mbits/sec 39497/0 0 > 59K/133(0) us 29697 > [ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.190/0.147/0.225/0.028 ms 0 14K/155 us 5260 rps > [ 2] 8.00-9.00 sec 3.77 MBytes 31.6 Mbits/sec 39533/0 0 > 59K/126(4) us 31375 > [ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.185/0.158/0.208/0.017 ms 0 14K/148 us 5414 rps > [ 2] 9.00-10.00 sec 3.77 MBytes 31.6 Mbits/sec 39519/0 0 > 59K/133(1) us 29714 > [ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec > 10=0.165/0.134/0.232/0.028 ms 0 14K/131 us 6064 rps > [ 2] 0.00-10.01 sec 37.7 MBytes 31.6 Mbits/sec 394886/0 0 > 59K/1430(2595) us 2759 > [ 1] 0.00-10.01 sec 19.7 KBytes 16.1 Kbits/sec > 101=0.194/0.131/0.887/0.075 ms 0 14K/1385 us 5167 rps > [ 1] 0.00-10.01 sec BB8-PDF: bin(w=100us):cnt(101)=2:69,3:31,9:1 > (5.00/95.00/99.7%=2/3/9,Outliers=0,obl/obu=0/0) > [ CT] final connect times (min/avg/max/stdev) = > 0.627/0.647/0.666/27.577 ms (tot/err) = 2/0 > > > Bob >>> On Oct 24, 2022, at 1:57 PM, Sebastian Moeller >>> wrote: >>> Hi Christoph >>> >>> On Oct 24, 2022, at 22:08, Christoph Paasch >>> wrote: >>> >>> Hello Sebastian, >>> >>> On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink >>> wrote: >>> >>> Hi Glenn, >>> >>> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm >>> wrote: >>> >>> As a classic died in the wool empiricist, granted that you can >>> identify "misery" factors, given a population of 1,000 users, how do >>> you propose deriving a misery index for that population? >>> >>> We can measure download, upload, ping, jitter pretty much without >>> user intervention. For the measurements you hypothesize, how you >>> you automatically extract those indecies without subjective user >>> contamination. >>> >>> I.e. my download speed sucks. Measure the download speed. >>> >>> My isp doesn't fix my problem. Measure what? How? >>> >>> Human survey technology is 70+ years old and it still has problems >>> figuring out how to correlate opinion with fact. >>> >>> Without an objective measurement scheme that doesn't require human >>> interaction, the misery index is a cool hypothesis with no way to >>> link to actual data. What objective measurements can be made? >>> Answer that and the index becomes useful. Otherwise it's just >>> consumer whining. >>> >>> Not trying to be combative here, in fact I like the concept you >>> support, but I'm hard pressed to see how the concept can lead to >>> data, and the data lead to policy proposals. >>> >>> [SM] So it seems that outside of seemingly simple to test >>> throughput numbers*, the next most important quality number (or the >>> most important depending on subjective ranking) is how does latency >>> change under "load". Absolute latency is also important albeit >>> static high latency can be worked around within limits so the change >>> under load seems more relevant. >>> All of flent's RRUL test, apple's networkQuality/RPM, and iperf2's >>> bounceback test offer methods to asses latency change under load**, >>> as do waveforms bufferbloat tests and even to a degree Ookla's >>> speedtest.net. IMHO something like latency increase under load or >>> apple's responsiveness measure RPM (basically the inverse of the >>> latency under load calculated on a per minute basis, so it scales in >>> the typical higher numbers are better way, unlike raw latency under >>> load numbers where smaller is better). >>> IMHO what networkQuality is missing ATM is to measure and report >>> the unloaded RPM as well as the loaded the first gives a measure >>> over the static latency the second over how well things keep working >>> if capacity gets tight. They report the base RTT which can be >>> converted to RPM. As an example: >>> >>> macbook:~ user$ networkQuality -v >>> ==== SUMMARY ==== >>> >>> Upload capacity: 24.341 Mbps >>> Download capacity: 91.951 Mbps >>> Upload flows: 20 >>> Download flows: 16 >>> Responsiveness: High (2123 RPM) >>> Base RTT: 16 >>> Start: 10/23/22, 13:44:39 >>> End: 10/23/22, 13:44:53 >>> OS Version: Version 12.6 (Build 21G115) >> >> You should update to latest macOS: >> >> $ networkQuality >> ==== SUMMARY ==== >> Uplink capacity: 326.789 Mbps >> Downlink capacity: 446.359 Mbps >> Responsiveness: High (2195 RPM) >> Idle Latency: 5.833 milli-seconds >> >> ;-) >> >> [SM] I wish... just updated to the latest and greatest for this >> hardware (A1398): >> >> macbook-pro:DPZ smoeller$ networkQuality >> ==== SUMMARY ==== >> >> Upload capacity: 7.478 Mbps >> Download capacity: 2.415 Mbps >> Upload flows: 16 >> Download flows: 20 >> Responsiveness: Low (90 RPM) >> macbook-pro:DPZ smoeller$ networkQuality -v >> ==== SUMMARY ==== >> >> Upload capacity: 5.830 Mbps >> Download capacity: 6.077 Mbps >> Upload flows: 12 >> Download flows: 20 >> Responsiveness: Low (56 RPM) >> Base RTT: 134 >> Start: 10/24/22, 22:47:48 >> End: 10/24/22, 22:48:09 >> OS Version: Version 12.6.1 (Build 21G217) >> macbook-pro:DPZ smoeller$ >> >> Still, I only see the "Base RTT" with the -v switch and I am not sure >> whether that is identical to your "Idle Latency". >> >> I guess I need to convince my employer to exchange that macbook >> (actually because the battery starts bulging and not because I am >> behind with networkQuality versions ;) ) >> >> Yes, you would need macOS Ventura to get the latest and greatest. >> >>>> But, what I read is: You are suggesting that “Idle Latency” >>>> should be expressed in RPM as well? Or, Responsiveness expressed >>>> in millisecond ? >>> >>> [SM] Yes, I am fine with either (or both) the idea is to make it >>> really easy to see whether/how much "working conditions" deteriorate >>> the responsiveness / increase the latency-under-load. At least in >>> verbose mode it would be sweet if nwtworkQuality could expose that >>> information. >> >> I see - let me think about that… >> >> Christoph >> >>> Here RPM 2133 corresponds to 60000/2123 = 28.26 ms latency under >>> load, while the Base RTT of 16ms corresponds to 60000/16 = 3750 RPM, >>> son on this link load reduces the responsiveness by 3750-2123 = 1627 >>> RPM a reduction by 100-100*2123/3750 = 43.4%, and that is with >>> competent AQM and scheduling on the router. >>> >>> Without competent AQM/shaping I get: >>> ==== SUMMARY ==== >>> >>> Upload capacity: 15.101 Mbps >>> Download capacity: 97.664 Mbps >>> Upload flows: 20 >>> Download flows: 12 >>> Responsiveness: Medium (427 RPM) >>> Base RTT: 16 >>> Start: 10/23/22, 13:51:50 >>> End: 10/23/22, 13:52:06 >>> OS Version: Version 12.6 (Build 21G115) >>> latency under load: 60000/427 = 140.52 ms >>> base RPM: 60000/16 = 3750 RPM >>> reduction RPM: 100-100*427/3750 = 88.6% >>> >>> I understand apple's desire to have a single reported number with a >>> single qualifier medium/high/... because in the end a link is only >>> reliably usable if responsiveness under load stays acceptable, but >>> with two numbers it is easier to see what one's ISP could do to >>> help. (I guess some ISPs might already be unhappy with the single >>> number, so this needs some diplomacy/tact) >>> >>> Regards >>> Sebastian >>> >>> *) Seemingly as quite some ISPs operate their own speedtest servers >>> in their network and ignore customers not reaching the contracted >>> rates into speedtest-servers located in different ASs. As the >>> product is called internet access I a inclined to expect that my ISP >>> maintains sufficient peering/transit capacity to reach the next tier >>> of AS at my contracted rate (the EU legislative seems to agree, see >>> EU directive 2015/2120). >>> >>> **) Most do by creating load themselves and measuring throughput at >>> the same time, bounceback IIUC will focus on the latency measurement >>> and leave the load generation optional (so offers a mode to measure >>> responsiveness of a live network with minimal measurement traffic). >>> @Bob, please correct me if this is wrong. >>> >>> On Fri, Oct 21, 2022, 5:20 PM Dave Taht wrote: >>> One of the best talks I've ever seen on how to measure customer >>> satisfaction properly just went up after the P99 Conference. >>> >>> It's called Misery Metrics. >>> >>> After going through a deep dive as to why and how we think and act >>> on >>> percentiles, bins, and other statistical methods as to how we use >>> the >>> web and internet are *so wrong* (well worth watching and thinking >>> about if you are relying on or creating network metrics today), it >>> then points to the real metrics that matter to users and the >>> ultimate >>> success of an internet business: Timeouts, retries, misses, failed >>> queries, angry phone calls, abandoned shopping carts and loss of >>> engagement. >>> >>> https://www.p99conf.io/session/misery-metrics-consequences/ >>> >>> The ending advice was - don't aim to make a specific percentile >>> acceptable, aim for an acceptable % of misery. >>> >>> I enjoyed the p99 conference more than any conference I've attended >>> in years. >>> >>> -- >>> This song goes out to all the folk that thought Stadia would work: >>> >> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz >>> Dave Täht CEO, TekLibre, LLC >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, >>> send an email to discuss+unsubscribe@measurementlab.net. >>> To view this discussion on the web visit >>> >> https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com. >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >>> >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/starlink >> _______________________________________________ >> Rpm mailing list >> Rpm@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm