Re: [Starlink] [Rpm] [M-Lab-Discuss] misery metrics & consequences

[rjmcmahon <rjmcmahon@rjmcmahon.com>]

Be careful about assuming network loads always worsen latency over 
networks. Below is an example over WiFi with a rasberry pi over a 
Netgear Nighthawk RAXE500 to a 1G wired linux host, without a load then 
with an upstream load. I've noticed similar with some hardware 
forwarding planes where a busier AP outperforms, in terms of latency, a 
lightly loaded one. (Note: Iperf 2 uses responses per second (RPS) as 
the SI Units of time is the second.)

rjmcmahon@ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i 1 
--bounceback
------------------------------------------------------------
Client connecting to 192.168.1.69, TCP port 5001 with pid 4148 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs & 
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.40%eth0 port 53750 connected with 192.168.1.69 
port 5001 (bb w/quickack len/hold=100/0) (sock=3) 
(icwnd/mss/irtt=14/1448/341) (ct=0.44 ms) on 2022-10-25 00:00:48 (UTC)
[ ID] Interval        Transfer    Bandwidth         BB 
cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.340/0.206/0.852/0.188 ms    0   14K/220 us    2941 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.481/0.362/0.572/0.057 ms    0   14K/327 us    2078 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.471/0.344/0.694/0.089 ms    0   14K/340 us    2123 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.406/0.330/0.595/0.072 ms    0   14K/318 us    2465 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.471/0.405/0.603/0.057 ms    0   14K/348 us    2124 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.428/0.355/0.641/0.079 ms    0   14K/324 us    2337 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.429/0.329/0.616/0.086 ms    0   14K/306 us    2329 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.445/0.325/0.673/0.092 ms    0   14K/321 us    2248 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.423/0.348/0.604/0.074 ms    0   14K/299 us    2366 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.463/0.369/0.729/0.108 ms    0   14K/306 us    2159 rps
[  1] 0.00-10.01 sec  19.7 KBytes  16.1 Kbits/sec    
101=0.438/0.206/0.852/0.102 ms    0   14K/1192 us    2285 rps
[  1] 0.00-10.01 sec BB8-PDF: 
bin(w=100us):cnt(101)=3:5,4:30,5:48,6:9,7:7,8:1,9:1 
(5.00/95.00/99.7%=4/7/9,Outliers=0,obl/obu=0/0)


rjmcmahon@ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i 1 
--bounceback --bounceback-congest=up,1
------------------------------------------------------------
Client connecting to 192.168.1.69, TCP port 5001 with pid 4152 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs & 
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  2] local 192.168.1.40%eth0 port 50462 connected with 192.168.1.69 
port 5001 (sock=4) (qack) (icwnd/mss/irtt=14/1448/475) (ct=0.63 ms) on 
2022-10-25 00:01:07 (UTC)
[  1] local 192.168.1.40%eth0 port 50472 connected with 192.168.1.69 
port 5001 (bb w/quickack len/hold=100/0) (sock=3) 
(icwnd/mss/irtt=14/1448/375) (ct=0.67 ms) on 2022-10-25 00:01:07 (UTC)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
Cwnd/RTT(var)        NetPwr
[  2] 0.00-1.00 sec  3.73 MBytes  31.3 Mbits/sec  39069/0          0     
   59K/133(1) us  29376
[ ID] Interval        Transfer    Bandwidth         BB 
cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.282/0.178/0.887/0.214 ms    0   14K/191 us    3547 rps
[  2] 1.00-2.00 sec  3.77 MBytes  31.6 Mbits/sec  39512/0          0     
   59K/133(1) us  29708
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.196/0.149/0.240/0.024 ms    0   14K/161 us    5115 rps
[  2] 2.00-3.00 sec  3.77 MBytes  31.6 Mbits/sec  39558/0          0     
   59K/125(8) us  31646
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.163/0.134/0.192/0.018 ms    0   14K/136 us    6124 rps
[  2] 3.00-4.00 sec  3.77 MBytes  31.6 Mbits/sec  39560/0          0     
   59K/133(1) us  29744
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.171/0.149/0.216/0.019 ms    0   14K/133 us    5838 rps
[  2] 4.00-5.00 sec  3.76 MBytes  31.6 Mbits/sec  39460/0          0     
   59K/131(2) us  30122
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.188/0.131/0.242/0.029 ms    0   14K/143 us    5308 rps
[  2] 5.00-6.00 sec  3.77 MBytes  31.6 Mbits/sec  39545/0          0     
   59K/133(0) us  29733
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.197/0.147/0.255/0.031 ms    0   14K/149 us    5079 rps
[  2] 6.00-7.00 sec  3.78 MBytes  31.7 Mbits/sec  39631/0          0     
   59K/133(1) us  29798
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.196/0.146/0.229/0.025 ms    0   14K/151 us    5102 rps
[  2] 7.00-8.00 sec  3.77 MBytes  31.6 Mbits/sec  39497/0          0     
   59K/133(0) us  29697
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.190/0.147/0.225/0.028 ms    0   14K/155 us    5260 rps
[  2] 8.00-9.00 sec  3.77 MBytes  31.6 Mbits/sec  39533/0          0     
   59K/126(4) us  31375
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.185/0.158/0.208/0.017 ms    0   14K/148 us    5414 rps
[  2] 9.00-10.00 sec  3.77 MBytes  31.6 Mbits/sec  39519/0          0    
    59K/133(1) us  29714
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    
10=0.165/0.134/0.232/0.028 ms    0   14K/131 us    6064 rps
[  2] 0.00-10.01 sec  37.7 MBytes  31.6 Mbits/sec  394886/0          0   
     59K/1430(2595) us  2759
[  1] 0.00-10.01 sec  19.7 KBytes  16.1 Kbits/sec    
101=0.194/0.131/0.887/0.075 ms    0   14K/1385 us    5167 rps
[  1] 0.00-10.01 sec BB8-PDF: bin(w=100us):cnt(101)=2:69,3:31,9:1 
(5.00/95.00/99.7%=2/3/9,Outliers=0,obl/obu=0/0)
[ CT] final connect times (min/avg/max/stdev) = 0.627/0.647/0.666/27.577 
ms (tot/err) = 2/0


Bob
>> On Oct 24, 2022, at 1:57 PM, Sebastian Moeller <moeller0@gmx.de>
>> wrote:
>> Hi Christoph
>> 
>> On Oct 24, 2022, at 22:08, Christoph Paasch <cpaasch@apple.com>
>> wrote:
>> 
>> Hello Sebastian,
>> 
>> On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink
>> <starlink@lists.bufferbloat.net> wrote:
>> 
>> Hi Glenn,
>> 
>> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm
>> <rpm@lists.bufferbloat.net> wrote:
>> 
>> As a classic died in the wool empiricist, granted that you can
>> identify "misery" factors, given a population of 1,000 users, how do
>> you propose deriving a misery index for that population?
>> 
>> We can measure download, upload, ping, jitter pretty much without
>> user intervention.  For the measurements you hypothesize, how you
>> you automatically extract those indecies without subjective user
>> contamination.
>> 
>> I.e.  my download speed sucks. Measure the download speed.
>> 
>> My isp doesn't fix my problem. Measure what? How?
>> 
>> Human survey technology is 70+ years old and it still has problems
>> figuring out how to correlate opinion with fact.
>> 
>> Without an objective measurement scheme that doesn't require human
>> interaction, the misery index is a cool hypothesis with no way to
>> link to actual data.  What objective measurements can be made?
>> Answer that and the index becomes useful. Otherwise it's just
>> consumer whining.
>> 
>> Not trying to be combative here, in fact I like the concept you
>> support, but I'm hard pressed to see how the concept can lead to
>> data, and the data lead to policy proposals.
>> 
>> [SM] So it seems that outside of seemingly simple to test
>> throughput numbers*, the next most important quality number (or the
>> most important depending on subjective ranking) is how does latency
>> change under "load". Absolute latency is also important albeit
>> static high latency can be worked around within limits so the change
>> under load seems more relevant.
>> All of flent's RRUL test, apple's networkQuality/RPM, and iperf2's
>> bounceback test offer methods to asses latency change under load**,
>> as do waveforms bufferbloat tests and even to a degree Ookla's
>> speedtest.net. IMHO something like latency increase under load or
>> apple's responsiveness measure RPM (basically the inverse of the
>> latency under load calculated on a per minute basis, so it scales in
>> the typical higher numbers are better way, unlike raw latency under
>> load numbers where smaller is better).
>> IMHO what networkQuality is missing ATM is to measure and report
>> the unloaded RPM as well as the loaded the first gives a measure
>> over the static latency the second over how well things keep working
>> if capacity gets tight. They report the base RTT which can be
>> converted to RPM. As an example:
>> 
>> macbook:~ user$ networkQuality -v
>> ==== SUMMARY ====
>> 
>> Upload capacity: 24.341 Mbps
>> Download capacity: 91.951 Mbps
>> Upload flows: 20
>> Download flows: 16
>> Responsiveness: High (2123 RPM)
>> Base RTT: 16
>> Start: 10/23/22, 13:44:39
>> End: 10/23/22, 13:44:53
>> OS Version: Version 12.6 (Build 21G115)
> 
> You should update to latest macOS:
> 
> $ networkQuality
> ==== SUMMARY ====
> Uplink capacity: 326.789 Mbps
> Downlink capacity: 446.359 Mbps
> Responsiveness: High (2195 RPM)
> Idle Latency: 5.833 milli-seconds
> 
> ;-)
> 
>  [SM] I wish... just updated to the latest and greatest for this
> hardware (A1398):
> 
> macbook-pro:DPZ smoeller$ networkQuality
> ==== SUMMARY ====
> 
> Upload capacity: 7.478 Mbps
> Download capacity: 2.415 Mbps
> Upload flows: 16
> Download flows: 20
> Responsiveness: Low (90 RPM)
> macbook-pro:DPZ smoeller$ networkQuality -v
> ==== SUMMARY ====
> 
> Upload capacity: 5.830 Mbps
> Download capacity: 6.077 Mbps
> Upload flows: 12
> Download flows: 20
> Responsiveness: Low (56 RPM)
> Base RTT: 134
> Start: 10/24/22, 22:47:48
> End: 10/24/22, 22:48:09
> OS Version: Version 12.6.1 (Build 21G217)
> macbook-pro:DPZ smoeller$
> 
> Still, I only see the "Base RTT" with the -v switch and I am not sure
> whether that is identical to your "Idle Latency".
> 
> I guess I need to convince my employer to exchange that macbook
> (actually because the battery starts bulging and not because I am
> behind with networkQuality versions ;) )
> 
> Yes, you would need macOS Ventura to get the latest and greatest.
> 
>>> But, what I read is: You are suggesting that “Idle Latency”
>>> should be expressed in RPM as well? Or, Responsiveness expressed
>>> in millisecond ?
>> 
>> [SM] Yes, I am fine with either (or both) the idea is to make it
>> really easy to see whether/how much "working conditions" deteriorate
>> the responsiveness / increase the latency-under-load. At least in
>> verbose mode it would be sweet if nwtworkQuality could expose that
>> information.
> 
> I see - let me think about that…
> 
> Christoph
> 
>> Here RPM 2133 corresponds to 60000/2123 = 28.26 ms latency under
>> load, while the Base RTT of 16ms corresponds to 60000/16 = 3750 RPM,
>> son on this link load reduces the responsiveness by 3750-2123 = 1627
>> RPM a reduction by 100-100*2123/3750 = 43.4%, and that is with
>> competent AQM and scheduling on the router.
>> 
>> Without competent AQM/shaping I get:
>> ==== SUMMARY ====
>> 
>> Upload capacity: 15.101 Mbps
>> Download capacity: 97.664 Mbps
>> Upload flows: 20
>> Download flows: 12
>> Responsiveness: Medium (427 RPM)
>> Base RTT: 16
>> Start: 10/23/22, 13:51:50
>> End: 10/23/22, 13:52:06
>> OS Version: Version 12.6 (Build 21G115)
>> latency under load: 60000/427 = 140.52 ms
>> base RPM: 60000/16 = 3750 RPM
>> reduction RPM: 100-100*427/3750 = 88.6%
>> 
>> I understand apple's desire to have a single reported number with a
>> single qualifier medium/high/... because in the end a link is only
>> reliably usable if responsiveness under load stays acceptable, but
>> with two numbers it is easier to see what one's ISP could do to
>> help. (I guess some ISPs might already be unhappy with the single
>> number, so this needs some diplomacy/tact)
>> 
>> Regards
>> Sebastian
>> 
>> *) Seemingly as quite some ISPs operate their own speedtest servers
>> in their network and ignore customers not reaching the contracted
>> rates into speedtest-servers located in different ASs. As the
>> product is called internet access I a inclined to expect that my ISP
>> maintains sufficient peering/transit capacity to reach the next tier
>> of AS at my contracted rate (the EU legislative seems to agree, see
>> EU directive 2015/2120).
>> 
>> **) Most do by creating load themselves and measuring throughput at
>> the same time, bounceback IIUC will focus on the latency measurement
>> and leave the load generation optional (so offers a mode to measure
>> responsiveness of a live network with minimal measurement traffic).
>> @Bob, please correct me if this is wrong.
>> 
>> On Fri, Oct 21, 2022, 5:20 PM Dave Taht <dave.taht@gmail.com> wrote:
>> One of the best talks I've ever seen on how to measure customer
>> satisfaction properly just went up after the P99 Conference.
>> 
>> It's called Misery Metrics.
>> 
>> After going through a deep dive as to why and how we think and act
>> on
>> percentiles, bins, and other statistical methods as to how we use
>> the
>> web and internet are *so wrong* (well worth watching and thinking
>> about if you are relying on or creating network metrics today), it
>> then points to the real metrics that matter to users and the
>> ultimate
>> success of an internet business: Timeouts, retries, misses, failed
>> queries, angry phone calls, abandoned shopping carts and loss of
>> engagement.
>> 
>> https://www.p99conf.io/session/misery-metrics-consequences/
>> 
>> The ending advice was - don't aim to make a specific percentile
>> acceptable, aim for an acceptable % of misery.
>> 
>> I enjoyed the p99 conference more than any conference I've attended
>> in years.
>> 
>> --
>> This song goes out to all the folk that thought Stadia would work:
>> 
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> Dave Täht CEO, TekLibre, LLC
>> 
>> --
>> You received this message because you are subscribed to the Google
>> Groups "discuss" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to discuss+unsubscribe@measurementlab.net.
>> To view this discussion on the web visit
>> 
> https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com.
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
>> 
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm