[Starlink] [Rpm] [M-Lab-Discuss] misery metrics & consequences
rjmcmahon
rjmcmahon at rjmcmahon.com
Mon Oct 24 20:12:39 EDT 2022
Sorry, the RPi4 rebooted and this test was over wired ethernet. But the
point is the same.
Bob
> Be careful about assuming network loads always worsen latency over
> networks. Below is an example over WiFi with a rasberry pi over a
> Netgear Nighthawk RAXE500 to a 1G wired linux host, without a load
> then with an upstream load. I've noticed similar with some hardware
> forwarding planes where a busier AP outperforms, in terms of latency,
> a lightly loaded one. (Note: Iperf 2 uses responses per second (RPS)
> as the SI Units of time is the second.)
>
> rjmcmahon at ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i
> 1 --bounceback
> ------------------------------------------------------------
> Client connecting to 192.168.1.69, TCP port 5001 with pid 4148 (1
> flows)
> Write buffer size: 100 Byte
> Bursting: 100 Byte writes 10 times every 1.00 second(s)
> Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
> tcp_quickack)
> TOS set to 0x0 and nodelay (Nagle off)
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 1] local 192.168.1.40%eth0 port 53750 connected with 192.168.1.69
> port 5001 (bb w/quickack len/hold=100/0) (sock=3)
> (icwnd/mss/irtt=14/1448/341) (ct=0.44 ms) on 2022-10-25 00:00:48 (UTC)
> [ ID] Interval Transfer Bandwidth BB
> cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
> [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.340/0.206/0.852/0.188 ms 0 14K/220 us 2941 rps
> [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.481/0.362/0.572/0.057 ms 0 14K/327 us 2078 rps
> [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.471/0.344/0.694/0.089 ms 0 14K/340 us 2123 rps
> [ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.406/0.330/0.595/0.072 ms 0 14K/318 us 2465 rps
> [ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.471/0.405/0.603/0.057 ms 0 14K/348 us 2124 rps
> [ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.428/0.355/0.641/0.079 ms 0 14K/324 us 2337 rps
> [ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.429/0.329/0.616/0.086 ms 0 14K/306 us 2329 rps
> [ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.445/0.325/0.673/0.092 ms 0 14K/321 us 2248 rps
> [ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.423/0.348/0.604/0.074 ms 0 14K/299 us 2366 rps
> [ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.463/0.369/0.729/0.108 ms 0 14K/306 us 2159 rps
> [ 1] 0.00-10.01 sec 19.7 KBytes 16.1 Kbits/sec
> 101=0.438/0.206/0.852/0.102 ms 0 14K/1192 us 2285 rps
> [ 1] 0.00-10.01 sec BB8-PDF:
> bin(w=100us):cnt(101)=3:5,4:30,5:48,6:9,7:7,8:1,9:1
> (5.00/95.00/99.7%=4/7/9,Outliers=0,obl/obu=0/0)
>
>
> rjmcmahon at ubuntu:/usr/local/src/iperf2-code$ iperf -c 192.168.1.69 -i
> 1 --bounceback --bounceback-congest=up,1
> ------------------------------------------------------------
> Client connecting to 192.168.1.69, TCP port 5001 with pid 4152 (1
> flows)
> Write buffer size: 100 Byte
> Bursting: 100 Byte writes 10 times every 1.00 second(s)
> Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
> tcp_quickack)
> TOS set to 0x0 and nodelay (Nagle off)
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 2] local 192.168.1.40%eth0 port 50462 connected with 192.168.1.69
> port 5001 (sock=4) (qack) (icwnd/mss/irtt=14/1448/475) (ct=0.63 ms) on
> 2022-10-25 00:01:07 (UTC)
> [ 1] local 192.168.1.40%eth0 port 50472 connected with 192.168.1.69
> port 5001 (bb w/quickack len/hold=100/0) (sock=3)
> (icwnd/mss/irtt=14/1448/375) (ct=0.67 ms) on 2022-10-25 00:01:07 (UTC)
> [ ID] Interval Transfer Bandwidth Write/Err Rtry
> Cwnd/RTT(var) NetPwr
> [ 2] 0.00-1.00 sec 3.73 MBytes 31.3 Mbits/sec 39069/0 0
> 59K/133(1) us 29376
> [ ID] Interval Transfer Bandwidth BB
> cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
> [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.282/0.178/0.887/0.214 ms 0 14K/191 us 3547 rps
> [ 2] 1.00-2.00 sec 3.77 MBytes 31.6 Mbits/sec 39512/0 0
> 59K/133(1) us 29708
> [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.196/0.149/0.240/0.024 ms 0 14K/161 us 5115 rps
> [ 2] 2.00-3.00 sec 3.77 MBytes 31.6 Mbits/sec 39558/0 0
> 59K/125(8) us 31646
> [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.163/0.134/0.192/0.018 ms 0 14K/136 us 6124 rps
> [ 2] 3.00-4.00 sec 3.77 MBytes 31.6 Mbits/sec 39560/0 0
> 59K/133(1) us 29744
> [ 1] 3.00-4.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.171/0.149/0.216/0.019 ms 0 14K/133 us 5838 rps
> [ 2] 4.00-5.00 sec 3.76 MBytes 31.6 Mbits/sec 39460/0 0
> 59K/131(2) us 30122
> [ 1] 4.00-5.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.188/0.131/0.242/0.029 ms 0 14K/143 us 5308 rps
> [ 2] 5.00-6.00 sec 3.77 MBytes 31.6 Mbits/sec 39545/0 0
> 59K/133(0) us 29733
> [ 1] 5.00-6.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.197/0.147/0.255/0.031 ms 0 14K/149 us 5079 rps
> [ 2] 6.00-7.00 sec 3.78 MBytes 31.7 Mbits/sec 39631/0 0
> 59K/133(1) us 29798
> [ 1] 6.00-7.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.196/0.146/0.229/0.025 ms 0 14K/151 us 5102 rps
> [ 2] 7.00-8.00 sec 3.77 MBytes 31.6 Mbits/sec 39497/0 0
> 59K/133(0) us 29697
> [ 1] 7.00-8.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.190/0.147/0.225/0.028 ms 0 14K/155 us 5260 rps
> [ 2] 8.00-9.00 sec 3.77 MBytes 31.6 Mbits/sec 39533/0 0
> 59K/126(4) us 31375
> [ 1] 8.00-9.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.185/0.158/0.208/0.017 ms 0 14K/148 us 5414 rps
> [ 2] 9.00-10.00 sec 3.77 MBytes 31.6 Mbits/sec 39519/0 0
> 59K/133(1) us 29714
> [ 1] 9.00-10.00 sec 1.95 KBytes 16.0 Kbits/sec
> 10=0.165/0.134/0.232/0.028 ms 0 14K/131 us 6064 rps
> [ 2] 0.00-10.01 sec 37.7 MBytes 31.6 Mbits/sec 394886/0 0
> 59K/1430(2595) us 2759
> [ 1] 0.00-10.01 sec 19.7 KBytes 16.1 Kbits/sec
> 101=0.194/0.131/0.887/0.075 ms 0 14K/1385 us 5167 rps
> [ 1] 0.00-10.01 sec BB8-PDF: bin(w=100us):cnt(101)=2:69,3:31,9:1
> (5.00/95.00/99.7%=2/3/9,Outliers=0,obl/obu=0/0)
> [ CT] final connect times (min/avg/max/stdev) =
> 0.627/0.647/0.666/27.577 ms (tot/err) = 2/0
>
>
> Bob
>>> On Oct 24, 2022, at 1:57 PM, Sebastian Moeller <moeller0 at gmx.de>
>>> wrote:
>>> Hi Christoph
>>>
>>> On Oct 24, 2022, at 22:08, Christoph Paasch <cpaasch at apple.com>
>>> wrote:
>>>
>>> Hello Sebastian,
>>>
>>> On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink
>>> <starlink at lists.bufferbloat.net> wrote:
>>>
>>> Hi Glenn,
>>>
>>> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm
>>> <rpm at lists.bufferbloat.net> wrote:
>>>
>>> As a classic died in the wool empiricist, granted that you can
>>> identify "misery" factors, given a population of 1,000 users, how do
>>> you propose deriving a misery index for that population?
>>>
>>> We can measure download, upload, ping, jitter pretty much without
>>> user intervention. For the measurements you hypothesize, how you
>>> you automatically extract those indecies without subjective user
>>> contamination.
>>>
>>> I.e. my download speed sucks. Measure the download speed.
>>>
>>> My isp doesn't fix my problem. Measure what? How?
>>>
>>> Human survey technology is 70+ years old and it still has problems
>>> figuring out how to correlate opinion with fact.
>>>
>>> Without an objective measurement scheme that doesn't require human
>>> interaction, the misery index is a cool hypothesis with no way to
>>> link to actual data. What objective measurements can be made?
>>> Answer that and the index becomes useful. Otherwise it's just
>>> consumer whining.
>>>
>>> Not trying to be combative here, in fact I like the concept you
>>> support, but I'm hard pressed to see how the concept can lead to
>>> data, and the data lead to policy proposals.
>>>
>>> [SM] So it seems that outside of seemingly simple to test
>>> throughput numbers*, the next most important quality number (or the
>>> most important depending on subjective ranking) is how does latency
>>> change under "load". Absolute latency is also important albeit
>>> static high latency can be worked around within limits so the change
>>> under load seems more relevant.
>>> All of flent's RRUL test, apple's networkQuality/RPM, and iperf2's
>>> bounceback test offer methods to asses latency change under load**,
>>> as do waveforms bufferbloat tests and even to a degree Ookla's
>>> speedtest.net. IMHO something like latency increase under load or
>>> apple's responsiveness measure RPM (basically the inverse of the
>>> latency under load calculated on a per minute basis, so it scales in
>>> the typical higher numbers are better way, unlike raw latency under
>>> load numbers where smaller is better).
>>> IMHO what networkQuality is missing ATM is to measure and report
>>> the unloaded RPM as well as the loaded the first gives a measure
>>> over the static latency the second over how well things keep working
>>> if capacity gets tight. They report the base RTT which can be
>>> converted to RPM. As an example:
>>>
>>> macbook:~ user$ networkQuality -v
>>> ==== SUMMARY ====
>>>
>>> Upload capacity: 24.341 Mbps
>>> Download capacity: 91.951 Mbps
>>> Upload flows: 20
>>> Download flows: 16
>>> Responsiveness: High (2123 RPM)
>>> Base RTT: 16
>>> Start: 10/23/22, 13:44:39
>>> End: 10/23/22, 13:44:53
>>> OS Version: Version 12.6 (Build 21G115)
>>
>> You should update to latest macOS:
>>
>> $ networkQuality
>> ==== SUMMARY ====
>> Uplink capacity: 326.789 Mbps
>> Downlink capacity: 446.359 Mbps
>> Responsiveness: High (2195 RPM)
>> Idle Latency: 5.833 milli-seconds
>>
>> ;-)
>>
>> [SM] I wish... just updated to the latest and greatest for this
>> hardware (A1398):
>>
>> macbook-pro:DPZ smoeller$ networkQuality
>> ==== SUMMARY ====
>>
>> Upload capacity: 7.478 Mbps
>> Download capacity: 2.415 Mbps
>> Upload flows: 16
>> Download flows: 20
>> Responsiveness: Low (90 RPM)
>> macbook-pro:DPZ smoeller$ networkQuality -v
>> ==== SUMMARY ====
>>
>> Upload capacity: 5.830 Mbps
>> Download capacity: 6.077 Mbps
>> Upload flows: 12
>> Download flows: 20
>> Responsiveness: Low (56 RPM)
>> Base RTT: 134
>> Start: 10/24/22, 22:47:48
>> End: 10/24/22, 22:48:09
>> OS Version: Version 12.6.1 (Build 21G217)
>> macbook-pro:DPZ smoeller$
>>
>> Still, I only see the "Base RTT" with the -v switch and I am not sure
>> whether that is identical to your "Idle Latency".
>>
>> I guess I need to convince my employer to exchange that macbook
>> (actually because the battery starts bulging and not because I am
>> behind with networkQuality versions ;) )
>>
>> Yes, you would need macOS Ventura to get the latest and greatest.
>>
>>>> But, what I read is: You are suggesting that “Idle Latency”
>>>> should be expressed in RPM as well? Or, Responsiveness expressed
>>>> in millisecond ?
>>>
>>> [SM] Yes, I am fine with either (or both) the idea is to make it
>>> really easy to see whether/how much "working conditions" deteriorate
>>> the responsiveness / increase the latency-under-load. At least in
>>> verbose mode it would be sweet if nwtworkQuality could expose that
>>> information.
>>
>> I see - let me think about that…
>>
>> Christoph
>>
>>> Here RPM 2133 corresponds to 60000/2123 = 28.26 ms latency under
>>> load, while the Base RTT of 16ms corresponds to 60000/16 = 3750 RPM,
>>> son on this link load reduces the responsiveness by 3750-2123 = 1627
>>> RPM a reduction by 100-100*2123/3750 = 43.4%, and that is with
>>> competent AQM and scheduling on the router.
>>>
>>> Without competent AQM/shaping I get:
>>> ==== SUMMARY ====
>>>
>>> Upload capacity: 15.101 Mbps
>>> Download capacity: 97.664 Mbps
>>> Upload flows: 20
>>> Download flows: 12
>>> Responsiveness: Medium (427 RPM)
>>> Base RTT: 16
>>> Start: 10/23/22, 13:51:50
>>> End: 10/23/22, 13:52:06
>>> OS Version: Version 12.6 (Build 21G115)
>>> latency under load: 60000/427 = 140.52 ms
>>> base RPM: 60000/16 = 3750 RPM
>>> reduction RPM: 100-100*427/3750 = 88.6%
>>>
>>> I understand apple's desire to have a single reported number with a
>>> single qualifier medium/high/... because in the end a link is only
>>> reliably usable if responsiveness under load stays acceptable, but
>>> with two numbers it is easier to see what one's ISP could do to
>>> help. (I guess some ISPs might already be unhappy with the single
>>> number, so this needs some diplomacy/tact)
>>>
>>> Regards
>>> Sebastian
>>>
>>> *) Seemingly as quite some ISPs operate their own speedtest servers
>>> in their network and ignore customers not reaching the contracted
>>> rates into speedtest-servers located in different ASs. As the
>>> product is called internet access I a inclined to expect that my ISP
>>> maintains sufficient peering/transit capacity to reach the next tier
>>> of AS at my contracted rate (the EU legislative seems to agree, see
>>> EU directive 2015/2120).
>>>
>>> **) Most do by creating load themselves and measuring throughput at
>>> the same time, bounceback IIUC will focus on the latency measurement
>>> and leave the load generation optional (so offers a mode to measure
>>> responsiveness of a live network with minimal measurement traffic).
>>> @Bob, please correct me if this is wrong.
>>>
>>> On Fri, Oct 21, 2022, 5:20 PM Dave Taht <dave.taht at gmail.com> wrote:
>>> One of the best talks I've ever seen on how to measure customer
>>> satisfaction properly just went up after the P99 Conference.
>>>
>>> It's called Misery Metrics.
>>>
>>> After going through a deep dive as to why and how we think and act
>>> on
>>> percentiles, bins, and other statistical methods as to how we use
>>> the
>>> web and internet are *so wrong* (well worth watching and thinking
>>> about if you are relying on or creating network metrics today), it
>>> then points to the real metrics that matter to users and the
>>> ultimate
>>> success of an internet business: Timeouts, retries, misses, failed
>>> queries, angry phone calls, abandoned shopping carts and loss of
>>> engagement.
>>>
>>> https://www.p99conf.io/session/misery-metrics-consequences/
>>>
>>> The ending advice was - don't aim to make a specific percentile
>>> acceptable, aim for an acceptable % of misery.
>>>
>>> I enjoyed the p99 conference more than any conference I've attended
>>> in years.
>>>
>>> --
>>> This song goes out to all the folk that thought Stadia would work:
>>>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>>> Dave Täht CEO, TekLibre, LLC
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to discuss+unsubscribe at measurementlab.net.
>>> To view this discussion on the web visit
>>>
>> https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com.
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>>>
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/starlink
>> _______________________________________________
>> Rpm mailing list
>> Rpm at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
> _______________________________________________
> Rpm mailing list
> Rpm at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
More information about the Starlink
mailing list