From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B51F73B29E; Mon, 24 Oct 2022 16:08:23 -0400 (EDT) Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 29OJx34B027360; Mon, 24 Oct 2022 13:08:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=av2dFlu9wOqxQ1W+SNg86QvrU1E46xGL4+GDVVdnRyc=; b=RVtaE6cGbw0Jbp0k4p72De305d3QoWrbKTHG6+WrHek146ukOyRI+JCBZdtfOczRlYW0 OfRdSDlU9KAwmmdBGk2Oh7TMCfyyXqc1I4q3Mpl9UUhtmYhLuw6hfAd6BKsOjNMrbcnE 4MhrgUMsuMETbjZIoIf1FQ+OSndQfOy63Gyp/R+qxgllUTw8VbXfsF1kAnru3w8U8vSb kp3n1V8ZHs8Kc/RYwVNk6zaOvDn0hFBlhAheLkX45Sr+lYuXdYOiHvItFlgUkU1I07Yv poc2/6NnZq1JO2uC56PnZu2d31JBKcxuDsPZlyt32jSOGQFgP6wYWiXAfC0XL5KtUZzx kw== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 3kcfgvkhhf-32 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 24 Oct 2022 13:08:17 -0700 Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.19.20220711 64bit (built Jul 11 2022)) with ESMTPS id <0RK900D86X9RKXK0@rn-mailsvcp-mta-lapp01.rno.apple.com>; Mon, 24 Oct 2022 13:08:15 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.19.20220711 64bit (built Jul 11 2022)) id <0RK900G00X3U5I00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Mon, 24 Oct 2022 13:08:15 -0700 (PDT) X-Va-A: X-Va-T-CD: 0af778c0afa90afa8c4c05937d25c782 X-Va-E-CD: aeb69bc064554ffba284960b63524668 X-Va-R-CD: ddef4e518872946d13d3100fb916098e X-Va-CD: 0 X-Va-ID: 2d4ef550-0ea7-46f7-b1db-2d18f2bc533a X-V-A: X-V-T-CD: 0af778c0afa90afa8c4c05937d25c782 X-V-E-CD: aeb69bc064554ffba284960b63524668 X-V-R-CD: ddef4e518872946d13d3100fb916098e X-V-CD: 0 X-V-ID: abf759e1-c54c-4b58-934a-98a6a0bd202f X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-10-24_07:2022-10-20, 2022-10-24 signatures=0 Received: from smtpclient.apple (unknown [17.192.155.174]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.19.20220711 64bit (built Jul 11 2022)) with ESMTPSA id <0RK900LKVX9QD900@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Mon, 24 Oct 2022 13:08:14 -0700 (PDT) From: Christoph Paasch Message-id: <26F033E5-2490-414E-8DFF-4ACD27B74075@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_8E969245-A6FF-4F08-A02E-01D65F6A8548" MIME-version: 1.0 (Mac OS X Mail 16.0 \(3731.200.92\)) Date: Mon, 24 Oct 2022 13:08:04 -0700 In-reply-to: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> Cc: Glenn Fishbine , Rpm , tsvwg IETF list , IETF IPPM WG , Dave Taht via Starlink , Measurement Analysis and Tools Working Group , discuss To: Sebastian Moeller References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> X-Mailer: Apple Mail (2.3731.200.92) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.545, 18.0.895 definitions=2022-10-24_07:2022-10-20, 2022-10-24 signatures=0 Subject: Re: [Rpm] [Starlink] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2022 20:08:23 -0000 --Apple-Mail=_8E969245-A6FF-4F08-A02E-01D65F6A8548 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Sebastian, > On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink = wrote: >=20 > Hi Glenn, >=20 >=20 >> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm = wrote: >>=20 >> As a classic died in the wool empiricist, granted that you can = identify "misery" factors, given a population of 1,000 users, how do you = propose deriving a misery index for that population? >>=20 >> We can measure download, upload, ping, jitter pretty much without = user intervention. For the measurements you hypothesize, how you you = automatically extract those indecies without subjective user = contamination. >>=20 >> I.e. my download speed sucks. Measure the download speed. >>=20 >> My isp doesn't fix my problem. Measure what? How? >>=20 >> Human survey technology is 70+ years old and it still has problems = figuring out how to correlate opinion with fact. >>=20 >> Without an objective measurement scheme that doesn't require human = interaction, the misery index is a cool hypothesis with no way to link = to actual data. What objective measurements can be made? Answer that = and the index becomes useful. Otherwise it's just consumer whining. >>=20 >> Not trying to be combative here, in fact I like the concept you = support, but I'm hard pressed to see how the concept can lead to data, = and the data lead to policy proposals. >=20 > [SM] So it seems that outside of seemingly simple to test = throughput numbers*, the next most important quality number (or the most = important depending on subjective ranking) is how does latency change = under "load". Absolute latency is also important albeit static high = latency can be worked around within limits so the change under load = seems more relevant.=20 > All of flent's RRUL test, apple's networkQuality/RPM, and = iperf2's bounceback test offer methods to asses latency change under = load**, as do waveforms bufferbloat tests and even to a degree Ookla's = speedtest.net . IMHO something like latency = increase under load or apple's responsiveness measure RPM (basically the = inverse of the latency under load calculated on a per minute basis, so = it scales in the typical higher numbers are better way, unlike raw = latency under load numbers where smaller is better). > IMHO what networkQuality is missing ATM is to measure and report = the unloaded RPM as well as the loaded the first gives a measure over = the static latency the second over how well things keep working if = capacity gets tight. They report the base RTT which can be converted to = RPM. As an example: >=20 > macbook:~ user$ networkQuality -v > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 24.341 Mbps > Download capacity: 91.951 Mbps > Upload flows: 20 > Download flows: 16 > Responsiveness: High (2123 RPM) > Base RTT: 16 > Start: 10/23/22, 13:44:39 > End: 10/23/22, 13:44:53 > OS Version: Version 12.6 (Build 21G115) You should update to latest macOS: $ networkQuality =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D Uplink capacity: 326.789 Mbps Downlink capacity: 446.359 Mbps Responsiveness: High (2195 RPM) Idle Latency: 5.833 milli-seconds ;-) But, what I read is: You are suggesting that =E2=80=9CIdle Latency=E2=80=9D= should be expressed in RPM as well? Or, Responsiveness expressed in = millisecond ? Christoph >=20 > Here RPM 2133 corresponds to 60000/2123 =3D 28.26 ms latency under = load, while the Base RTT of 16ms corresponds to 60000/16 =3D 3750 RPM, = son on this link load reduces the responsiveness by 3750-2123 =3D 1627 = RPM a reduction by 100-100*2123/3750 =3D 43.4%, and that is with = competent AQM and scheduling on the router. >=20 > Without competent AQM/shaping I get: > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 15.101 Mbps > Download capacity: 97.664 Mbps > Upload flows: 20 > Download flows: 12 > Responsiveness: Medium (427 RPM) > Base RTT: 16 > Start: 10/23/22, 13:51:50 > End: 10/23/22, 13:52:06 > OS Version: Version 12.6 (Build 21G115) > latency under load: 60000/427 =3D 140.52 ms=20 > base RPM: 60000/16 =3D 3750 RPM > reduction RPM: 100-100*427/3750 =3D 88.6% >=20 >=20 > I understand apple's desire to have a single reported number with a = single qualifier medium/high/... because in the end a link is only = reliably usable if responsiveness under load stays acceptable, but with = two numbers it is easier to see what one's ISP could do to help. (I = guess some ISPs might already be unhappy with the single number, so this = needs some diplomacy/tact) >=20 > Regards > Sebastian >=20 >=20 >=20 > *) Seemingly as quite some ISPs operate their own speedtest servers in = their network and ignore customers not reaching the contracted rates = into speedtest-servers located in different ASs. As the product is = called internet access I a inclined to expect that my ISP maintains = sufficient peering/transit capacity to reach the next tier of AS at my = contracted rate (the EU legislative seems to agree, see EU directive = 2015/2120). >=20 > **) Most do by creating load themselves and measuring throughput at = the same time, bounceback IIUC will focus on the latency measurement and = leave the load generation optional (so offers a mode to measure = responsiveness of a live network with minimal measurement traffic). = @Bob, please correct me if this is wrong. >=20 >=20 >>=20 >>=20 >> On Fri, Oct 21, 2022, 5:20 PM Dave Taht wrote: >> One of the best talks I've ever seen on how to measure customer >> satisfaction properly just went up after the P99 Conference. >>=20 >> It's called Misery Metrics. >>=20 >> After going through a deep dive as to why and how we think and act on >> percentiles, bins, and other statistical methods as to how we use the >> web and internet are *so wrong* (well worth watching and thinking >> about if you are relying on or creating network metrics today), it >> then points to the real metrics that matter to users and the ultimate >> success of an internet business: Timeouts, retries, misses, failed >> queries, angry phone calls, abandoned shopping carts and loss of >> engagement. >>=20 >> https://www.p99conf.io/session/misery-metrics-consequences/ >>=20 >> The ending advice was - don't aim to make a specific percentile >> acceptable, aim for an acceptable % of misery. >>=20 >> I enjoyed the p99 conference more than any conference I've attended = in years. >>=20 >> --=20 >> This song goes out to all the folk that thought Stadia would work: >> = https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665= 607352320-FXtz >> Dave T=C3=A4ht CEO, TekLibre, LLC >>=20 >> --=20 >> You received this message because you are subscribed to the Google = Groups "discuss" group. >> To unsubscribe from this group and stop receiving emails from it, = send an email to discuss+unsubscribe@measurementlab.net. >> To view this discussion on the web visit = https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27= a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com. >> _______________________________________________ >> Rpm mailing list >> Rpm@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm >=20 > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --Apple-Mail=_8E969245-A6FF-4F08-A02E-01D65F6A8548 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello = Sebastian,

On Oct 23, 2022, = at 4:57 AM, Sebastian Moeller via Starlink = <starlink@lists.bufferbloat.net> wrote:

Hi Glenn,


On Oct 23, 2022, = at 02:17, Glenn Fishbine via Rpm <rpm@lists.bufferbloat.net> = wrote:

As a classic died in the wool empiricist, granted that you = can identify "misery" factors, given a population of 1,000 users, how do = you propose deriving a misery index for that population?

We can = measure download, upload, ping, jitter pretty much without user = intervention.  For the measurements you hypothesize, how you you = automatically extract those indecies without subjective user = contamination.

I.e.  my download speed sucks. Measure the = download speed.

My isp doesn't fix my problem. Measure what? = How?

Human survey technology is 70+ years old and it still has = problems figuring out how to correlate opinion with fact.

Without = an objective measurement scheme that doesn't require human interaction, = the misery index is a cool hypothesis with no way to link to actual = data.  What objective measurements can be made?  Answer that = and the index becomes useful. Otherwise it's just consumer = whining.

Not trying to be combative here, in fact I like the = concept you support, but I'm hard pressed to see how the concept can = lead to data, and the data lead to policy proposals.

[SM] So it seems that = outside of seemingly simple to test throughput numbers*, the next most = important quality number (or the most important depending on subjective = ranking) is how does latency change under "load". Absolute latency is = also important albeit static high latency can be worked around within = limits so the change under load seems more relevant. 
All of flent's RRUL = test, apple's networkQuality/RPM, and iperf2's bounceback test offer = methods to asses latency change under load**, as do waveforms = bufferbloat tests and even to a degree Ookla's speedtest.net. IMHO something like latency increase = under load or apple's responsiveness measure RPM (basically the inverse = of the latency under load calculated on a per minute basis, so it scales = in the typical higher numbers are better way, unlike raw latency under = load numbers where smaller is better).
= IMHO what networkQuality = is missing ATM is to measure and report the unloaded RPM as well as the = loaded the first gives a measure over the static latency the second over = how well things keep working if capacity gets tight. They report the = base RTT which can be converted to RPM. As an example:

macbook:~ user$ networkQuality -v
=3D=3D=3D=3D SUMMARY =3D=3D=3D=3D =             &n= bsp;           &nbs= p;            =             &n= bsp;           &nbs= p;            =             &n= bsp; 
Upload capacity: 24.341 = Mbps
Download capacity: = 91.951 Mbps
Upload flows: = 20
Download flows: = 16
Responsiveness: High = (2123 RPM)
Base RTT: 16
Start: 10/23/22, 13:44:39
End: 10/23/22, 13:44:53
OS Version: Version 12.6 (Build = 21G115)

You should update to = latest macOS:

$ = networkQuality
=3D=3D=3D=3D SUMMARY =3D=3D=3D=3D
Uplin= k capacity: 326.789 Mbps
Downlink capacity: 446.359 = Mbps
Responsiveness: High (2195 RPM)
Idle Latency: = 5.833 = milli-seconds

;-)


But, what I read is: You are suggesting that =E2=80=9CIdle = Latency=E2=80=9D should be expressed in RPM as well? Or, Responsiveness = expressed in millisecond = ?


Christoph




Here RPM 2133 corresponds to 60000/2123 =3D = 28.26 ms latency under load, while the Base RTT of 16ms corresponds to = 60000/16 =3D 3750 RPM, son on this link load reduces the responsiveness = by 3750-2123 =3D 1627 RPM a reduction by 100-100*2123/3750 =3D 43.4%, = and that is with competent AQM and scheduling on the router.

Without competent AQM/shaping I = get:
=3D=3D=3D=3D SUMMARY = =3D=3D=3D=3D =             &n= bsp;           &nbs= p;            =             &n= bsp;           &nbs= p;            =             &n= bsp; 
Upload capacity: 15.101 = Mbps
Download capacity: = 97.664 Mbps
Upload flows: = 20
Download flows: = 12
Responsiveness: Medium = (427 RPM)
Base RTT: 16
Start: 10/23/22, 13:51:50
End: 10/23/22, 13:52:06
OS Version: Version 12.6 (Build = 21G115)
latency under load: = 60000/427 =3D 140.52 ms 
base RPM: 60000/16 =3D 3750 RPM
reduction RPM: 100-100*427/3750 =3D = 88.6%


I understand apple's = desire to have a single reported number with a single qualifier = medium/high/... because in the end a link is only reliably usable if = responsiveness under load stays acceptable, but with two numbers it is = easier to see what one's ISP could do to help. (I guess some ISPs might = already be unhappy with the single number, so this needs some = diplomacy/tact)

Regards
= Sebastian



*) Seemingly as quite some ISPs operate = their own speedtest servers in their network and ignore customers not = reaching the contracted rates into speedtest-servers located in = different ASs. As the product is called internet access I a inclined to = expect that my ISP maintains sufficient peering/transit capacity to = reach the next tier of AS at my contracted rate (the EU legislative = seems to agree, see EU directive 2015/2120).

**) Most do by creating load themselves and = measuring throughput at the same time, bounceback IIUC will focus on the = latency measurement and leave the load generation optional (so offers a = mode to measure responsiveness of a live network with minimal = measurement traffic). @Bob, please correct me if this is = wrong.




On Fri, = Oct 21, 2022, 5:20 PM Dave Taht <dave.taht@gmail.com> = wrote:
One of the best talks I've ever seen on how to measure = customer
satisfaction properly just went up after the P99 = Conference.

It's called Misery Metrics.

After going = through a deep dive as to why and how we think and act = on
percentiles, bins, and other statistical methods as to how we use = the
web and internet are *so wrong* (well worth watching and = thinking
about if you are relying on or creating network metrics = today), it
then points to the real metrics that matter to users and = the ultimate
success of an internet business: Timeouts, retries, = misses, failed
queries, angry phone calls, abandoned shopping carts = and loss = of
engagement.

https://www.p99conf.io/session/misery-metrics-con= sequences/

The ending advice was - don't aim to make a specific = percentile
acceptable, aim for an acceptable % of misery.

I = enjoyed the p99 conference more than any conference I've attended in = years.

-- 
This song goes out to = all the folk that thought Stadia would = work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6= 981366665607352320-FXtz
Dave T=C3=A4ht CEO, TekLibre, = LLC

-- 
You = received this message because you are subscribed to the Google Groups = "discuss" group.
To unsubscribe from this group and stop receiving = emails from it, send an email to = discuss+unsubscribe@measurementlab.net.
To view this discussion on = the web visit = https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27= a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com.
________= _______________________________________
Rpm mailing list
Rpm@lists.bufferbloat.nethttps://lists.bufferbl= oat.net/listinfo/rpm

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

= --Apple-Mail=_8E969245-A6FF-4F08-A02E-01D65F6A8548--