From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 290653CB37 for ; Mon, 24 Oct 2022 22:28:52 -0400 (EDT) Received: by mail-io1-xd33.google.com with SMTP id i65so9340532ioa.0 for ; Mon, 24 Oct 2022 19:28:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=HotxTBbYYzjd560IXzRyou05rfj9gAol5NBvPjJgqTk=; b=i24DH5D6uQdb3ao7Y+eT5FMJAMrSUqkQVI+d73829gG6cDY7gZGR5hiBQtpw+KSrlr EGDOx/bnFgs2zJSeMyC2yaq6MObv8tsWvlmMXAnqBCoY3YH6vfWZboXjrIbsthQHYNe4 gFmDxylRnTzxs3WLgMfEbvDRg60NluXkvbqBbjWIBpKqUM92gbIyyKxc4/Ugdq9GzKos 3fr/BuS4FUJtmxxHhG8hcXKPwOt0Uc80UyrHBVtc+FjR9qTMYqJY+HgIJM8jW2uXUX4s K89g/aBM2cZAZFWVVUc+ZqpCyO+qt5otLkYrjMZTVoMhqS14Go3msLo+77l+E+9L/W/Y NFFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HotxTBbYYzjd560IXzRyou05rfj9gAol5NBvPjJgqTk=; b=e6Ci4nRJJSGX5UineeMHSKIVxhtszaTGO4j1LOmSoCoMR/iqrStrek89EsafzRciRn tPFr7nAZRLpHOou2Gekljinf9EibTmi4Z/6qqaFZYW3YkN2Kruv2CApAQXJd7U9yaqz0 up8i1mIBlQ2MdXLY3HvLg3iZQ7mqUZDuyL6rG3Lwh5P0hwLF5yFHvQGxMOMN4ODWClr7 G8ZW10TErbPAqYmEjrIbfwYcls6AzlsztWe7X4G1QV0uIU4rvVtYPcN/xmJVXbKEuS9k sDgPtTSFI/EWAlrfSlLMTfGJCpOQtaP5A4BJjk0NOf2gMA3X05wVpa5LUsikgsjZ8uyw DVuQ== X-Gm-Message-State: ACrzQf1s9TPlLRXhn/GmS/m6bGcpbfqJGO/FsI9LY+24qF3QuTXDXJxT aBYDGSxAEwbTeX9sUza3WT4Fp8/GPvl3nKz1KM3ugA== X-Google-Smtp-Source: AMsMyM4fTr+YpWYF2fKoR5o8HOnVVrliYHgBNuLNNwnNsfbNO/5sykMfo+0j0bjbYXwYDZUfIAsvCTd1dcZcmy1Jw2s= X-Received: by 2002:a05:6638:370e:b0:374:4930:1389 with SMTP id k14-20020a056638370e00b0037449301389mr4241670jav.20.1666664931165; Mon, 24 Oct 2022 19:28:51 -0700 (PDT) MIME-Version: 1.0 References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> <26F033E5-2490-414E-8DFF-4ACD27B74075@apple.com> In-Reply-To: From: Neal Cardwell Date: Mon, 24 Oct 2022 22:28:34 -0400 Message-ID: To: Christoph Paasch Cc: Sebastian Moeller , Glenn Fishbine , Rpm , tsvwg IETF list , IETF IPPM WG , Dave Taht via Starlink , Measurement Analysis and Tools Working Group , discuss Content-Type: multipart/alternative; boundary="0000000000002f079705ebd2ab33" Subject: Re: [Starlink] [tsvwg] [Rpm] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2022 02:28:52 -0000 --0000000000002f079705ebd2ab33 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Oct 24, 2022 at 7:44 PM Christoph Paasch wrote: > > > On Oct 24, 2022, at 1:57 PM, Sebastian Moeller wrote: > > Hi Christoph > > On Oct 24, 2022, at 22:08, Christoph Paasch wrote: > > Hello Sebastian, > > On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink < > starlink@lists.bufferbloat.net> wrote: > > Hi Glenn, > > > On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm < > rpm@lists.bufferbloat.net> wrote: > > As a classic died in the wool empiricist, granted that you can identify > "misery" factors, given a population of 1,000 users, how do you propose > deriving a misery index for that population? > > We can measure download, upload, ping, jitter pretty much without user > intervention. For the measurements you hypothesize, how you you > automatically extract those indecies without subjective user contaminatio= n. > > I.e. my download speed sucks. Measure the download speed. > > My isp doesn't fix my problem. Measure what? How? > > Human survey technology is 70+ years old and it still has problems > figuring out how to correlate opinion with fact. > > Without an objective measurement scheme that doesn't require human > interaction, the misery index is a cool hypothesis with no way to link to > actual data. What objective measurements can be made? Answer that and t= he > index becomes useful. Otherwise it's just consumer whining. > > Not trying to be combative here, in fact I like the concept you support, > but I'm hard pressed to see how the concept can lead to data, and the dat= a > lead to policy proposals. > > > [SM] So it seems that outside of seemingly simple to test throughput > numbers*, the next most important quality number (or the most important > depending on subjective ranking) is how does latency change under "load". > Absolute latency is also important albeit static high latency can be work= ed > around within limits so the change under load seems more relevant. > All of flent's RRUL test, apple's networkQuality/RPM, and iperf2's > bounceback test offer methods to asses latency change under load**, as do > waveforms bufferbloat tests and even to a degree Ookla's speedtest.net. > IMHO something like latency increase under load or apple's responsiveness > measure RPM (basically the inverse of the latency under load calculated o= n > a per minute basis, so it scales in the typical higher numbers are better > way, unlike raw latency under load numbers where smaller is better). > IMHO what networkQuality is missing ATM is to measure and report the > unloaded RPM as well as the loaded the first gives a measure over the > static latency the second over how well things keep working if capacity > gets tight. They report the base RTT which can be converted to RPM. As an > example: > > macbook:~ user$ networkQuality -v > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > > Upload capacity: 24.341 Mbps > Download capacity: 91.951 Mbps > Upload flows: 20 > Download flows: 16 > Responsiveness: High (2123 RPM) > Base RTT: 16 > Start: 10/23/22, 13:44:39 > End: 10/23/22, 13:44:53 > OS Version: Version 12.6 (Build 21G115) > > > You should update to latest macOS: > > $ networkQuality > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 326.789 Mbps > Downlink capacity: 446.359 Mbps > Responsiveness: High (2195 RPM) > Idle Latency: 5.833 milli-seconds > > ;-) > > > > [SM] I wish... just updated to the latest and greatest for this hardware > (A1398): > > macbook-pro:DPZ smoeller$ networkQuality > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > > Upload capacity: 7.478 Mbps > Download capacity: 2.415 Mbps > Upload flows: 16 > Download flows: 20 > Responsiveness: Low (90 RPM) > macbook-pro:DPZ smoeller$ networkQuality -v > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > > Upload capacity: 5.830 Mbps > Download capacity: 6.077 Mbps > Upload flows: 12 > Download flows: 20 > Responsiveness: Low (56 RPM) > Base RTT: 134 > Start: 10/24/22, 22:47:48 > End: 10/24/22, 22:48:09 > OS Version: Version 12.6.1 (Build 21G217) > macbook-pro:DPZ smoeller$ > > Still, I only see the "Base RTT" with the -v switch and I am not sure > whether that is identical to your "Idle Latency". > > > I guess I need to convince my employer to exchange that macbook (actually > because the battery starts bulging and not because I am behind with > networkQuality versions ;) ) > > > Yes, you would need macOS Ventura to get the latest and greatest. > > But, what I read is: You are suggesting that =E2=80=9CIdle Latency=E2=80= =9D should be > expressed in RPM as well? Or, Responsiveness expressed in millisecond ? > > > [SM] Yes, I am fine with either (or both) the idea is to make it really > easy to see whether/how much "working conditions" deteriorate the > responsiveness / increase the latency-under-load. At least in verbose mod= e > it would be sweet if nwtworkQuality could expose that information. > > > I see - let me think about that=E2=80=A6 > +1 w/ Sebastian's point here. IMHO it would be great if the responsiveness under load and when idle were reported: (a) symmetrically, with the same metrics for both cases, and (b) in both RPM and ms terms for both cases So instead of: Responsiveness: High (2195 RPM) Idle Latency: 5.833 milli-seconds Perhaps something like: Loaded Responsiveness: High (XXXX RPM) Loaded Latency: X.XXX milli-seconds Idle Responsiveness: High (XXXX RPM) Idle Latency: X.XXX milli-seconds Having both RPM and ms available for loaded and unloaded cases would seem to make it easier to compare loaded and idle performance more directly and in a more apples-to-apples way. best, neal --0000000000002f079705ebd2ab33 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Oct 24, 2022 at 7:44 PM Chris= toph Paasch <cpaasch=3D40a= pple.com@dmarc.ietf.org> wrote:


On = Oct 24, 2022, at 1:57 PM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Christoph
On Oct 24, 2022, = at 22:08, Christoph Paasch <cpaasch@apple.com> wrote:

Hello Sebastian,
On Oct 23, 2022, at 4:57 AM, Sebastian Moeller v= ia Starlink <starlink@lists.bufferbloat.net> wrote:

Hi Glenn,
=

On Oct 23, 2022, at 02:17, Glenn Fishbine= via Rpm <rpm@lists.bufferbloat.net> wrote:

As a classic died in the w= ool empiricist, granted that you can identify "misery" factors, g= iven a population of 1,000 users, how do you propose deriving a misery inde= x for that population?

We can measure download, upload, ping, jitter= pretty much without user intervention.=C2=A0 For the measurements you hypo= thesize, how you you automatically extract those indecies without subjectiv= e user contamination.

I.e. =C2=A0my download speed sucks. Measure th= e download speed.

My isp doesn't fix my problem. Measure what? H= ow?

Human survey technology is 70+ years old and it still has proble= ms figuring out how to correlate opinion with fact.

Without an objec= tive measurement scheme that doesn't require human interaction, the mis= ery index is a cool hypothesis with no way to link to actual data.=C2=A0 Wh= at objective measurements can be made?=C2=A0 Answer that and the index beco= mes useful. Otherwise it's just consumer whining.

Not trying to = be combative here, in fact I like the concept you support, but I'm hard= pressed to see how the concept can lead to data, and the data lead to poli= cy proposals.

[SM] So it seems that outside of seemingly simple to test throughput nu= mbers*, the next most important quality number (or the most important depen= ding on subjective ranking) is how does latency change under "load&quo= t;. Absolute latency is also important albeit static high latency can be wo= rked around within limits so the change under load seems more relevant.=C2=A0
All of flen= t's RRUL test, apple's networkQuality/RPM, and iperf2's bounceb= ack test offer methods to asses latency change under load**, as do waveform= s bufferbloat tests and even to a degree Ookla's speedtest.net. IMHO something like latency= increase under load or apple's responsiveness measure RPM (basically t= he inverse of the latency under load calculated on a per minute basis, so i= t scales in the typical higher numbers are better way, unlike raw latency u= nder load numbers where smaller is better).
IMHO what networkQuality is missing ATM is to measure and = report the unloaded RPM as well as the loaded the first gives a measure ove= r the static latency the second over how well things keep working if capaci= ty gets tight. They report the base RTT which can be converted to RPM. As a= n example:

macbook:~ user$ networkQuality -v
=3D=3D=3D=3D SUMMARY= =3D=3D=3D=3D =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0
Upload capacity: 24.341 Mbps
Download capacity: 91= .951 Mbps
Upload flows: 20
Download flows: 16
Responsiveness: High= (2123 RPM)
Base RTT: 16
Start: 10/23/22, 13:44:39
End: 10/23/22, = 13:44:53
OS Version: Version 12.6 (Build 21G115)

You= should update to latest macOS:

$ networkQuality
=3D=3D=3D=3D SUM= MARY =3D=3D=3D=3D
Uplink capacity: 326.789 Mbps
Downlink capacity: 44= 6.359 Mbps
Responsiveness: High (2195 RPM)
Idle Latency: 5.833 milli-= seconds

;-)



[SM= ] I wish... just updated to the latest and greatest for this hardware (A139= 8):

macbook-pro:DPZ smoeller$ networkQuality
=3D=3D=3D=3D SUMMARY =3D=3D= =3D=3D =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0
Upload capacity: 7.478 Mbps
Download capacity: 2.415 Mbps
Upload flows: 16Download flows:= 20
Res= ponsiveness: Low (90 RPM)
macbook-pro:DPZ smoeller$ networkQuality -v
=3D=3D=3D=3D SUMMAR= Y =3D=3D=3D=3D =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0
Upload capacity: 5.830 Mbps
Download capacity: 6.077 Mbps
Upload flows: 12
Download= flows: 20
Responsiveness: Low (56 RPM)
Base RTT: 134
Start: 10/24/22, 22:47:48
End: 10/24/22, 22:48:0= 9
OS Ve= rsion: Version 12.6.1 (Build 21G217)
macbook-pro:DPZ smoeller$=C2=A0<= /span>

Still, I only see the "Base RTT" with the -v switch and= I am not sure whether that is identical to your "Idle Latency".<= /span>


I guess I need to convince my employer to exchan= ge that macbook (actually because the battery starts bulging and not becaus= e I am behind with networkQuality versions ;) )

Yes, you would need macOS Ventura to get= the latest and greatest.

But, what I read is: You are suggesting that = =E2=80=9CIdle Latency=E2=80=9D should be expressed in RPM as well? Or, Resp= onsiveness expressed in millisecond ?

[SM] = Yes, I am fine with either (or both) the idea is to make it really easy to = see whether/how much "working conditions" deteriorate the respons= iveness / increase the latency-under-load. At least in verbose mode it woul= d be sweet if nwtworkQuality could expose that information.

I see - let me think about t= hat=E2=80=A6

+1 w/ Sebast= ian's point here. IMHO it would be great if the responsiveness under lo= ad and when idle were reported:

=C2=A0 (a) symmetr= ically, with the same metrics for both cases, and=C2=A0

=C2=A0 (b) in both RPM and ms terms for both cases

=
So instead of:

Responsiveness: High (21= 95 RPM)
Idle Latency: 5.833 milli-seconds

Perhaps something like:

Loaded Responsiveness: H= igh (XXXX RPM)
Loaded Latency: X.XXX milli-seconds
Idle= Responsiveness: High (XXXX RPM)
Idle Latency: X.XXX milli-second= s=C2=A0

Having both RPM and ms available for loade= d and unloaded cases would seem to make it easier to compare loaded and idl= e performance more directly and in a more apples-to-apples way.
<= br>
best,
neal


--0000000000002f079705ebd2ab33--