From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AF78C3B2A4; Sun, 23 Oct 2022 07:57:23 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1666526238; bh=TtmvplZmgfznMx/QLphc1Z+dHa8zw2vtupW9RotrlTg=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=kNOsvNd1lyyTG6RJKbgrzspBa+fd8bvh0rQGLXpew+iqXc+1KZ4tEMAdmbVdPtTuo TQebih+hVaMFcWph2Uv1r1o0gfDtNz6FZ08gAkFjFf1Wd/zrlZFI8YpkrSLxzjcj6z GS5x8UV9JRwk8GzBE0QbrFkMFoAxLz+MZChns9cM= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([95.116.66.209]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M42jQ-1omZbS2jI0-0003P0; Sun, 23 Oct 2022 13:57:18 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Sun, 23 Oct 2022 13:57:16 +0200 Cc: =?utf-8?Q?Dave_T=C3=A4ht?= , Dave Taht via Starlink , tsvwg IETF list , IETF IPPM WG , Rpm , Measurement Analysis and Tools Working Group , discuss Content-Transfer-Encoding: quoted-printable Message-Id: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> References: To: Glenn Fishbine X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:GmUjbopl6lwvEf4OaadlAZvzGa3Mz6tNjDgTBElMwyCelyjoERP hlCX5g+Jtapz6djHtJ6OfimPYtpODYkcHJ98c/d9YcRLtJuX4ieOQ4ZxA/+lFQFpwjiOvRm yhOeOf771gvLdJN1nl1rQj/5fZs2wKmDV3tcd2bQWCMArDts1jWbzusxs6u9IFMF9vEj91K qlZiK/a1n13z/AXjdrrvQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:fYRxdp5epS4=:sFKWaHJpCpQjLrf7W4MdRU ZtEB1Amxm275aq8gK4ITOHVu1JIi2Txk/sMLSOS3G7HA5SfQCp1wmY3uOx3ZBj9YEbGaw7MOM lI6AZ9xTPcUJU2QS+0wYjbWw1v6TFtdPnvksuFsfgatatZshxfFQI2rA9bwC4IN+J2MZ7ZtIN n5wj6GXkgenXNCFEg6MRJOQ2/YiFxa5MoOZ+TEQQu6GXcrpQtq6Zc/f3+aHikn1g7xCCKd93i E1xlAcIPY+iOzirLf3N+/pmkzRl/ClsLHZdzqvGIgpxbXLLDROpWaOeH05Ta2x/0NN6RsHDhb lsTPRFyM79xkFz9jlXe14VFoLOKBDT/4oupT0Y4winSv/hWXkADAhneWIMMhEuARq8lLlimLZ j2H0DrAXvNJvGPZp249h4QlTmVit1l4isw0ki0R2+W7iBYGJsEJoNT1qzz9PCe4k0Qd9G6KSC WbqfgPYsNt0wtklhmPhqjQ3E0n6GtwymkY/MFXgsi9XJbF4R+xisAXvqKULwZvLa3VYxIZ9CL vM/vr/honEJJM2xFtaW5y8dhrf+Ws7eMN4QeVwudyyuAQalrgSmtchvZyCLbPpJQp6LB4i+K3 Px2nX1j9AufIRHpv6LEM46ljJR8Zk7M+hMHGHLlIkg8s0aX66ukmv/9GkAhPWsUoWoUmNlIUr 1ABd+W98nSIoNUdehzWhczwrS8ltgQ7ACd/twZEA3d/7tC5DwbzMQtYxreuIMm9LgMUtQISh9 ojakrsGWHB7qs0vpn76ceLQcccM8QnKX3XFBN7coYqHQqaAglPurR5vbwyAJKDWbXWlSFOtju yi2a64+2fcer55NUZ0l445oLwn5L0wAZAih0TmXyvqKokGl+vFMKuFmTug0m59JffFcXSpimt DW0pcGu09wYOqglfV2JDGf7oeToFehbRO55fOE55gZe6w2Ajpn13MY80VqL1VDjRpI4ILh//z aK7XW+MT1FsuFpdq9mE9Dux51eFkkyYFkEUtB5XBLmB6RAKXU3P/scnvUruqtIjPSttbZAVGL LfOxyfalZ7X2wJWYTQZk5EzsHBKgT2dGFkOihs2ZpICbf5j/RGdXBLZ0EUS7klvQvjrb+yO40 h7CIDHt5dUSWxHLAsO1eBL5hDFF0vJ/CAF5G9pQzOiCz/ZQDkwDk8Vwvst+tAXYit0H8/wygA /mkr7MbeYh4l55QfK7mnMIyTuawXFrl0xdbxqHzD+XGz/F+GFgM627Fz4/nrpau3njObkvtIY cpZtGW1/EpYAVt/CiArgSZo3fP5McFVi6ixyqnvZWyzeuSctf8al5gfXppoL8dZhCCn0I1a+x zYniYpWTybwt1gZ5BPZbPLxK/hqebNwSw3EpvyYX6zI7lRdKKJWcMLJHGlhl/Xe9TQSvDM53L hTuNkcmRyNDKnElH4ZAFT1a0rn8hrxL29/PB3jKkn0/EH+/hOl3QyXZcyv4IvCoAMaaj0/0Cq j/RMLntlIBnsZFRe4zRbfmCsvwi52z25hc94LkIAjOAyY7UymwnjOqF7H4pstd4B9ixbLtN4c erKPvRqW3mqrboehMynKCyKPV07DR/zNlwLhfAqRq2XuG Subject: Re: [Rpm] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2022 11:57:24 -0000 Hi Glenn, > On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm = wrote: >=20 > As a classic died in the wool empiricist, granted that you can = identify "misery" factors, given a population of 1,000 users, how do you = propose deriving a misery index for that population? >=20 > We can measure download, upload, ping, jitter pretty much without user = intervention. For the measurements you hypothesize, how you you = automatically extract those indecies without subjective user = contamination. >=20 > I.e. my download speed sucks. Measure the download speed. >=20 > My isp doesn't fix my problem. Measure what? How? >=20 > Human survey technology is 70+ years old and it still has problems = figuring out how to correlate opinion with fact. >=20 > Without an objective measurement scheme that doesn't require human = interaction, the misery index is a cool hypothesis with no way to link = to actual data. What objective measurements can be made? Answer that = and the index becomes useful. Otherwise it's just consumer whining. >=20 > Not trying to be combative here, in fact I like the concept you = support, but I'm hard pressed to see how the concept can lead to data, = and the data lead to policy proposals. [SM] So it seems that outside of seemingly simple to test = throughput numbers*, the next most important quality number (or the most = important depending on subjective ranking) is how does latency change = under "load". Absolute latency is also important albeit static high = latency can be worked around within limits so the change under load = seems more relevant.=20 All of flent's RRUL test, apple's networkQuality/RPM, and = iperf2's bounceback test offer methods to asses latency change under = load**, as do waveforms bufferbloat tests and even to a degree Ookla's = speedtest.net. IMHO something like latency increase under load or = apple's responsiveness measure RPM (basically the inverse of the latency = under load calculated on a per minute basis, so it scales in the typical = higher numbers are better way, unlike raw latency under load numbers = where smaller is better). IMHO what networkQuality is missing ATM is to measure and report = the unloaded RPM as well as the loaded the first gives a measure over = the static latency the second over how well things keep working if = capacity gets tight. They report the base RTT which can be converted to = RPM. As an example: macbook:~ user$ networkQuality -v =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 Upload capacity: 24.341 Mbps Download capacity: 91.951 Mbps Upload flows: 20 Download flows: 16 Responsiveness: High (2123 RPM) Base RTT: 16 Start: 10/23/22, 13:44:39 End: 10/23/22, 13:44:53 OS Version: Version 12.6 (Build 21G115) Here RPM 2133 corresponds to 60000/2123 =3D 28.26 ms latency under load, = while the Base RTT of 16ms corresponds to 60000/16 =3D 3750 RPM, son on = this link load reduces the responsiveness by 3750-2123 =3D 1627 RPM a = reduction by 100-100*2123/3750 =3D 43.4%, and that is with competent AQM = and scheduling on the router. Without competent AQM/shaping I get: =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 Upload capacity: 15.101 Mbps Download capacity: 97.664 Mbps Upload flows: 20 Download flows: 12 Responsiveness: Medium (427 RPM) Base RTT: 16 Start: 10/23/22, 13:51:50 End: 10/23/22, 13:52:06 OS Version: Version 12.6 (Build 21G115) latency under load: 60000/427 =3D 140.52 ms=20 base RPM: 60000/16 =3D 3750 RPM reduction RPM: 100-100*427/3750 =3D 88.6% I understand apple's desire to have a single reported number with a = single qualifier medium/high/... because in the end a link is only = reliably usable if responsiveness under load stays acceptable, but with = two numbers it is easier to see what one's ISP could do to help. (I = guess some ISPs might already be unhappy with the single number, so this = needs some diplomacy/tact) Regards Sebastian *) Seemingly as quite some ISPs operate their own speedtest servers in = their network and ignore customers not reaching the contracted rates = into speedtest-servers located in different ASs. As the product is = called internet access I a inclined to expect that my ISP maintains = sufficient peering/transit capacity to reach the next tier of AS at my = contracted rate (the EU legislative seems to agree, see EU directive = 2015/2120). **) Most do by creating load themselves and measuring throughput at the = same time, bounceback IIUC will focus on the latency measurement and = leave the load generation optional (so offers a mode to measure = responsiveness of a live network with minimal measurement traffic). = @Bob, please correct me if this is wrong. >=20 >=20 > On Fri, Oct 21, 2022, 5:20 PM Dave Taht wrote: > One of the best talks I've ever seen on how to measure customer > satisfaction properly just went up after the P99 Conference. >=20 > It's called Misery Metrics. >=20 > After going through a deep dive as to why and how we think and act on > percentiles, bins, and other statistical methods as to how we use the > web and internet are *so wrong* (well worth watching and thinking > about if you are relying on or creating network metrics today), it > then points to the real metrics that matter to users and the ultimate > success of an internet business: Timeouts, retries, misses, failed > queries, angry phone calls, abandoned shopping carts and loss of > engagement. >=20 > https://www.p99conf.io/session/misery-metrics-consequences/ >=20 > The ending advice was - don't aim to make a specific percentile > acceptable, aim for an acceptable % of misery. >=20 > I enjoyed the p99 conference more than any conference I've attended in = years. >=20 > --=20 > This song goes out to all the folk that thought Stadia would work: > = https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665= 607352320-FXtz > Dave T=C3=A4ht CEO, TekLibre, LLC >=20 > --=20 > You received this message because you are subscribed to the Google = Groups "discuss" group. > To unsubscribe from this group and stop receiving emails from it, send = an email to discuss+unsubscribe@measurementlab.net. > To view this discussion on the web visit = https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27= a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com. > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm