From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cnet.fi.uba.ar (cnet.fi.uba.ar [157.92.58.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8A5913B29D; Tue, 25 Oct 2022 12:07:12 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by cnet.fi.uba.ar (Postfix) with ESMTP id D623F140077; Tue, 25 Oct 2022 13:07:06 -0300 (ART) X-Virus-Scanned: Debian amavisd-new at cnet.fi.uba.ar Received: from cnet.fi.uba.ar ([127.0.0.1]) by localhost (cnet.fi.uba.ar [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nb1Wn2X0X7kn; Tue, 25 Oct 2022 13:07:00 -0300 (ART) Received: from smtpclient.apple (unknown [157.92.5.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by cnet.fi.uba.ar (Postfix) with ESMTPSA id ECD49140068; Tue, 25 Oct 2022 13:06:59 -0300 (ART) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: J Ignacio Alvarez-Hamelin In-Reply-To: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> Date: Tue, 25 Oct 2022 13:07:01 -0300 Cc: Glenn Fishbine , Rpm , tsvwg IETF list , IETF IPPM WG , Dave Taht via Starlink , Measurement Analysis and Tools Working Group , discuss Content-Transfer-Encoding: quoted-printable Message-Id: References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> To: Sebastian Moeller X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Mailman-Approved-At: Tue, 01 Nov 2022 17:22:28 -0400 Subject: Re: [Starlink] [ippm] [Rpm] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2022 16:07:12 -0000 Dear all,=20 After some time in silence on the IPPM list, I like to make some = comments here. As we presented in the draft-ietf-ippm-route-00 (now the = RFC9198), the main problem is the traffic follows heavy-tailed = distributions when it is seen from the end-to-end points: the origin of = most of the issues in that video. Therefore, treating it as parametric = distribution is not possible, unless you are dealing with a complex = distribution like the Stable distribution: =E2=80=A2 B. Mandelbrot, =E2=80=9CNew methods in statistical = economics,=E2=80=9D Journal of political economy, vol. 71, no. 5, pp. = 421=E2=80=93440, 1963. =E2=80=A2 =E2=80=94=E2=80=94, =E2=80=9CThe variation of certain = speculative prices,=E2=80=9D The journal of business, vol. 36, no. 4, = pp. 394=E2=80=93419, 1963. (and so it will be extremely complex a high computing demands.) This is why we propose to use quartiles to characterize delays in the = RFC9198. Then, I am doing some research to understand how the delay can = change with network load, using the quartiles.=20 You can see some measurements done during the pandemic, showing the = congestion as a function of the time (24 hours maximum): = https://cnet.fi.uba.ar/ignacio.alvarez-hamelin/RIPE-Atlas-measurement-2468= 1441_m_win_data_world_map.html [you can zoom in and out, pan it, clicking on the Xs you can close = dialogs, to reopen them click on the link] Best, Ignacio ___________________________________ _______________________________________________________________ Dr. Ing. Jos=C3=A9 Ignacio Alvarez-Hamelin CONICET and Facultad de Ingenier=C3=ADa, Universidad de Buenos Aires Av. Paseo Col=C3=B3n 850 - C1063ACV - Buenos Aires - Argentina +54 (11) 5285 0716 / 5285 0705 e-mail: ihameli@cnet.fi.uba.ar web: http://cnet.fi.uba.ar/ignacio.alvarez-hamelin/ _______________________________________________________________ > On 23 Oct 2022, at 08:57, Sebastian Moeller wrote: >=20 > Hi Glenn, >=20 >=20 >> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm = wrote: >>=20 >> As a classic died in the wool empiricist, granted that you can = identify "misery" factors, given a population of 1,000 users, how do you = propose deriving a misery index for that population? >>=20 >> We can measure download, upload, ping, jitter pretty much without = user intervention. For the measurements you hypothesize, how you you = automatically extract those indecies without subjective user = contamination. >>=20 >> I.e. my download speed sucks. Measure the download speed. >>=20 >> My isp doesn't fix my problem. Measure what? How? >>=20 >> Human survey technology is 70+ years old and it still has problems = figuring out how to correlate opinion with fact. >>=20 >> Without an objective measurement scheme that doesn't require human = interaction, the misery index is a cool hypothesis with no way to link = to actual data. What objective measurements can be made? Answer that = and the index becomes useful. Otherwise it's just consumer whining. >>=20 >> Not trying to be combative here, in fact I like the concept you = support, but I'm hard pressed to see how the concept can lead to data, = and the data lead to policy proposals. >=20 > [SM] So it seems that outside of seemingly simple to test = throughput numbers*, the next most important quality number (or the most = important depending on subjective ranking) is how does latency change = under "load". Absolute latency is also important albeit static high = latency can be worked around within limits so the change under load = seems more relevant.=20 > All of flent's RRUL test, apple's networkQuality/RPM, and = iperf2's bounceback test offer methods to asses latency change under = load**, as do waveforms bufferbloat tests and even to a degree Ookla's = speedtest.net. IMHO something like latency increase under load or = apple's responsiveness measure RPM (basically the inverse of the latency = under load calculated on a per minute basis, so it scales in the typical = higher numbers are better way, unlike raw latency under load numbers = where smaller is better). > IMHO what networkQuality is missing ATM is to measure and report = the unloaded RPM as well as the loaded the first gives a measure over = the static latency the second over how well things keep working if = capacity gets tight. They report the base RTT which can be converted to = RPM. As an example: >=20 > macbook:~ user$ networkQuality -v > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 24.341 Mbps > Download capacity: 91.951 Mbps > Upload flows: 20 > Download flows: 16 > Responsiveness: High (2123 RPM) > Base RTT: 16 > Start: 10/23/22, 13:44:39 > End: 10/23/22, 13:44:53 > OS Version: Version 12.6 (Build 21G115) >=20 > Here RPM 2133 corresponds to 60000/2123 =3D 28.26 ms latency under = load, while the Base RTT of 16ms corresponds to 60000/16 =3D 3750 RPM, = son on this link load reduces the responsiveness by 3750-2123 =3D 1627 = RPM a reduction by 100-100*2123/3750 =3D 43.4%, and that is with = competent AQM and scheduling on the router. >=20 > Without competent AQM/shaping I get: > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 > Upload capacity: 15.101 Mbps > Download capacity: 97.664 Mbps > Upload flows: 20 > Download flows: 12 > Responsiveness: Medium (427 RPM) > Base RTT: 16 > Start: 10/23/22, 13:51:50 > End: 10/23/22, 13:52:06 > OS Version: Version 12.6 (Build 21G115) > latency under load: 60000/427 =3D 140.52 ms=20 > base RPM: 60000/16 =3D 3750 RPM > reduction RPM: 100-100*427/3750 =3D 88.6% >=20 >=20 > I understand apple's desire to have a single reported number with a = single qualifier medium/high/... because in the end a link is only = reliably usable if responsiveness under load stays acceptable, but with = two numbers it is easier to see what one's ISP could do to help. (I = guess some ISPs might already be unhappy with the single number, so this = needs some diplomacy/tact) >=20 > Regards > Sebastian >=20 >=20 >=20 > *) Seemingly as quite some ISPs operate their own speedtest servers in = their network and ignore customers not reaching the contracted rates = into speedtest-servers located in different ASs. As the product is = called internet access I a inclined to expect that my ISP maintains = sufficient peering/transit capacity to reach the next tier of AS at my = contracted rate (the EU legislative seems to agree, see EU directive = 2015/2120). >=20 > **) Most do by creating load themselves and measuring throughput at = the same time, bounceback IIUC will focus on the latency measurement and = leave the load generation optional (so offers a mode to measure = responsiveness of a live network with minimal measurement traffic). = @Bob, please correct me if this is wrong. >=20 >=20 >>=20 >>=20 >> On Fri, Oct 21, 2022, 5:20 PM Dave Taht wrote: >> One of the best talks I've ever seen on how to measure customer >> satisfaction properly just went up after the P99 Conference. >>=20 >> It's called Misery Metrics. >>=20 >> After going through a deep dive as to why and how we think and act on >> percentiles, bins, and other statistical methods as to how we use the >> web and internet are *so wrong* (well worth watching and thinking >> about if you are relying on or creating network metrics today), it >> then points to the real metrics that matter to users and the ultimate >> success of an internet business: Timeouts, retries, misses, failed >> queries, angry phone calls, abandoned shopping carts and loss of >> engagement. >>=20 >> https://www.p99conf.io/session/misery-metrics-consequences/ >>=20 >> The ending advice was - don't aim to make a specific percentile >> acceptable, aim for an acceptable % of misery. >>=20 >> I enjoyed the p99 conference more than any conference I've attended = in years. >>=20 >> --=20 >> This song goes out to all the folk that thought Stadia would work: >> = https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665= 607352320-FXtz >> Dave T=C3=A4ht CEO, TekLibre, LLC >>=20 >> --=20 >> You received this message because you are subscribed to the Google = Groups "discuss" group. >> To unsubscribe from this group and stop receiving emails from it, = send an email to discuss+unsubscribe@measurementlab.net. >> To view this discussion on the web visit = https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27= a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com. >> _______________________________________________ >> Rpm mailing list >> Rpm@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm >=20 > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm