From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 9D0D03CB37 for ; Mon, 31 Oct 2022 14:52:39 -0400 (EDT) Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id C0AF01B277; Mon, 31 Oct 2022 11:52:38 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com C0AF01B277 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1667242358; bh=8GZC3YPvJbHhttlLDGAY6TJfTxfEHMLLUQsfLxNtSK0=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=IC1CSI22qPVwpmgTm2XmLG4oYkbqjfozy7E5RaxOFvxatnyqR8/7F/Wbh32SmrNbA ZhjL+GKSFWHMYl93IFkM3uBz8z7x7rm2PtaUvmoKZgoNwPKOS4f/n/2fGnw5CQiad2 L4XwRTseI6bYJ6H2wP9WG9WcgwSqdtwnkGuvafSM= MIME-Version: 1.0 Date: Mon, 31 Oct 2022 11:52:38 -0700 From: rjmcmahon To: Dave Taht Cc: "MORTON JR., AL" , Rpm , ippm@ietf.org In-Reply-To: References: Message-ID: <0a8cc31c7077918bf84fddf9db50db02@rjmcmahon.com> X-Sender: rjmcmahon@rjmcmahon.com Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Rpm] [ippm] Preliminary measurement comparison of "Working Latency" metrics X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Oct 2022 18:52:39 -0000 Would it be possible to get some iperf 2 bounceback test results too? https://sourceforge.net/projects/iperf2/ Also, for the hunt algo, maybe use TCP first to get a starting point and then hunt? Just a thought. Thanks, Bob > Thank you very much for the steer to RFC9097. I'd completely missed > that. > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL > wrote: >> >> (astute readers may have guessed that I pressed "send" too soon on >> previous message...) >> >> I also conducted upstream tests this time, here are the results: >> (capacity in Mbps, delays in ms, h and m are RPM categories, High and >> Medium) >> >> Net Qual UDPST (RFC9097) Ookla >> UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap >> Ping(no load) >> 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 >> 8 >> 22 281 m 214ms 8ms 27 (52) 25 5-248 22 >> 8 >> 22 290 m 207ms 8ms 27 (55) 28 0-253 22 >> 9 >> 21 330 m 182ms 11ms 23 (44) 28 0-255 22 >> 7 >> 22 334 m 180ms 9ms 33 (56) 25 0-255 22 >> 9 >> >> The Upstream capacity measurements reflect an interesting feature that >> we can reliably and repeatably measure with UDPST. The first ~3 >> seconds of upstream data experience a "turbo mode" of ~50Mbps. UDPST >> displays this behavior in its 1 second sub-interval measurements and >> has a bimodal reporting option that divides the complete measurement >> interval in two time intervals to report an initial (turbo) max >> capacity and a steady-state max capacity for the later intervals. The >> UDPST capacity results present both measurements: steady-state first. > > Certainly we can expect bi-model distributions from many ISPs, as, for > one thing, the "speedboost" concept remains popular, except that it's > misnamed, as it should be called speed-subtract or speed-lose. Worse, > it is often configured "sneakily", in that it doesn't kick in for the > typical observed duration of the test, for some, they cut the > available bandwidth about 20s in, others, 1 or 5 minutes. > > One of my biggest issues with the rpm spec so far is that it should, > at least, sometimes, run randomly longer than the overly short > interval it runs for and the tools also allow for manual override of > length. > > we caught a lot of tomfoolery with flent's rrul test running by default > for 1m. > > Also, AQMs on the path can take a while to find the optimal drop or > mark rate. > >> >> The capacity processing in networkQuality and Ookla appear to report >> the steady-state result. > > Ookla used to basically report the last result. Also it's not a good > indicator of web traffic behavior at all, watching the curve > go up much more slowly in their test on say, fiber 2ms, vs starlink, > (40ms).... > > So adding another mode - how quickly is peak bandwidth actually > reached, would be nice. > > I haven't poked into the current iteration of the goresponsiveness > test at all: https://github.com/network-quality/goresponsiveness it > would be good to try collecting more statistics and histograms and > methods of analyzing the data in that libre-source version. > > How does networkQuality compare vs a vs your tool vs a vs > goresponsiveness? > >> I watched the upstream capacity measurements on the Ookla app, and >> could easily see the initial rise to 40-50Mbps, then the drop to >> ~22Mbps for most of the test which determined the final result. > > I tend to get upset when I see ookla's new test flash a peak result in > the seconds and then settle on some lower number somehow. > So far as I know they are only sampling the latency every 250ms. > >> >> The working latency is about 200ms in networkQuality and about 280ms >> as measured by UDPST (RFC9097). Note that the networkQuality minimum >> delay is ~20ms lower than the UDPST RTTmin, so this accounts for some >> of the difference in working latency. Also, we used the very dynamic >> Type C load adjustment/search algorithm in UDPST during all of this >> testing, which could explain the higher working latency to some >> degree. >> >> So, it's worth noting that the measurements needed for assessing >> working latency/responsiveness are available in the UDPST utility, and >> that the UDPST measurements are conducted on UDP transport (used by a >> growing fraction of Internet traffic). > > Thx, didn't know of this work til now! > > have you tried irtt? > >> >> comments welcome of course, >> Al >> >> > -----Original Message----- >> > From: ippm On Behalf Of MORTON JR., AL >> > Sent: Sunday, October 30, 2022 8:09 PM >> > To: ippm@ietf.org >> > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency" >> > metrics >> > >> > >> > Hi again RPM friends and IPPM'ers, >> > >> > As promised, I repeated the tests shared last week, this time using both the >> > verbose (-v) and sequential (-s) dwn/up test options of networkQuality. I >> > followed Sebastian's calculations as well. >> > >> > Working Latency & Capacity Summary >> > >> > Net Qual UDPST Ookla >> > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap >> > Ping(no load) >> > 885 916 m 66ms 8ms 970 28 0-20 940 8 >> > 888 1355 h 44ms 8ms 966 28 0-23 940 8 >> > 891 1109 h 54ms 8ms 968 27 0-19 940 9 >> > 887 1141 h 53ms 11ms 966 27 0-18 937 7 >> > 884 1151 h 52ms 9ms 968 28 0-20 937 9 >> > >> > With the sequential test option, I noticed that networkQuality achieved nearly >> > the maximum capacity reported almost immediately at the start of a test. >> > However, the reported capacities are low by about 60Mbps, especially when >> > compared to the Ookla TCP measurements. >> > >> > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high end of the >> > RTTrange), for example 54ms compared to (27+19=46). Most of the networkQuality >> > RPM measurements were categorized as "High". There doesn't seem to be much >> > buffering in the downstream direction. >> > >> > >> > >> > > -----Original Message----- >> > > From: ippm On Behalf Of MORTON JR., AL >> > > Sent: Monday, October 24, 2022 6:36 PM >> > > To: ippm@ietf.org >> > > Subject: [ippm] Preliminary measurement comparison of "Working Latency" >> > > metrics >> > > >> > > >> > > Hi RPM friends and IPPM'ers, >> > > >> > > I was wondering what a comparison of some of the "working latency" metrics >> > > would look like, so I ran some tests using a service on DOCSIS 3.1, with the >> > > downlink provisioned for 1Gbps. >> > > >> > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla >> > Speedtest >> > > with as similar connectivity as possible (but we know that the traffic will >> > > diverge to different servers and we can't change that aspect). >> > > >> > > Here's a quick summary of yesterday's results: >> > > >> > > Working Latency & Capacity Summary >> > > >> > > Net Qual UDPST Ookla >> > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(no load) >> > > 878 62 970 28 0-19 941 6 >> > > 891 92 970 27 0-20 940 7 >> > > 891 120 966 28 0-22 937 9 >> > > 890 112 970 28 0-21 940 8 >> > > 903 70 970 28 0-16 935 9 >> > > >> > > Note: all RPM values were categorized as Low. >> > > >> > > networkQuality downstream capacities are always on the low side compared to >> > > others. We would expect about 940Mbps for TCP, and that's mostly what Ookla >> > > achieved. I think that a longer test duration might be needed to achieve the >> > > actual 1Gbps capacity with networkQuality; intermediate values observed were >> > > certainly headed in the right direction. (I recently upgraded to Monterey >> > 12.6 >> > > on my MacBook, so should have the latest version.) >> > > >> > > Also, as Sebastian Moeller's message to the list reminded me, I should have >> > > run the tests with the -v option to help with comparisons. I'll repeat this >> > > test when I can make time. >> > > >> > > The UDPST measurements of RTTmin (minimum RTT observed during the test) and >> > > the range of variation above the minimum (RTTVarRnge) add-up to very >> > > reasonable responsiveness IMO, so I'm not clear why RPM graded this access >> > and >> > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago >> > > conducting tests, so the minimum 28ms is typical. UDPST measurements were >> > run >> > > on an Ubuntu VM in my MacBook. >> > > >> > > The big disappointment was that the Ookla desktop app I updated over the >> > > weekend did not include the new responsiveness metric! I included the ping >> > > results anyway, and it was clearly using a server in the nearby area. >> > > >> > > So, I have some more work to do, but I hope this is interesting-enough to >> > > start some comparison discussions, and bring-out some suggestions. >> > > >> > > happy testing all, >> > > Al >> > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > ippm mailing list >> > > ippm@ietf.org >> > > >> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd >> > > >> > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_xKqg4$ >> > >> > _______________________________________________ >> > ippm mailing list >> > ippm@ietf.org >> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd >> > T!g-FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw$ >> >> _______________________________________________ >> ippm mailing list >> ippm@ietf.org >> https://www.ietf.org/mailman/listinfo/ippm