From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BE27E3CB37 for ; Mon, 31 Oct 2022 19:10:44 -0400 (EDT) Received: by mail-wm1-x335.google.com with SMTP id f16-20020a05600c491000b003cf66a2e7c0so4312410wmp.5 for ; Mon, 31 Oct 2022 16:10:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=iwRUswWoxoyUbVNEGGqKC/EZSmHBXjJRp2JJJaEGPi0=; b=TeS15DISEnNzsgbHKlv3ooiDa70/Y3D6BwsIP+GcApm27ISqM0858HA0gfsyapanIt 2jmCU0eWGc7jvnvBbzS1OxG8IYaemLQ3sImu9xedoR+zu7ZFDz4g4YcDc4X5oxRakBPU CVnIPD5RBDTLt3w40XXpF/cUxOVjnqTTmgqNe9fWot5ewnCvnlGBYCUMB/JTE8iStwvX G5foJz+EfF33i2QUZ4XBQbrtVmHmMWT66z4utQsEhwL0TO2hVHVULynjfZ/CagyiW2xM FAtzpTW3xSB4X+H6IgL4abIKa2x+938BcNIiRSUViyQ9+2YvluA8ShY9DCZxssfzyJIE gymw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iwRUswWoxoyUbVNEGGqKC/EZSmHBXjJRp2JJJaEGPi0=; b=KUONtobVoEESTo5s9Jc3tzSrq7vVvZFb/bbESRZzBJ66+LLGHAIo1FN4UyDKJ2Mtwd fYiuDnkX8/n6N+zj9/DOAC/5iwjtLkEEJIWwI+PDz7yCHj8hQP+SmS5gm8rPT3HJ/Qad 7MBkLTu+FvkyhVjXFl0zotUEFHKH/1cL+RmpVZA1r3X939dQaSxpELYZbI7BcykSyaOT pMFlCwrpX/bAQBG3t23OaEJ+su6K8SwWao2cWpeRel+XBFCLtBYgbh93V1KIXADyq2/k f7ODbsxJ6FlNSlxl9Id4QvTqV/I7aC2sfY7uwJeKtOyRzxipeY3ekW3G33ixXH3wJYrM GkrA== X-Gm-Message-State: ACrzQf1XOScD7G0Q27acHrIYhl//cAcuQmpwynvkfmGiUldPzNduh2xL 8+DwWGY1xqLxahIQPcLjpw9nsvosHnkRGgnoOuVKJUZNlMQ= X-Google-Smtp-Source: AMsMyM6G/zsQ6tWaGAcgIx8ASPg/B3YmBg+dGOjlOTKSqgd1Yi10f1/a4YnqZ06350l1yhxjoAHLxufXQaTRaHSYSVo= X-Received: by 2002:a05:600c:1906:b0:3c6:f83e:d15f with SMTP id j6-20020a05600c190600b003c6f83ed15fmr19919172wmq.205.1667257843086; Mon, 31 Oct 2022 16:10:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Mon, 31 Oct 2022 16:10:31 -0700 Message-ID: To: libreqos Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [LibreQoS] Fwd: [ippm] Preliminary measurement comparison of "Working Latency" metrics X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Oct 2022 23:10:45 -0000 I'd be rather interested in a result from https://github.com/BroadbandForum/obudpst over a libreqos'd network. The IPPM meeting at ietf was rather contentious, I'm told. A bit more below: ---------- Forwarded message --------- From: MORTON JR., AL Date: Mon, Oct 31, 2022 at 1:41 PM Subject: RE: [ippm] Preliminary measurement comparison of "Working Latency" metrics To: Dave Taht Cc: ippm@ietf.org , Rpm Thanks for your read/reply/suggestions, Dave! Allow me to pull a few of your comments forward to try for a more cogent re= ply. Dave wrote: > Thank you very much for the steer to RFC9097. I'd completely missed that. You're quite welcome! We all have a hand on the elephant with eyes closed, and only learn the whole story when we talk to each other... The repo for UDPST is here: https://github.com/BroadbandForum/obudpst We are also working to standardize the protocol the UDPST uses to measure RFC 9097: https://datatracker.ietf.org/doc/html/draft-ietf-ippm-capacity-protocol-03 and potentially many aspects of network and application performance. Dave wrote: > Certainly we can expect bi-modal distributions ... should be called ... s= peed-lose. Agreed. I'm glad we added the bimodal analysis feature in UDPST. It works with our max test duration (60s, we didn't want people to run capacity tests ad nauseum), but we won't be able to detect speed-loose beyond that. Dave wrote: > One of my biggest issues with the rpm spec so far is that it should, > at least, sometimes, run randomly longer than the overly short interval <= now> ... We don't have adaptable duration either. Another perspective on duration comes from folks who test paths with mobile access: they prefer 5-7 second duration, and the Type C algo helps. Dave wrote: > So adding another mode - how quickly is peak bandwidth actually > reached, would be nice. I think we report this in our prolific JSON-formatted output (#sub-interval with the Max Cap). Dave wrote: > How does networkQuality compare vs a vs your tool vs a vs goresponsivenes= s? I'll try to install goresponsiveness later this week, so that we have a view of this. Dave wrote: > have you tried irtt? (https://github.com/heistp/irtt ) I have not. Seems like a reasonable tool for UDP testing. The feature I didn't like in my scan of the documentation is the use of Inter-packet delay variation (IPDV) instead of packet delay variation (PDV): variation from the minimum (or reference) delay. The morbidly curious can find my analysis in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481 irtt's use of IPDV means that the results won=E2=80=99t compare with UDPST, and possibly networkQuality. But I may give it a try anyway... thanks again, Dave. Al > -----Original Message----- > From: Dave Taht > Sent: Monday, October 31, 2022 12:52 PM > To: MORTON JR., AL > Cc: ippm@ietf.org; Rpm > Subject: Re: [ippm] Preliminary measurement comparison of "Working Latenc= y" > metrics > > Thank you very much for the steer to RFC9097. I'd completely missed that. > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL wrote: > > > > (astute readers may have guessed that I pressed "send" too soon on prev= ious > message...) > > > > I also conducted upstream tests this time, here are the results: > > (capacity in Mbps, delays in ms, h and m are RPM categories, High and > Medium) > > > > Net Qual UDPST (RFC9097) Ookla > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpCap > Ping(no load) > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 = 8 > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22 = 8 > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22 = 9 > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22 = 7 > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22 = 9 > > > > The Upstream capacity measurements reflect an interesting feature that = we > can reliably and repeatably measure with UDPST. The first ~3 seconds of > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays this > behavior in its 1 second sub-interval measurements and has a bimodal repo= rting > option that divides the complete measurement interval in two time interva= ls to > report an initial (turbo) max capacity and a steady-state max capacity fo= r the > later intervals. The UDPST capacity results present both measurements: st= eady- > state first. > > Certainly we can expect bi-model distributions from many ISPs, as, for > one thing, the "speedboost" concept remains popular, except that it's > misnamed, as it should be called speed-subtract or speed-lose. Worse, > it is often configured "sneakily", in that it doesn't kick in for the > typical observed duration of the test, for some, they cut the > available bandwidth about 20s in, others, 1 or 5 minutes. > > One of my biggest issues with the rpm spec so far is that it should, > at least, sometimes, run randomly longer than the overly short > interval it runs for and the tools also allow for manual override of leng= th. > > we caught a lot of tomfoolery with flent's rrul test running by default f= or > 1m. > > Also, AQMs on the path can take a while to find the optimal drop or mark = rate. > > > > > The capacity processing in networkQuality and Ookla appear to report th= e > steady-state result. > > Ookla used to basically report the last result. Also it's not a good > indicator of web traffic behavior at all, watching the curve > go up much more slowly in their test on say, fiber 2ms, vs starlink, > (40ms).... > > So adding another mode - how quickly is peak bandwidth actually > reached, would be nice. > > I haven't poked into the current iteration of the goresponsiveness > test at all: https://urldefense.com/v3/__https://github.com/network- > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NR= Msdvb > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it > would be good to try collecting more statistics and histograms and > methods of analyzing the data in that libre-source version. > > How does networkQuality compare vs a vs your tool vs a vs goresponsivenes= s? > > >I watched the upstream capacity measurements on the Ookla app, and could > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps for mo= st of > the test which determined the final result. > > I tend to get upset when I see ookla's new test flash a peak result in > the seconds and then settle on some lower number somehow. > So far as I know they are only sampling the latency every 250ms. > > > > > The working latency is about 200ms in networkQuality and about 280ms as > measured by UDPST (RFC9097). Note that the networkQuality minimum delay i= s > ~20ms lower than the UDPST RTTmin, so this accounts for some of the diffe= rence > in working latency. Also, we used the very dynamic Type C load > adjustment/search algorithm in UDPST during all of this testing, which co= uld > explain the higher working latency to some degree. > > > > So, it's worth noting that the measurements needed for assessing workin= g > latency/responsiveness are available in the UDPST utility, and that the U= DPST > measurements are conducted on UDP transport (used by a growing fraction o= f > Internet traffic). > > Thx, didn't know of this work til now! > > have you tried irtt? > > > > > comments welcome of course, > > Al > > > > > -----Original Message----- > > > From: ippm On Behalf Of MORTON JR., AL > > > Sent: Sunday, October 30, 2022 8:09 PM > > > To: ippm@ietf.org > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working > Latency" > > > metrics > > > > > > > > > Hi again RPM friends and IPPM'ers, > > > > > > As promised, I repeated the tests shared last week, this time using b= oth > the > > > verbose (-v) and sequential (-s) dwn/up test options of networkQualit= y. I > > > followed Sebastian's calculations as well. > > > > > > Working Latency & Capacity Summary > > > > > > Net Qual UDPST Ookla > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange DnCap > > > Ping(no load) > > > 885 916 m 66ms 8ms 970 28 0-20 940 = 8 > > > 888 1355 h 44ms 8ms 966 28 0-23 940 = 8 > > > 891 1109 h 54ms 8ms 968 27 0-19 940 = 9 > > > 887 1141 h 53ms 11ms 966 27 0-18 937 = 7 > > > 884 1151 h 52ms 9ms 968 28 0-20 937 = 9 > > > > > > With the sequential test option, I noticed that networkQuality achiev= ed > nearly > > > the maximum capacity reported almost immediately at the start of a te= st. > > > However, the reported capacities are low by about 60Mbps, especially = when > > > compared to the Ookla TCP measurements. > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the high e= nd of > the > > > RTTrange), for example 54ms compared to (27+19=3D46). Most of the > networkQuality > > > RPM measurements were categorized as "High". There doesn't seem to be= much > > > buffering in the downstream direction. > > > > > > > > > > > > > -----Original Message----- > > > > From: ippm On Behalf Of MORTON JR., AL > > > > Sent: Monday, October 24, 2022 6:36 PM > > > > To: ippm@ietf.org > > > > Subject: [ippm] Preliminary measurement comparison of "Working Late= ncy" > > > > metrics > > > > > > > > > > > > Hi RPM friends and IPPM'ers, > > > > > > > > I was wondering what a comparison of some of the "working latency" > metrics > > > > would look like, so I ran some tests using a service on DOCSIS 3.1,= with > the > > > > downlink provisioned for 1Gbps. > > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and Ookl= a > > > Speedtest > > > > with as similar connectivity as possible (but we know that the traf= fic > will > > > > diverge to different servers and we can't change that aspect). > > > > > > > > Here's a quick summary of yesterday's results: > > > > > > > > Working Latency & Capacity Summary > > > > > > > > Net Qual UDPST Ookla > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap Ping(= no > load) > > > > 878 62 970 28 0-19 941 6 > > > > 891 92 970 27 0-20 940 7 > > > > 891 120 966 28 0-22 937 9 > > > > 890 112 970 28 0-21 940 8 > > > > 903 70 970 28 0-16 935 9 > > > > > > > > Note: all RPM values were categorized as Low. > > > > > > > > networkQuality downstream capacities are always on the low side com= pared > to > > > > others. We would expect about 940Mbps for TCP, and that's mostly wh= at > Ookla > > > > achieved. I think that a longer test duration might be needed to ac= hieve > the > > > > actual 1Gbps capacity with networkQuality; intermediate values obse= rved > were > > > > certainly headed in the right direction. (I recently upgraded to > Monterey > > > 12.6 > > > > on my MacBook, so should have the latest version.) > > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I sho= uld > have > > > > run the tests with the -v option to help with comparisons. I'll rep= eat > this > > > > test when I can make time. > > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during the t= est) > and > > > > the range of variation above the minimum (RTTVarRnge) add-up to ver= y > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded this > access > > > and > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in Chic= ago > > > > conducting tests, so the minimum 28ms is typical. UDPST measurement= s > were > > > run > > > > on an Ubuntu VM in my MacBook. > > > > > > > > The big disappointment was that the Ookla desktop app I updated ove= r the > > > > weekend did not include the new responsiveness metric! I included t= he > ping > > > > results anyway, and it was clearly using a server in the nearby are= a. > > > > > > > > So, I have some more work to do, but I hope this is interesting-eno= ugh > to > > > > start some comparison discussions, and bring-out some suggestions. > > > > > > > > happy testing all, > > > > Al > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > ippm mailing list > > > > ippm@ietf.org > > > > > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;= !!Bhd > > > > > > > > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzWef_x= Kqg4$ > > > > > > _______________________________________________ > > > ippm mailing list > > > ippm@ietf.org > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;= !!Bhd > > > T!g- > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1EBpiw= $ > > > > _______________________________________________ > > ippm mailing list > > ippm@ietf.org > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;= !!Bhd > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48iQFu= b_gMs > KXU$ > > > > -- > This song goes out to all the folk that thought Stadia would work: > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-mush= room- > song-activity-6981366665607352320- > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7= PxzxT > c48iQFub34zz4iE$ > Dave T=C3=A4ht CEO, TekLibre, LLC --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC