From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 9D6283B29D for ; Mon, 28 Mar 2022 04:41:28 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1648456886; bh=E1i5VAphvXL/LtKqRSpzp9/mDqbFNRnRl5pvyOFslRg=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=BwY9+aWUMu8CRZB1HbcTbjhJpINjA5m2cQKrRbLtWaQWuBBLIL/2zIJaiK3YfSshB eGtM+rU4aGcQ2vH8HHEBdCHpZcZcb0z/cUux1Mthw7lOFGP7XT0tRcJE6D44LlV6cQ 9xdBYPizNVUrTSfh5OV85tbT8hnT0wocSY6qVcoM= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MXXuB-1nXonx1KgB-00YwDu; Mon, 28 Mar 2022 10:41:26 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) From: Sebastian Moeller In-Reply-To: Date: Mon, 28 Mar 2022 10:41:25 +0200 Cc: Will Hawkins , Rpm Content-Transfer-Encoding: quoted-printable Message-Id: References: <0D7D64F2-984B-428D-AC42-C862AB141E01@gmx.de> To: Sebastian Moeller X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Provags-ID: V03:K1:NyrXS0r4khilFmL/NWebOid0TgHqQqXXqGvQSUR+XHgN3XXaTCU Gjj11TWvetBL4lprrQbm3xA5PkkP9jxLdH1uHQfkrJNSodlObiV0wfGMzrfVZb2SnfKv7wp LehCE2aoexE4ZMJt2O4iCI+Qg1ff0albdlLo26dM7s9alQR6epaBE1T3GDFzMepsVzRPijI sIP7QsLhgREH/kZDNV8Ew== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:Z+P8vD8lhCE=:6U/VwWit3gh+Shd+eGTCnu Su//qZOuuI/ft+lSy/B2w2UrUBMAMfHLKZshxTAA1KffLKKehRsglVZ8Fkrn9t9mqe6xdCWfF yJ7/uMEGYgJB6M9TBTUu5bWUdaunVplKynJwawkQo8GU2hkEsbtmD5u+1a9ZgDXvQnXU4Lc91 zusv80n+/b0irqTmuOpTaqcprBw4yUKCCjkS4UObNm3H52VuoowmbZays+EReOUvko6ejkYEL W4Pw+cqWdkT5791APUq6Ygpz3q4gR4978Adtx2WpQN354mVBMpju+gYorMfPVnrQRNujXLkrY EYzzGcUn/0/UiKRz0MylViAdFagQLqAPylEtn7VNigcK2uEYqW5/3/g2EtDZX6YHynGmFfMwP Fz3+fZ4E6vUT6KZ+SqbelCDR/94efeD45iuL8nPcMM+6TCuXkeBToWDaCjzMdArQXu/oR3aln leQgcga4CzsBQ3D4YXbnmNduuwNd0mpm47hQnUro1m4d2aGeSHfowi7MPHsmJagpuJ6VeJYme sO6tFxm0Ur2w73XggKMcMqf9oIS2MbRFiltvVJZIVRcDrDjNYSBvMoJhK3JJx4xLidR+nuHeg lCBF1ZDYgI2db6CjHzbjjt0mn3qAAGnHd1hLn1P95oBmjn9sGRqEi7viNRq7c1for6oXGRqHJ Y8odUzpZRn+UVTzor5FqDWt+Pk5SteUjlxftsQaKYOEtX8OvA/TcvlQxaTxg8Pbb30GR8B3u6 nIpjQ1uKLtR+X9oRNITBbwcSWOL9IEGvIP0YZcB1tiKw1glyWLbHK28PjEZAh+HEcoXm2ROty dz7N0xnIrJp84ttjnjAH9W9USQIXNZ3cYV8npeSgBR4SaniaHhCtanFgXYuXuUvEdfblyfqYm MczMlirRfTuAh2RxDJeTd8SRCCXorcS6PB2lGdR1zq8O4/r5FupM7JTe3E5WQqRjnpysDzsH/ xlW2FWhbqbPcZ/ErDI6oiNoliWyBJlguVoumptTBdY7S0+sFKg6pvWMNMgoS7BLeetrwoeyYi WUEaZ0sj+vFchNhY71jHaWQGlk3SRpkFFf+Fi/pBGZmEAD+SfcEJzNL5kpAvxpf1n1chB5cob J9evLSGhT9ehoQ= Subject: Re: [Rpm] rpm meeting notes from last week X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2022 08:41:29 -0000 Hi Will, just had a look at the draft and my expected run times assume = bi-directional saturation, while the draft recommends two sets of = uni-directional saturation... so my estimates are off not by roughly a = factor of 2 but rather more like a factor of 4... I will try to read the = go code to see what it intends to do here, but I am not a fluent go = reader (@dayjob I use matlab, so far far away from c-ish languages). Regards Sebastian > On Mar 28, 2022, at 09:58, Sebastian Moeller via Rpm = wrote: >=20 > Hi Will, >=20 >=20 > first, but belated, let me express my excitement about finding your = project. It is great that apple distributes something like this in their = recent OSs but for the "rest of us" that does not help (pun intended). >=20 >=20 >> On Mar 28, 2022, at 00:26, Will Hawkins wrote: >>=20 >> On Sun, Mar 27, 2022 at 2:34 PM Sebastian Moeller via Rpm >> wrote: >>>=20 >>> Dear RPMers, >>>=20 >>> I just managed to install and test goresponsiveness on an Ubuntu20.4 = LTS linux host (where the default go is a bit on the old side), and I am = 100% not confident that the test runs sufficiently long to either = saturate the link or to measure the saturating load sufficiently precise = and IMHO most likely neither... >>>=20 >>> Here is the text of an issue I posted as an issue to the nice = goresponsiveness project: >>>=20 >>> The total measurement duration of the test against apples servers = appears quite short. It would be nice if the duration could be = controlled from the client (or if that is not according to the spec if = the servers could be configured to measure for longer durations): >>>=20 >>> user@ubuntu20.4LTS:~/CODE/goresponsiveness$ time ./networkQuality = --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config = --timeout 60 >>> 03-27-2022 17:51:51 UTC Go Responsiveness to = mensura.cdn-apple.com:443... >>> Download: 87.499 Mbps ( 10.937 MBps), using 12 parallel = connections. >>> Upload: 26.531 Mbps ( 3.316 MBps), using 12 parallel = connections. >>> Total RTTs measured: 5 >>> RPM: 829 >>>=20 >>> real 0m5.504s >>> user 0m1.277s >>> sys 0m1.225s >>>=20 >>>=20 >>> ~6 seconds, that is even shorter than run-of-the-mill speedtests = (which IMHO tend to clock in at ~10 seconds per direction)... This might = or might not result in a saturating load, but it most certainly will = result in an imprecise throughput measurement... >>>=20 >>> Experience with bufferbloat measurements seems to indicate that >=3D = 20-30 seconds are required to really saturate a link. >>>=20 >>> Trying to confirm this with flent/netperf: >>>=20 >>> date ; ping -c 10 netperf-eu.bufferbloat.net ; ./run-flent --ipv4 -l = 30 -H netperf-eu.bufferbloat.net rrul_var = --remote-metadata=3Droot@192.168.42.1 = --test-parameter=3Dcpu_stats_hosts=3Droot@192.168.42.1 --step-size=3D.05 = --socket-stats --test-parameter bidir_streams=3D8 --test-parameter = markings=3D0,0,0,0,0,0,0,0 --test-parameter ping_hosts=3D1.1.1.1 -D . -t = IPv4_SQM_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of= 116797K_work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.6-pppoe-wan-eth2.7_2_br= idged-BTHH5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloa= t.net --log-file >>>=20 >>> [...] >>>=20 >>> Ping (ms) ICMP 1.1.1.1 (extra) : 12.20 12.10 = 21.91 ms 914 >>> Ping (ms) avg : 26.11 N/A = N/A ms 914 >>> Ping (ms)::ICMP : 25.30 25.10 = 35.11 ms 914 >>> Ping (ms)::UDP 0 (0) : 25.42 25.43 = 33.43 ms 914 >>> Ping (ms)::UDP 1 (0) : 25.36 25.32 = 33.77 ms 914 >>> Ping (ms)::UDP 2 (0) : 26.99 26.88 = 36.13 ms 914 >>> Ping (ms)::UDP 3 (0) : 25.27 25.20 = 34.42 ms 914 >>> Ping (ms)::UDP 4 (0) : 25.26 25.22 = 33.32 ms 914 >>> Ping (ms)::UDP 5 (0) : 27.99 27.85 = 37.20 ms 914 >>> Ping (ms)::UDP 6 (0) : 27.95 27.89 = 36.29 ms 914 >>> Ping (ms)::UDP 7 (0) : 25.42 25.32 = 34.56 ms 914 >>> TCP download avg : 11.93 N/A = N/A Mbits/s 914 >>> TCP download sum : 95.44 N/A = N/A Mbits/s 914 >>> TCP download::0 (0) : 11.85 12.26 = 13.97 Mbits/s 914 >>> TCP download::1 (0) : 11.81 12.28 = 13.81 Mbits/s 914 >>> TCP download::2 (0) : 12.13 12.31 = 16.28 Mbits/s 914 >>> TCP download::3 (0) : 12.04 12.31 = 13.76 Mbits/s 914 >>> TCP download::4 (0) : 11.78 12.28 = 13.83 Mbits/s 914 >>> TCP download::5 (0) : 12.07 12.30 = 14.44 Mbits/s 914 >>> TCP download::6 (0) : 11.78 12.29 = 13.67 Mbits/s 914 >>> TCP download::7 (0) : 11.98 12.30 = 13.92 Mbits/s 914 >>> TCP totals : 126.34 N/A = N/A Mbits/s 914 >>> TCP upload avg : 3.86 N/A = N/A Mbits/s 914 >>> TCP upload sum : 30.90 N/A = N/A Mbits/s 914 >>> [...] >>>=20 >>> This is a bidirectional test that only reports actual TCP goodput, = neither the latency probes nor the reverse ACK traffic is accounted and = yet: >>> TCP download sum: 95.44 >>> TCP upload sum: 30.90 >>>=20 >>> which compared to the actual shape settings of 105/36 (with 34 bytes = overhead) seems sane: >>> theoretical maximal throughput: >>> IPv4 >>> 105 * ((1500-8-20-20-12)/(1500+26)) =3D 99.08 Mbps >>> 36 * ((1500-8-20-20-12)/(1500+26)) =3D 33.97 Mbps >>> IPv6 >>> 105 * ((1500-8-40-20-12)/(1500+26)) =3D 97.71 Mbps >>> 36 * ((1500-8-40-20-12)/(1500+26)) =3D 33.50 Mbps >>>=20 >>> Compared to these numbers the reported 87.499/26.531 do not seems = either saturating or precise, probably neither.... I do wonder how = reliable the RPM number is gong to be if the "saturate the link" part = apparently failed? >>>=20 >>> Question: is full saturation actually intended here, or is = "sufficiently high load" the goal (if the later what constitutes = sufficiently high)? >>=20 >> According to the spec, you may not get completely full saturation. = The >> definition of saturation per the spec is fairly specific: See 4.1.4 = of >> = https://github.com/network-quality/draft-ietf-ippm-responsiveness/blob/mas= ter/draft-ietf-ippm-responsiveness.txt. >=20 > Thanks for the pointer... Not sure I am convinced of that logic = regarding saturation.... I think I like the "add more flows" approach to = generating load, not sure I think that the 4 second increments seem long = enough... also aiming for 20 seconds run time seems like a decent goal, = but soldering on in case the results are not stable at that point seems = more helpful than leaving that undefined, no? >=20 > Also, I wish the test would report something like the raw delays as = well as the RPMs, and personally I think reporting delay/RPM without = load as well also seems worthwhile, as that basically indicates how much = a specific link could actually gain from traffic-shaping/competent = AQM... >=20 >=20 >=20 >>=20 >> We are still working through some of the timing issues in the >> goresponsiveness client and doing our best. >=20 > Ah, that seems to be the case from my understanding each = additional set of 4 flows will increase the run time for up to 4 = seconds, so with 12 parrallel connections I would expect a run time of = between 8+1 to 8+4 so 9-12, which is a bit above the reported 5.5 = seconds (goresponsivess results above) > and similar for the results below > 20 flows -> expected: 17-20 seconds actual: 9.7 > 24 flows -> expected: 21-24 seconds actual: 12.9 >=20 > So it seems like the run time is roughly half as the draft seems to = recommend, no? >=20 >=20 >> Thank you for your >> patience and helpful, constructive and kind feedback. >=20 > Sorry, on second reading my mail is neither helpful, = constructive or kind, but appears rather rude. I want to apologize for = this as this was not my intent. I very much appreciate your project and = actually just want to understand it better. >=20 >=20 >>=20 >> Will >>=20 >>=20 >>>=20 >>> Regards >>> Sebastian >>>=20 >>> P.S.: When disabling my traffic shaper I get: >>>=20 >>> user@ubuntu20.4LTS:~/CODE/flent$ time = ../goresponsiveness/networkQuality --config mensura.cdn-apple.com --port = 443 --path /api/v1/gm/config --timeout 60 >>> 03-27-2022 18:20:06 UTC Go Responsiveness to = mensura.cdn-apple.com:443... >>> Download: 99.390 Mbps ( 12.424 MBps), using 12 parallel = connections. >>> Upload: 27.156 Mbps ( 3.395 MBps), using 20 parallel = connections. >>> Total RTTs measured: 5 >>> RPM: 512 >>>=20 >>> real 0m9.728s >>> user 0m1.763s >>> sys 0m1.660s >>> user@ubuntu20.4LTS:~/CODE/flent$ time = ../goresponsiveness/networkQuality --config mensura.cdn-apple.com --port = 443 --path /api/v1/gm/config --timeout 60 >>> 03-27-2022 18:20:27 UTC Go Responsiveness to = mensura.cdn-apple.com:443... >>> Download: 97.608 Mbps ( 12.201 MBps), using 12 parallel = connections. >>> Upload: 27.375 Mbps ( 3.422 MBps), using 24 parallel = connections. >>> Total RTTs measured: 5 >>> RPM: 366 >>>=20 >>> real 0m12.926s >>> user 0m2.330s >>> sys 0m2.442s >>>=20 >>> Which for the download approaches/exceeds the theoretical limit* yet = for the upload still falls a bit short of the 8 flow flent result >>>=20 >>>=20 >>> *) This is nothing particular unique to goresponsiveness most = multi-stream tests fail to actually report robust and reliable values = for thoughput, even though this "ain't rocket science". Instead of = trying to discount the TCP start-up phase somehow, simply run the test = long enough so the start up does not dominate the result and just divide = total transported volume by total duration... yes that is going to = slightly underestimate the total link capacity, but that seems IMHO = considerably more benign/useful than reporting 99.39 of 99.08 possible = or rather 97.71 possible , since the actual test uses IPv6 on my link... >>>=20 >>>=20 >>>=20 >>>> On Mar 27, 2022, at 16:25, Dave Taht via Rpm = wrote: >>>>=20 >>>> Sasha has re-organized the meeting notes we've been taking which = will >>>> be permanently up here: >>>>=20 >>>> = https://docs.google.com/document/d/19K14vrBLNvX_KOdRMK_s0SCYTnj70IJ-dQubL3= 7kQgw/edit >>>>=20 >>>> The public meetings continue at 10AM PDT tuesdays. >>>>=20 >>>> One thing came up in that meeting that bears further reflection: = The >>>> structure of the the responsiveness test is e2e, and = differentiating >>>> between the wifi hop and/or sourcing tests from the router - is = hard. >>>> In the latter case, we are asking a router not only to saturate its >>>> network connection, but to do it using crypto, which is a mighty = task >>>> to ask of a router that's job is primarily routing (with offloads). >>>>=20 >>>> A thought might be to attempt p2p tests between say, an ios phone = and >>>> an osx laptop, which would exercise the router as a wifi router. A >>>> docker container for the server would be straightforward. (It's = still >>>> hard for me to >>>> trust containers or vm's to run fast or accurately enough, too). >>>>=20 >>>> Former case is still e2e but with the icmp unreachable idea I = described in wtbb. >>>>=20 >>>> -- >>>> I tried to build a better future, a few times: >>>> https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org >>>>=20 >>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>> _______________________________________________ >>>> Rpm mailing list >>>> Rpm@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/rpm >>>=20 >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >=20 > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm