From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4539F3B2A4 for ; Mon, 28 Mar 2022 12:56:51 -0400 (EDT) Received: by mail-ed1-x530.google.com with SMTP id g20so17662945edw.6 for ; Mon, 28 Mar 2022 09:56:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=obs-cr.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=JYwHFZUoIjVF58+t484TngSj4gxa157G3RssWlu5Cdw=; b=3PUhLzNXSmKU6X/WpeuPxvQC0so0mBgCEJ5Y8K/F/BD6lCJbuMz3Oh4W3QREGBM7uI 6f3J8Q41Mt4jCwRx0dyQ76ldf2jTWsKoY9ImFISrIqXCYO/c053/0IJ16Eh8h3eh9aO9 FIMVKQWcamSczmSjG15pB6Fmtx9973W5sdsI4ODUAsztJuvkPRerlvA4cDnDBH+k+yWS mrdZZ2aTjqwlseGlS/BekDvrQWiB+OWgtrdJNQ472u02W5vILrTcHtz6+jDNuNqFSmQ4 drIQZ4MZq8DImlBjDNiduCK71wJO2jRWJxASIEw6FVSbMMjkeCeWpcDfGL57nRJgxMbA RWTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=JYwHFZUoIjVF58+t484TngSj4gxa157G3RssWlu5Cdw=; b=cPa5SJIoOIkvYr/hBRemwsdgoK70Ci5KNVKzhmaKAoWfeqUgluCvPby7n02B9eUtG/ oO1AFU5c+IIzUJuP08mT3PZfuJdMHCl14nworXJqIpW/nqrGrLwgGsxSjJtH+qTQZBrV 1mr0tS82TjzzmA48PVKx7ysKlscg3/Za0suu1bzi5R2tH8xJF7JyQQm8shG5x2aFyTBc edGXHSOaCARwqM9rQWw6xHBg1uH9ZUIbhCm8bWMV9/dFdcVqZzraB5z3ba4EyGZtt+aS eREtg372Jr2xDawxBQNWDxbiyuDK6kpIB6EE3ZdKjJ7W5xWONVCnk5F+e+usl8FDVS9K 7how== X-Gm-Message-State: AOAM533vj2HYa+F7kf6TfSuhiqrSgN8B/QDfU6hPrfo39vUZ6ObM3pAG ErJs/2qMqdStN56B/zrvrpWswYaQog0cOfayuaeT7+8sl2Y= X-Google-Smtp-Source: ABdhPJxcYKwa+GYTG2VVGOppeogNJhsuE7C8P5Hp+EvKyQsDa39uvWnDNi93dpy1r/YYLsdXVNltpOrA2XOkHjQk02M= X-Received: by 2002:a05:6402:1907:b0:418:d2a8:a7f3 with SMTP id e7-20020a056402190700b00418d2a8a7f3mr17553097edz.245.1648486609728; Mon, 28 Mar 2022 09:56:49 -0700 (PDT) MIME-Version: 1.0 References: <0D7D64F2-984B-428D-AC42-C862AB141E01@gmx.de> In-Reply-To: From: Will Hawkins Date: Mon, 28 Mar 2022 12:56:38 -0400 Message-ID: To: Sebastian Moeller Cc: Rpm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Rpm] rpm meeting notes from last week X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2022 16:56:51 -0000 First, again, sorry for the top-post! I really, *really* appreciate your email and your feedback, Sebastian! It's great to work with people like you!! I will write more as I have time -- ${DAY_JOB} is busy today!! Thanks for everything! Will On Mon, Mar 28, 2022 at 4:41 AM Sebastian Moeller wrote: > > Hi Will, > > just had a look at the draft and my expected run times assume bi-directio= nal saturation, while the draft recommends two sets of uni-directional satu= ration... so my estimates are off not by roughly a factor of 2 but rather m= ore like a factor of 4... I will try to read the go code to see what it int= ends to do here, but I am not a fluent go reader (@dayjob I use matlab, so = far far away from c-ish languages). > > Regards > Sebastian > > > > > > > On Mar 28, 2022, at 09:58, Sebastian Moeller via Rpm wrote: > > > > Hi Will, > > > > > > first, but belated, let me express my excitement about finding your pro= ject. It is great that apple distributes something like this in their recen= t OSs but for the "rest of us" that does not help (pun intended). > > > > > >> On Mar 28, 2022, at 00:26, Will Hawkins wrote: > >> > >> On Sun, Mar 27, 2022 at 2:34 PM Sebastian Moeller via Rpm > >> wrote: > >>> > >>> Dear RPMers, > >>> > >>> I just managed to install and test goresponsiveness on an Ubuntu20.4 = LTS linux host (where the default go is a bit on the old side), and I am 10= 0% not confident that the test runs sufficiently long to either saturate th= e link or to measure the saturating load sufficiently precise and IMHO most= likely neither... > >>> > >>> Here is the text of an issue I posted as an issue to the nice gorespo= nsiveness project: > >>> > >>> The total measurement duration of the test against apples servers app= ears quite short. It would be nice if the duration could be controlled from= the client (or if that is not according to the spec if the servers could b= e configured to measure for longer durations): > >>> > >>> user@ubuntu20.4LTS:~/CODE/goresponsiveness$ time ./networkQuality --c= onfig mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --timeout 6= 0 > >>> 03-27-2022 17:51:51 UTC Go Responsiveness to mensura.cdn-apple.com:44= 3... > >>> Download: 87.499 Mbps ( 10.937 MBps), using 12 parallel connections. > >>> Upload: 26.531 Mbps ( 3.316 MBps), using 12 parallel connections. > >>> Total RTTs measured: 5 > >>> RPM: 829 > >>> > >>> real 0m5.504s > >>> user 0m1.277s > >>> sys 0m1.225s > >>> > >>> > >>> ~6 seconds, that is even shorter than run-of-the-mill speedtests (whi= ch IMHO tend to clock in at ~10 seconds per direction)... This might or mig= ht not result in a saturating load, but it most certainly will result in an= imprecise throughput measurement... > >>> > >>> Experience with bufferbloat measurements seems to indicate that >=3D = 20-30 seconds are required to really saturate a link. > >>> > >>> Trying to confirm this with flent/netperf: > >>> > >>> date ; ping -c 10 netperf-eu.bufferbloat.net ; ./run-flent --ipv4 -l = 30 -H netperf-eu.bufferbloat.net rrul_var --remote-metadata=3Droot@192.168.= 42.1 --test-parameter=3Dcpu_stats_hosts=3Droot@192.168.42.1 --step-size=3D.= 05 --socket-stats --test-parameter bidir_streams=3D8 --test-parameter marki= ngs=3D0,0,0,0,0,0,0,0 --test-parameter ping_hosts=3D1.1.1.1 -D . -t IPv4_SQ= M_cake_layer-cake_LLA-ETH_OH34_U097pct36000of36998K-D090pct105000of116797K_= work-horse-eth0_2_TurrisOmnia-TurrisOS.5.3.6-pppoe-wan-eth2.7_2_bridged-BTH= H5A-OpenWrt-r18441-ba7cee05ed-Hvt-VDSL100_2_netperf-eu.bufferbloat.net --lo= g-file > >>> > >>> [...] > >>> > >>> Ping (ms) ICMP 1.1.1.1 (extra) : 12.20 12.10 = 21.91 ms 914 > >>> Ping (ms) avg : 26.11 N/A = N/A ms 914 > >>> Ping (ms)::ICMP : 25.30 25.10 = 35.11 ms 914 > >>> Ping (ms)::UDP 0 (0) : 25.42 25.43 = 33.43 ms 914 > >>> Ping (ms)::UDP 1 (0) : 25.36 25.32 = 33.77 ms 914 > >>> Ping (ms)::UDP 2 (0) : 26.99 26.88 = 36.13 ms 914 > >>> Ping (ms)::UDP 3 (0) : 25.27 25.20 = 34.42 ms 914 > >>> Ping (ms)::UDP 4 (0) : 25.26 25.22 = 33.32 ms 914 > >>> Ping (ms)::UDP 5 (0) : 27.99 27.85 = 37.20 ms 914 > >>> Ping (ms)::UDP 6 (0) : 27.95 27.89 = 36.29 ms 914 > >>> Ping (ms)::UDP 7 (0) : 25.42 25.32 = 34.56 ms 914 > >>> TCP download avg : 11.93 N/A = N/A Mbits/s 914 > >>> TCP download sum : 95.44 N/A = N/A Mbits/s 914 > >>> TCP download::0 (0) : 11.85 12.26 = 13.97 Mbits/s 914 > >>> TCP download::1 (0) : 11.81 12.28 = 13.81 Mbits/s 914 > >>> TCP download::2 (0) : 12.13 12.31 = 16.28 Mbits/s 914 > >>> TCP download::3 (0) : 12.04 12.31 = 13.76 Mbits/s 914 > >>> TCP download::4 (0) : 11.78 12.28 = 13.83 Mbits/s 914 > >>> TCP download::5 (0) : 12.07 12.30 = 14.44 Mbits/s 914 > >>> TCP download::6 (0) : 11.78 12.29 = 13.67 Mbits/s 914 > >>> TCP download::7 (0) : 11.98 12.30 = 13.92 Mbits/s 914 > >>> TCP totals : 126.34 N/A = N/A Mbits/s 914 > >>> TCP upload avg : 3.86 N/A = N/A Mbits/s 914 > >>> TCP upload sum : 30.90 N/A = N/A Mbits/s 914 > >>> [...] > >>> > >>> This is a bidirectional test that only reports actual TCP goodput, ne= ither the latency probes nor the reverse ACK traffic is accounted and yet: > >>> TCP download sum: 95.44 > >>> TCP upload sum: 30.90 > >>> > >>> which compared to the actual shape settings of 105/36 (with 34 bytes = overhead) seems sane: > >>> theoretical maximal throughput: > >>> IPv4 > >>> 105 * ((1500-8-20-20-12)/(1500+26)) =3D 99.08 Mbps > >>> 36 * ((1500-8-20-20-12)/(1500+26)) =3D 33.97 Mbps > >>> IPv6 > >>> 105 * ((1500-8-40-20-12)/(1500+26)) =3D 97.71 Mbps > >>> 36 * ((1500-8-40-20-12)/(1500+26)) =3D 33.50 Mbps > >>> > >>> Compared to these numbers the reported 87.499/26.531 do not seems eit= her saturating or precise, probably neither.... I do wonder how reliable th= e RPM number is gong to be if the "saturate the link" part apparently faile= d? > >>> > >>> Question: is full saturation actually intended here, or is "sufficien= tly high load" the goal (if the later what constitutes sufficiently high)? > >> > >> According to the spec, you may not get completely full saturation. The > >> definition of saturation per the spec is fairly specific: See 4.1.4 of > >> https://github.com/network-quality/draft-ietf-ippm-responsiveness/blob= /master/draft-ietf-ippm-responsiveness.txt. > > > > Thanks for the pointer... Not sure I am convinced of that logic r= egarding saturation.... I think I like the "add more flows" approach to gen= erating load, not sure I think that the 4 second increments seem long enoug= h... also aiming for 20 seconds run time seems like a decent goal, but sold= ering on in case the results are not stable at that point seems more helpfu= l than leaving that undefined, no? > > > > Also, I wish the test would report something like the raw delays as wel= l as the RPMs, and personally I think reporting delay/RPM without load as w= ell also seems worthwhile, as that basically indicates how much a specific = link could actually gain from traffic-shaping/competent AQM... > > > > > > > >> > >> We are still working through some of the timing issues in the > >> goresponsiveness client and doing our best. > > > > Ah, that seems to be the case from my understanding each addition= al set of 4 flows will increase the run time for up to 4 seconds, so with 1= 2 parrallel connections I would expect a run time of between 8+1 to 8+4 so = 9-12, which is a bit above the reported 5.5 seconds (goresponsivess results= above) > > and similar for the results below > > 20 flows -> expected: 17-20 seconds actual: 9.7 > > 24 flows -> expected: 21-24 seconds actual: 12.9 > > > > So it seems like the run time is roughly half as the draft seems to rec= ommend, no? > > > > > >> Thank you for your > >> patience and helpful, constructive and kind feedback. > > > > Sorry, on second reading my mail is neither helpful, constructive= or kind, but appears rather rude. I want to apologize for this as this was= not my intent. I very much appreciate your project and actually just want = to understand it better. > > > > > >> > >> Will > >> > >> > >>> > >>> Regards > >>> Sebastian > >>> > >>> P.S.: When disabling my traffic shaper I get: > >>> > >>> user@ubuntu20.4LTS:~/CODE/flent$ time ../goresponsiveness/networkQual= ity --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --ti= meout 60 > >>> 03-27-2022 18:20:06 UTC Go Responsiveness to mensura.cdn-apple.com:44= 3... > >>> Download: 99.390 Mbps ( 12.424 MBps), using 12 parallel connections. > >>> Upload: 27.156 Mbps ( 3.395 MBps), using 20 parallel connections. > >>> Total RTTs measured: 5 > >>> RPM: 512 > >>> > >>> real 0m9.728s > >>> user 0m1.763s > >>> sys 0m1.660s > >>> user@ubuntu20.4LTS:~/CODE/flent$ time ../goresponsiveness/networkQual= ity --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --ti= meout 60 > >>> 03-27-2022 18:20:27 UTC Go Responsiveness to mensura.cdn-apple.com:44= 3... > >>> Download: 97.608 Mbps ( 12.201 MBps), using 12 parallel connections. > >>> Upload: 27.375 Mbps ( 3.422 MBps), using 24 parallel connections. > >>> Total RTTs measured: 5 > >>> RPM: 366 > >>> > >>> real 0m12.926s > >>> user 0m2.330s > >>> sys 0m2.442s > >>> > >>> Which for the download approaches/exceeds the theoretical limit* yet = for the upload still falls a bit short of the 8 flow flent result > >>> > >>> > >>> *) This is nothing particular unique to goresponsiveness most multi-s= tream tests fail to actually report robust and reliable values for thoughpu= t, even though this "ain't rocket science". Instead of trying to discount t= he TCP start-up phase somehow, simply run the test long enough so the start= up does not dominate the result and just divide total transported volume b= y total duration... yes that is going to slightly underestimate the total l= ink capacity, but that seems IMHO considerably more benign/useful than repo= rting 99.39 of 99.08 possible or rather 97.71 possible , since the actual t= est uses IPv6 on my link... > >>> > >>> > >>> > >>>> On Mar 27, 2022, at 16:25, Dave Taht via Rpm wrote: > >>>> > >>>> Sasha has re-organized the meeting notes we've been taking which wil= l > >>>> be permanently up here: > >>>> > >>>> https://docs.google.com/document/d/19K14vrBLNvX_KOdRMK_s0SCYTnj70IJ-= dQubL37kQgw/edit > >>>> > >>>> The public meetings continue at 10AM PDT tuesdays. > >>>> > >>>> One thing came up in that meeting that bears further reflection: The > >>>> structure of the the responsiveness test is e2e, and differentiating > >>>> between the wifi hop and/or sourcing tests from the router - is hard= . > >>>> In the latter case, we are asking a router not only to saturate its > >>>> network connection, but to do it using crypto, which is a mighty tas= k > >>>> to ask of a router that's job is primarily routing (with offloads). > >>>> > >>>> A thought might be to attempt p2p tests between say, an ios phone an= d > >>>> an osx laptop, which would exercise the router as a wifi router. A > >>>> docker container for the server would be straightforward. (It's stil= l > >>>> hard for me to > >>>> trust containers or vm's to run fast or accurately enough, too). > >>>> > >>>> Former case is still e2e but with the icmp unreachable idea I descri= bed in wtbb. > >>>> > >>>> -- > >>>> I tried to build a better future, a few times: > >>>> https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > >>>> > >>>> Dave T=C3=A4ht CEO, TekLibre, LLC > >>>> _______________________________________________ > >>>> Rpm mailing list > >>>> Rpm@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/rpm > >>> > >>> _______________________________________________ > >>> Rpm mailing list > >>> Rpm@lists.bufferbloat.net > >>> https://lists.bufferbloat.net/listinfo/rpm > > > > _______________________________________________ > > Rpm mailing list > > Rpm@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/rpm >