From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D1CF43B29E; Fri, 17 May 2019 04:29:53 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1558081790; bh=RuxBgBY1NMhH82moqUVayIk6U5jkrXSfCZmgE2iUmVw=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=AM5JAY4RLsM+k8Zhcqk2Fev5cWyl6i/pbzG9J1c6WUUgxs8WERzogSO4HCgmS4aUX 9el3CPNG0wlccUx94r0HctyNII1TwYFc7x/xQJ/phnSCsq8vbaviYUAJuomW+b2pl9 ioPldnCKcunUJc1LmjWVR6qqt75a4XaAnp30tlgI= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from [10.11.12.12] ([134.76.241.253]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MK3Vu-1h635E2AHO-00LSN5; Fri, 17 May 2019 10:29:50 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) From: Sebastian Moeller In-Reply-To: Date: Fri, 17 May 2019 10:29:49 +0200 Cc: Jonathan Foulkes , Rich Brown , cerowrt-devel , "David P. Reed" , bloat Content-Transfer-Encoding: quoted-printable Message-Id: <8D2260CE-0A93-44AD-86A9-1F1B332F80D4@gmx.de> References: To: =?utf-8?Q?Dave_T=C3=A4ht?= X-Mailer: Apple Mail (2.3445.104.11) X-Provags-ID: V03:K1:Vz8ZBhbxtb93+86kT04LMOEbDI3t/pemHJKeKbLNb7/hD+arlKw +PBm/MSEkvwzOb6fV8hEyYbVNCxPojvoHv75qVQhvvP/e8kb3xyWtYNT8375pgFcx+RlrCo SX3ODs/RwotSG7kZr1TNGJ3TgZnRrc7/ar4DvrjZBvcwCTOukhHztywNsEn0nxwJigUp8DY lIwj+2V6P99VaTlE9bApQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:esJfxlvty8A=:JYgOzojnG3k13PsGBSlm7S NFEvSyAEEykXxIZjt1o6vyKN2aieDfHWlNmZKvYZ4azhva8SEwnfamks0rq2JZXKu8LhfrGK3 TITlpWjjKBBHW9HAaUEajQfs/NZPuhs+9b9ROXpErf/BVfo9Xb8YtqonRTG5ZLizROace605w gcdaW/yJzwho5c8sRtAk0WARsrG7WEl1x+8O/rC3nETynLXfwfZx2kkN15KctHjiRYwC+UuK/ WBP96gx4ZmFvZBMaT4qFg4dkzLVi/R3wLbvHjTUGNb81Pp1y+g2YJozyYHqvX7/02vMSIqJmW y57jlJ5wj357jsh6VMCLyP9BUFxh0D6bR2c1gdIJxBa70KwFcDbdFuJPSIYNbVPgNHKY/A/ro JscsBNK1y94WYz/d8x7EaVTlVWAL3P3ZmdcInuqZITkGvKIrUatVYHktuLq8bdsCeQV3ALlCE jBSGNvcsKuBtxQp/K/EESC9c+5DfDpZ/Wv4Xjch5pqLqXVtgPte1vs35XNgR/UC1cRQz86e+j Al4XvtUgThMOftzpmEvqzwclz3kHRMN9nBbU0oPchJPHPkRz+/W7MCF3iIgiDn8Q4v1efRNbO pfxfc7XKAwODwXHz89BQv8+hoM/x1hoAGoQwqOaxqO2C2VbP7l7mZ4bysbQOBaULr8HKNB/6e 6nsvC3mXjV9tGF6RGm4UvINEsO2UG20dEIVMtu0wtdbHPwHLi4ZIbjHDwb/K61nb0+fmy+esW WH/1mzlKkBtULfdh/0TWHiXG6qnwlnXJJ1dX2R60d8Cy8olahUuXeEyJFavf5mPrgf7i4BZYK vIpt3s56p95tJ6IKFkZcBXXUYRTFQMfFXjU+C0RLKCUZ3a+VaHHspF5BCEG3qkwqzn4Spq48+ x+vrUpJ7Y1eRXS/N7ix77xvbOUVe3QIVJMbTXMAdAVZmJMtQH/X9ex2MSfSpTyjWTnMtHEs9m aSTYOg7r9qCSvPJzdYvIs4sfau9jCQe3PHLpmaQpocUClJiNGXoDI Subject: Re: [Bloat] Internet Speed Measurement: Current Challenges and Future Recommendations X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2019 08:29:54 -0000 > On May 17, 2019, at 09:36, Dave Taht wrote: >=20 > Jason is on the bloat list, nick feamster used to be, but isn't. > Sometimes I wish I could drag in various folk at the FCC, FTC, or some > other regulatory body more often. BITAG. Places like that. Having some > co-ordinated "Here are our comments on your thing" might be better > than the general chaos on the bloat list, and, since google is so > ineffective at indexing mailing lists these days some way to raise it > in the search results. >=20 > On Fri, May 17, 2019 at 12:01 AM Jonathan Foulkes > wrote: >>=20 >> Thanks for sharing Dave. >>=20 >> A good paper, but there are few gaps worthy of mentioning on this = list: >>=20 >> Testing when there is an AQM present means the test must adapt to the = challenge of smaller cwnd existing for any one stream, therefore it will = take many more streams to saturate a line with cwnd =3D 30 than if the = cwnd is allowed to grow to >1,000 >> In general, the impact of cwnd relative to saturation and impact on = delay was not visited, and yet it=E2=80=99s critical. One of the reasons = for spiky delays on high speed lines is the ginormous cwnds hogging the = line with their 800ms+ RTT=E2=80=99s Good point, I note that the detailed dslreports results also = contain information about the average cwnd, and I admit that so far I = have never looked at those numbers; will do in the future >>=20 >> Asymmetry of provisioned upload relative to download, at some point, = the ack-stream can be held up by either lack of capacity or bloat in the = uplink. So even though a link can deliver 300Mbps down, a bloated uplink = of 5mbps might never allow that level to be reached. IIRC a back of the envelop estimate assuming MTU~1500 and one ACK every = two full MSS packets gives the following percentage of ACK traffic for a = given amount of reverse traffic (I assume a cable link here for = overhead, but that ratio does not depend too much on the actual = per-packet-overhead): 100 * ((18+(20+20+12))/(1500+18)) =3D 4.61% ~. 1/20 ~ 5% Techniques like ACK filtering/thinning and GSO/GRO can reduce this = number considerably. So the 300 Mbps link needs to have at least = 300*0.05 =3D 15 Mbps uplink to allow saturation with bog-standard = TCP-Reno flows. Most asymmetries I have seen in the wild seem to keep = this in mind, but I would not be amazed to see something like 300/5 in = the wild ;) >> There are ISPs provisioning truly crazy asymmetric service. +1; or rather -1 as while this is true I do not want to endorse = that >>=20 >> They do make a good point about the local network, WiFi specifically = being the new bottleneck, which is why we included an iperf instance = that can be started on the IQrouter to help run client to server tests = that help spot local network capacity limits, typically on WiFi. Enlightened as so often. I note the official German speedtest = will ask for lan/wlan connection and will only consider tests via lan as = oficial (their desktop app even will not run over wifi at all, which I = find obnoxious). >>=20 >> Regarding their point about =E2=80=98Cross traffic=E2=80=99 impact on = measurements, Cake=E2=80=99s per-host / per-target fairness also = complicates AQM-enabled testing from client devices. Which is why we = make the built-in speed test the arbiter of true line capacity, as it = factors for ALL traffic flowing through the router.=20 +1. There is a toy project going on in the openwrt forum trying = to collect traffic, CPU load & frequency, and RTT data for concurrently = running saturating loads; the idea being the same as yours, measuring = the ingress/egress traffic directly at the router and just using the = speedtest(s) as a load generator(s). But that project has not gone very = far. (It statred from the question what shaper performance one can = expect from different SoCs) >> But, as you mention, that is also a challenge from a CPU resource = standpoint on higher speeds. Yes this is why monitoring the CPU load during the test seems = like a great idea. >>=20 >> The biggest gap in this paper is not paying sufficient attention to = latency as a critical metric, and one that is controllable by an AQM. = Bufferbloat metrics have more impact on end-user experience than +/- = 50Mbps on a 100mbps baseline. >> I was rather miffed they do not even mention the DSLreports.com = speedtest, or the fast.com test, as those are the two that provide a = bufferbloat metric. I might add that https://www.thinkbroadband.com/speedtest also = does a few things well, a) they do run a single stream and a 6 stream = test in sequence and report the numbers for both and b) they also do a = latency-under-load test (click the Analysis button in the first results = graph) . Unfortunately the latency-under-load grpahs is inaccessible = from the main and final results page... >=20 > I keep hoping government will get it. >=20 > I note that I'm in general quite pleased with the overall trendlines > on dslreports ( > http://www.dslreports.com/speedtest/results/bufferbloat?up=3D1 ). A = lot > of the bloat on cablemodems disappearing is due to uplink speeds > getting much better, and holding buffer sizes constant, of course, > rather than the docsis 3.1 deployment or our efforts, but I do tend to > look at the long fuzzy blue data on the left of the green 30ms line > and think that's mostly us. >=20 > The thing that scares though, is that since that test is exaustively > used by those tuning their links with fq_codel, is that that dataset > is now quite polluted. So what does reality actually look like? On > this trip I encountered network after network essentially in a state > of congestion collapse. >=20 > The other big problems with dslreports is that the test does not run > for long enough at this higher rates, After a free registration, one can configure the test duration = (as can be done for fast.com) in = https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreport= s-speedtest-bufferbloat-testing/2803 we collect recommendations how to = configure the test for better bufferbloat assessment icludingsetting the = duration to 30 seconds (which at a point in the past was the maximum the = test allowed, but I believe the limit was increased since then; but = since dslreports has to pay for their bandwidth I believe recommending = something shortish is the right thing to do, and in my expereience 30 = seconds is already much better than the typical ~10 second durations for = other speedttests). > and that, without seeing the raw > data (just the inverse of what's being displayed, what dslreports is > throwing out), I worry. It's a cosmic background bufferbloat detector > that it's needed. Still dsl sucks, still there's data at +1 second > (With a cutoff of 4 sec that is an artifact of the test), those really > worry me. >=20 >> The industry as whole MUST pay attention and socialize the relevancy = of managed latencies as being critical to customer satisfaction and good = application performance. And that starts with tests that clearly grade = that critical aspect. >=20 > One can wish. I think only the bufferbloat issue breaking out > thoroughly into the business press, in places like the economist or > cto or fortune, will make a difference here. In the EU getting the european regulators on board and convinced = that bofferbloat is important and to be avoided would help a lot; as = they are responsible for the reference official speedtests in europe. Best Regards Sebastian >=20 >> Cheers, >>=20 >> Jonathan >>=20 >>> On May 15, 2019, at 3:58 AM, Dave Taht wrote: >>>=20 >>> If it helps any: Nick Feamster and Jason Livingood just published " >>> Internet Speed Measurement: Current Challenges and Future >>> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few = days >>> ago, and outlines quite a few problems going forward at higher = speeds. >>> I do wish the document had pointed out more clearly that router = based >>> measurements have problems also, with weaker cpus unable to source >>> enough traffic for an accurate measurement, but I do hope this >>> document has impact, and it's a good read, regardless. >>>=20 >>> Still, somehow getting it right at lower speeds is always on my = mind. >>> I'd long ago hoped that DSL devices would adopt BQL, and that >>> cablemodems would also, thus moving packet processing a little = higher >>> on the stack so more advanced algorithms like cake could take hold. >>>=20 >>> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller = wrote: >>>>=20 >>>> Hi All, >>>>=20 >>>>=20 >>>> I believe the following to be relevant to this discussion: = https://apenwarr.ca/log/20180808 >>>> Where he discusses a similar idea including implementation albeit = aimed at lower bandwidth and sans the automatic bandwidth tracking. >>>>=20 >>>>=20 >>>>> On May 15, 2019, at 01:34, David P. Reed = wrote: >>>>>=20 >>>>>=20 >>>>> Ideally, it would need to be self-configuring, though... I.e., = something >>>>> like the IQRouter auto-measuring of the upstream bandwidth to tune = the >>>>> shaper. >>>>=20 >>>> @Jonathan from your experience how tricky is it to get reliable = speedtest endpoints and how reliable are they in practice. And do you do = any sanitization, like take another measure immediate if the measured = rate differs from the last by more than XX% or something like that? >>>>=20 >>>>=20 >>>>>=20 >>>>> Sure, seems like this is easy to code because there are exactly = two ports to measure, they can even be labeled physically "up" and = "down" to indicate their function. >>>>=20 >>>> IMHO the real challenge is automated measurements over the internet = at Gbps speeds. It is not hard to get some test going (by e.g. tapping = into ookla's fast net of confederated measurement endpoints) but getting = something where the servers can reliably saturate 1Gbps+ seems somewhat = trickier (last time I looked one required a 1Gbps connection to the = server to participate in speedtest.net, obviously not really suited for = measuring Gbps speeds). >>>> In the EU there exists a mandate for national regulators to = establish and/or endorse an anointed "official" speedtests, untended to = keep ISP marketing honest, that come with stricter guarantees (e.g. the = official German speedtest, breitbandmessung.de will only admit tests if = the servers are having sufficient bandwidth reserves to actually = saturate the link; the enduser is required to select the speed-tier = giving them a strong hint about the required rates I believe). >>>> For my back-burner toy project "per-packet-overhead estimation on = arbitrary link technology" I am currently facing the same problem, I = need a traffic sink and source that can reliably saturate my link so I = can measure maximum achievable goodput, so if anybody in the list has = ideas, I am all ears/eyes. >>>>=20 >>>>>=20 >>>>> For reference, the GL.iNet routers are tiny and nicely packaged, = and run >>>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I = very >>>>> much doubt it can actually push a gigabit, though, but I haven't = had a >>>>> chance to test it. However, losing the WiFi, and getting a = slightly >>>>> beefier SoC in there will probably be doable without the price = going >>>>> over $100, no? >>>>>=20 >>>>> I assume the WiFi silicon is probably the most costly piece of = intellectual property in the system. So yeah. Maybe with the right parts = being available, one could aim at $50 or less, without sales channel = markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that = might be because the GigE interfaces are a bit pricey. However, the = ARM64 SoC's available are typically Celeron-class multicore systems. I = don't know why there aren't more ARM64 systems on a chip with dual GigE, = but I suspect searching for them would turn up some). >>>>=20 >>>> The turris MOX (https://www.turris.cz/en/specification/) might be a = decent startimg point as it comes with one Gbethernet port and both a = SGMII and a PCIe signals routed on a connector, they also have a 4 and = an 8 port switch module, but for our purposes it might be possible to = just create a small single Gb ethernet port board to get started. >>>>=20 >>>> Best Regards >>>> Sebastian >>>>=20 >>>>>=20 >>>>> -Toke >>>>>=20 >>>>> [0] https://www.gl-inet.com/products/gl-ar750s/ >>>>> _______________________________________________ >>>>> Cerowrt-devel mailing list >>>>> Cerowrt-devel@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>=20 >>>> _______________________________________________ >>>> Bloat mailing list >>>> Bloat@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/bloat >>>=20 >>>=20 >>>=20 >>> -- >>>=20 >>> Dave T=C3=A4ht >>> CTO, TekLibre, LLC >>> http://www.teklibre.com >>> Tel: 1-831-205-9740 >>> _______________________________________________ >>> Bloat mailing list >>> Bloat@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>=20 >=20 >=20 > -- >=20 > Dave T=C3=A4ht > CTO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-831-205-9740