From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 667493B29E; Fri, 17 May 2019 03:36:17 -0400 (EDT) Received: by mail-it1-x141.google.com with SMTP id e184so10520696ite.1; Fri, 17 May 2019 00:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=/ftPGBeLV+KIva6Xd85X6IENAHEXRVzLmoQowWV0GIE=; b=nb2Xh+kCIQlrVq4vSt7/1hrOqDWtUsATslPkXqxktBGZK07Z3Dryolwb1xaWybBmK9 KlG+AYmrfotT49mDbMq1sTCAL0dzAik6k26MVe4fkPtlMwFgDNThHMbSgIYRoqThQBzu Wgu2BH0Hvq5t9I7xwoBLdFXU5LxlUUdo6T/EwGla499TU4UTT5gXZGD+eMAtoso4PVT6 Be5KZcts302OIq9omw7zO7qxVsoHv1ja6e3n8W6+A67aJlKO8Hmy5iUGby3nd9W0T0vy ksM8nltMNHMW8iZdh5vmgyKqQPCEgbpCvP3tV3vFayZvly8ORon8BlzBLeqj1jsSDNC5 bdJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=/ftPGBeLV+KIva6Xd85X6IENAHEXRVzLmoQowWV0GIE=; b=ZAtEruwF0xQZh/vz8anFDULMcqHC7PVqtD+mD/RDylT3ll+gAIwLEkBGYzv3PM4z67 pjEBsIlsSSeJjEKMtl2Dmt1GdpNTpl/zt3kqQPskvtXb9n6XATUWK+5PksaN7tAcwN2D N73KszBexJjwdcP8Z0L3UhuYVl6Wp6fuk+8pJXAzqvx/QNkYc0CnygK6s7rQ01n95eXs EF08H4jKAUGeJ5KJfAGA++JYmflFFO0MTIYL5XwAWA4soRFD7i2XKrs1B0rP0A19iEVq YPE63kdckVZt16cC2P/LlCJwDgFeji2YpVOQXqx1pvaHniC9+TCf+seXxUZEd+/M44uR vPfw== X-Gm-Message-State: APjAAAWJqN1IOFpeAUlub+JMk07phSnlrEIEjkJuTN496nJGkxOIieHv ldAPjJOCf1/mR5SIm6kZjb8zIA3xs0pf1BPLR9U= X-Google-Smtp-Source: APXvYqykppiTqULw4a5nykaKPAMvuzGyvQXSlhZJ5Ad9r16+oSk2MiBYzRS6HCOZA7aPVbvRxMiuEzYB3HskTy9DcGw= X-Received: by 2002:a24:cec3:: with SMTP id v186mr16938108itg.173.1558078576614; Fri, 17 May 2019 00:36:16 -0700 (PDT) MIME-Version: 1.0 From: Dave Taht Date: Fri, 17 May 2019 09:36:05 +0200 Message-ID: To: Jonathan Foulkes Cc: Sebastian Moeller , Rich Brown , cerowrt-devel , "David P. Reed" , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [Cerowrt-devel] Internet Speed Measurement: Current Challenges and Future Recommendations X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2019 07:36:17 -0000 Jason is on the bloat list, nick feamster used to be, but isn't. Sometimes I wish I could drag in various folk at the FCC, FTC, or some other regulatory body more often. BITAG. Places like that. Having some co-ordinated "Here are our comments on your thing" might be better than the general chaos on the bloat list, and, since google is so ineffective at indexing mailing lists these days some way to raise it in the search results. On Fri, May 17, 2019 at 12:01 AM Jonathan Foulkes wrote: > > Thanks for sharing Dave. > > A good paper, but there are few gaps worthy of mentioning on this list: > > Testing when there is an AQM present means the test must adapt to the cha= llenge of smaller cwnd existing for any one stream, therefore it will take = many more streams to saturate a line with cwnd =3D 30 than if the cwnd is a= llowed to grow to >1,000 > In general, the impact of cwnd relative to saturation and impact on delay= was not visited, and yet it=E2=80=99s critical. One of the reasons for spi= ky delays on high speed lines is the ginormous cwnds hogging the line with = their 800ms+ RTT=E2=80=99s > > Asymmetry of provisioned upload relative to download, at some point, the = ack-stream can be held up by either lack of capacity or bloat in the uplink= . So even though a link can deliver 300Mbps down, a bloated uplink of 5mbps= might never allow that level to be reached. > There are ISPs provisioning truly crazy asymmetric service. > > They do make a good point about the local network, WiFi specifically bein= g the new bottleneck, which is why we included an iperf instance that can b= e started on the IQrouter to help run client to server tests that help spot= local network capacity limits, typically on WiFi. > > Regarding their point about =E2=80=98Cross traffic=E2=80=99 impact on mea= surements, Cake=E2=80=99s per-host / per-target fairness also complicates A= QM-enabled testing from client devices. Which is why we make the built-in s= peed test the arbiter of true line capacity, as it factors for ALL traffic = flowing through the router. But, as you mention, that is also a challenge f= rom a CPU resource standpoint on higher speeds. > > The biggest gap in this paper is not paying sufficient attention to laten= cy as a critical metric, and one that is controllable by an AQM. Bufferbloa= t metrics have more impact on end-user experience than +/- 50Mbps on a 100m= bps baseline. > I was rather miffed they do not even mention the DSLreports.com speedtest= , or the fast.com test, as those are the two that provide a bufferbloat met= ric. I keep hoping government will get it. I note that I'm in general quite pleased with the overall trendlines on dslreports ( http://www.dslreports.com/speedtest/results/bufferbloat?up=3D1 ). A lot of the bloat on cablemodems disappearing is due to uplink speeds getting much better, and holding buffer sizes constant, of course, rather than the docsis 3.1 deployment or our efforts, but I do tend to look at the long fuzzy blue data on the left of the green 30ms line and think that's mostly us. The thing that scares though, is that since that test is exaustively used by those tuning their links with fq_codel, is that that dataset is now quite polluted. So what does reality actually look like? On this trip I encountered network after network essentially in a state of congestion collapse. The other big problems with dslreports is that the test does not run for long enough at this higher rates, and that, without seeing the raw data (just the inverse of what's being displayed, what dslreports is throwing out), I worry. It's a cosmic background bufferbloat detector that it's needed. Still dsl sucks, still there's data at +1 second (With a cutoff of 4 sec that is an artifact of the test), those really worry me. > The industry as whole MUST pay attention and socialize the relevancy of m= anaged latencies as being critical to customer satisfaction and good applic= ation performance. And that starts with tests that clearly grade that criti= cal aspect. One can wish. I think only the bufferbloat issue breaking out thoroughly into the business press, in places like the economist or cto or fortune, will make a difference here. > Cheers, > > Jonathan > > > On May 15, 2019, at 3:58 AM, Dave Taht wrote: > > > > If it helps any: Nick Feamster and Jason Livingood just published " > > Internet Speed Measurement: Current Challenges and Future > > Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days > > ago, and outlines quite a few problems going forward at higher speeds. > > I do wish the document had pointed out more clearly that router based > > measurements have problems also, with weaker cpus unable to source > > enough traffic for an accurate measurement, but I do hope this > > document has impact, and it's a good read, regardless. > > > > Still, somehow getting it right at lower speeds is always on my mind. > > I'd long ago hoped that DSL devices would adopt BQL, and that > > cablemodems would also, thus moving packet processing a little higher > > on the stack so more advanced algorithms like cake could take hold. > > > > On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller wro= te: > >> > >> Hi All, > >> > >> > >> I believe the following to be relevant to this discussion: https://ape= nwarr.ca/log/20180808 > >> Where he discusses a similar idea including implementation albeit aime= d at lower bandwidth and sans the automatic bandwidth tracking. > >> > >> > >>> On May 15, 2019, at 01:34, David P. Reed wrote: > >>> > >>> > >>> Ideally, it would need to be self-configuring, though... I.e., someth= ing > >>> like the IQRouter auto-measuring of the upstream bandwidth to tune th= e > >>> shaper. > >> > >> @Jonathan from your experience how tricky is it to get reliable speedt= est endpoints and how reliable are they in practice. And do you do any sani= tization, like take another measure immediate if the measured rate differs = from the last by more than XX% or something like that? > >> > >> > >>> > >>> Sure, seems like this is easy to code because there are exactly two p= orts to measure, they can even be labeled physically "up" and "down" to ind= icate their function. > >> > >> IMHO the real challenge is automated measurements over the internet at= Gbps speeds. It is not hard to get some test going (by e.g. tapping into o= okla's fast net of confederated measurement endpoints) but getting somethin= g where the servers can reliably saturate 1Gbps+ seems somewhat trickier (l= ast time I looked one required a 1Gbps connection to the server to particip= ate in speedtest.net, obviously not really suited for measuring Gbps speeds= ). > >> In the EU there exists a mandate for national regulators to establish = and/or endorse an anointed "official" speedtests, untended to keep ISP mark= eting honest, that come with stricter guarantees (e.g. the official German = speedtest, breitbandmessung.de will only admit tests if the servers are hav= ing sufficient bandwidth reserves to actually saturate the link; the enduse= r is required to select the speed-tier giving them a strong hint about the = required rates I believe). > >> For my back-burner toy project "per-packet-overhead estimation on arbi= trary link technology" I am currently facing the same problem, I need a tra= ffic sink and source that can reliably saturate my link so I can measure ma= ximum achievable goodput, so if anybody in the list has ideas, I am all ear= s/eyes. > >> > >>> > >>> For reference, the GL.iNet routers are tiny and nicely packaged, and = run > >>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I ve= ry > >>> much doubt it can actually push a gigabit, though, but I haven't had = a > >>> chance to test it. However, losing the WiFi, and getting a slightly > >>> beefier SoC in there will probably be doable without the price going > >>> over $100, no? > >>> > >>> I assume the WiFi silicon is probably the most costly piece of intell= ectual property in the system. So yeah. Maybe with the right parts being av= ailable, one could aim at $50 or less, without sales channel markup. (Raspb= erry Pi ARM64 boards don't have GigE, and I think that might be because the= GigE interfaces are a bit pricey. However, the ARM64 SoC's available are t= ypically Celeron-class multicore systems. I don't know why there aren't mor= e ARM64 systems on a chip with dual GigE, but I suspect searching for them = would turn up some). > >> > >> The turris MOX (https://www.turris.cz/en/specification/) might be a de= cent startimg point as it comes with one Gbethernet port and both a SGMII a= nd a PCIe signals routed on a connector, they also have a 4 and an 8 port s= witch module, but for our purposes it might be possible to just create a sm= all single Gb ethernet port board to get started. > >> > >> Best Regards > >> Sebastian > >> > >>> > >>> -Toke > >>> > >>> [0] https://www.gl-inet.com/products/gl-ar750s/ > >>> _______________________________________________ > >>> Cerowrt-devel mailing list > >>> Cerowrt-devel@lists.bufferbloat.net > >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel > >> > >> _______________________________________________ > >> Bloat mailing list > >> Bloat@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > > > > > > > > -- > > > > Dave T=C3=A4ht > > CTO, TekLibre, LLC > > http://www.teklibre.com > > Tel: 1-831-205-9740 > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > -- Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740