[Bloat] Internet Speed Measurement: Current Challenges and Future Recommendations

Fri May 17 04:29:49 EDT 2019

> On May 17, 2019, at 09:36, Dave Taht <dave.taht at gmail.com> wrote:
> 
> Jason is on the bloat list, nick feamster used to be, but isn't.
> Sometimes I wish I could drag in various folk at the FCC, FTC, or some
> other regulatory body more often. BITAG. Places like that. Having some
> co-ordinated "Here are our comments on your thing" might be better
> than the general chaos on the bloat list, and, since google is so
> ineffective at indexing mailing lists these days some way to raise it
> in the search results.
> 
> On Fri, May 17, 2019 at 12:01 AM Jonathan Foulkes
> <jf at jonathanfoulkes.com> wrote:
>> 
>> Thanks for sharing Dave.
>> 
>> A good paper, but there are few gaps worthy of mentioning on this list:
>> 
>> Testing when there is an AQM present means the test must adapt to the challenge of smaller cwnd existing for any one stream, therefore it will take many more streams to saturate a line with cwnd = 30 than if the cwnd is allowed to grow to >1,000
>> In general, the impact of cwnd relative to saturation and impact on delay was not visited, and yet it’s critical. One of the reasons for spiky delays on high speed lines is the ginormous cwnds hogging the line with their 800ms+ RTT’s

	Good point, I note that the detailed dslreports results also contain information about the average cwnd, and I admit that so far I have never looked at those numbers; will do in the future

>> 
>> Asymmetry of provisioned upload relative to download, at some point, the ack-stream can be held up by either lack of capacity or bloat in the uplink. So even though a link can deliver 300Mbps down, a bloated uplink of 5mbps might never allow that level to be reached.

IIRC a back of the envelop estimate assuming MTU~1500 and one ACK every two full MSS packets gives the following percentage of ACK traffic for a given amount of reverse traffic (I assume a cable link here for overhead, but that ratio does not depend too much on the actual per-packet-overhead):
100 * ((18+(20+20+12))/(1500+18)) = 4.61% ~. 1/20 ~ 5%
Techniques like ACK filtering/thinning and GSO/GRO can reduce this number considerably. So the 300 Mbps link needs to have at least 300*0.05 = 15 Mbps uplink to allow saturation with bog-standard TCP-Reno flows. Most asymmetries I have seen in the wild seem to keep this in mind, but I would not be amazed to see something like 300/5 in the wild ;)

>> There are ISPs provisioning truly crazy asymmetric service.

	+1; or rather -1 as while this is true I do not want to endorse that

>> 
>> They do make a good point about the local network, WiFi specifically being the new bottleneck, which is why we included an iperf instance that can be started on the IQrouter to help run client to server tests that help spot local network capacity limits, typically on WiFi.

	Enlightened as so often. I note the official German speedtest will ask for lan/wlan connection and will only consider tests via lan as oficial (their desktop app even will not run over wifi at all, which I find obnoxious).

>> 
>> Regarding their point about ‘Cross traffic’ impact on measurements, Cake’s per-host / per-target fairness also complicates AQM-enabled testing from client devices. Which is why we make the built-in speed test the arbiter of true line capacity, as it factors for ALL traffic flowing through the router. 

	+1. There is  a toy project going on in the openwrt forum trying to collect traffic, CPU load & frequency, and RTT data for concurrently running saturating loads; the idea being the same as yours, measuring the ingress/egress traffic directly at the router and just using the speedtest(s) as a load generator(s). But that project has not gone very far. (It statred from the question what shaper performance one can expect from different SoCs)

>> But, as you mention, that is also a challenge from a CPU resource standpoint on higher speeds.

	Yes this is why monitoring the CPU load during the test seems like a great idea.

>> 
>> The biggest gap in this paper is not paying sufficient attention to latency as a critical metric, and one that is controllable by an AQM. Bufferbloat metrics have more impact on end-user experience than +/- 50Mbps on a 100mbps baseline.
>> I was rather miffed they do not even mention the DSLreports.com speedtest, or the fast.com test, as those are the two that provide a bufferbloat metric.

	I might add that https://www.thinkbroadband.com/speedtest also does a few things well, a) they do run a single stream and a 6 stream test in sequence and report the numbers for both and b) they also do a latency-under-load test (click the Analysis button in the first results graph) . Unfortunately the latency-under-load grpahs is inaccessible from the main and final results page...

> 
> I keep hoping government will get it.
> 
> I note that I'm in general quite pleased with the overall trendlines
> on dslreports (
> http://www.dslreports.com/speedtest/results/bufferbloat?up=1 ). A lot
> of the bloat on cablemodems disappearing is due to uplink speeds
> getting much better, and holding buffer sizes constant, of course,
> rather than the docsis 3.1 deployment or our efforts, but I do tend to
> look at the long fuzzy blue data on the left of the green 30ms line
> and think that's mostly us.
> 
> The thing that scares though, is that since that test is exaustively
> used by those tuning their links with fq_codel, is that that dataset
> is now quite polluted. So what does reality actually look like? On
> this trip I encountered network after network essentially in a state
> of congestion collapse.
> 
> The other big problems with dslreports is that the test does not run
> for long enough at this higher rates,

	After a free registration, one can configure the test duration (as can be done for fast.com) in https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803 we collect recommendations how to configure the test for better bufferbloat assessment icludingsetting the duration to 30 seconds (which at a point in the past was the maximum the test allowed, but I believe the limit was increased since then; but since dslreports has to pay for their bandwidth I believe recommending something shortish is the right thing to do, and in my expereience 30 seconds is already much better than the typical ~10 second durations for other speedttests).

> and that, without seeing the raw
> data (just the inverse of what's being displayed, what dslreports is
> throwing out), I worry. It's a cosmic background bufferbloat detector
> that it's needed. Still dsl sucks, still there's data at +1 second
> (With a cutoff of 4 sec that is an artifact of the test), those really
> worry me.
> 
>> The industry as whole MUST pay attention and socialize the relevancy of managed latencies as being critical to customer satisfaction and good application performance. And that starts with tests that clearly grade that critical aspect.
> 
> One can wish. I think only the bufferbloat issue breaking out
> thoroughly into the business press, in places like the economist or
> cto or fortune, will make a difference here.

	In the EU getting the european regulators on board and convinced that bofferbloat is important and to be avoided would help a lot; as they are responsible for the reference official speedtests in europe.

Best Regards
	Sebastian

> 
>> Cheers,
>> 
>> Jonathan
>> 
>>> On May 15, 2019, at 3:58 AM, Dave Taht <dave.taht at gmail.com> wrote:
>>> 
>>> If it helps any: Nick Feamster and Jason Livingood just published "
>>> Internet Speed Measurement: Current Challenges and Future
>>> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
>>> ago, and outlines quite a few problems going forward at higher speeds.
>>> I do wish the document had pointed out more clearly that router based
>>> measurements have problems also, with weaker cpus unable to source
>>> enough traffic for an accurate measurement, but I do hope this
>>> document has impact, and it's a good read, regardless.
>>> 
>>> Still, somehow getting it right at lower speeds is always on my mind.
>>> I'd long ago hoped that DSL devices would adopt BQL, and that
>>> cablemodems would also, thus moving packet processing a little higher
>>> on the stack so more advanced algorithms like cake could take hold.
>>> 
>>> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <moeller0 at gmx.de> wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> 
>>>> I believe the following to be relevant to this discussion: https://apenwarr.ca/log/20180808
>>>> Where he discusses a similar idea including implementation albeit aimed at lower bandwidth and sans the automatic bandwidth tracking.
>>>> 
>>>> 
>>>>> On May 15, 2019, at 01:34, David P. Reed <dpreed at deepplum.com> wrote:
>>>>> 
>>>>> 
>>>>> Ideally, it would need to be self-configuring, though... I.e., something
>>>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>>>> shaper.
>>>> 
>>>> @Jonathan from your experience how tricky is it to get reliable speedtest endpoints and how reliable are they in practice. And do you do any sanitization, like take another measure immediate if the measured rate differs from the last by more than XX% or something like that?
>>>> 
>>>> 
>>>>> 
>>>>> Sure, seems like this is easy to code because there are exactly two ports to measure, they can even be labeled physically "up" and "down" to indicate their function.
>>>> 
>>>> IMHO the real challenge is automated measurements over the internet at Gbps speeds. It is not hard to get some test going (by e.g. tapping into ookla's fast net of confederated measurement endpoints) but getting something where the servers can reliably saturate 1Gbps+ seems somewhat trickier (last time I looked one required a 1Gbps connection to the server to participate in speedtest.net, obviously not really suited for measuring Gbps speeds).
>>>> In the EU there exists a mandate for national regulators to establish and/or endorse an anointed "official" speedtests, untended to keep ISP marketing honest, that come with stricter guarantees (e.g. the official German speedtest, breitbandmessung.de will only admit tests if the servers are having sufficient bandwidth reserves to actually saturate the link; the enduser is required to select the speed-tier giving them a strong hint about the required rates I believe).
>>>> For my back-burner toy project "per-packet-overhead estimation on arbitrary link technology" I am currently facing the same problem, I need a traffic sink and source that can reliably saturate my link so I can measure maximum achievable goodput, so if anybody in the list has ideas, I am all ears/eyes.
>>>> 
>>>>> 
>>>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>>>> chance to test it. However, losing the WiFi, and getting a slightly
>>>>> beefier SoC in there will probably be doable without the price going
>>>>> over $100, no?
>>>>> 
>>>>> I assume the WiFi silicon is probably the most costly piece of intellectual property in the system. So yeah. Maybe with the right parts being available, one could aim at $50 or less, without sales channel markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that might be because the GigE interfaces are a bit pricey. However, the ARM64 SoC's available are typically Celeron-class multicore systems. I don't know why there aren't more ARM64 systems on a chip with dual GigE, but I suspect searching for them would turn up some).
>>>> 
>>>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent startimg point as it comes with one Gbethernet port and both a SGMII and a PCIe signals routed on a connector, they also have a 4 and an 8 port switch module, but for our purposes it might be possible to just create a small single Gb ethernet port board to get started.
>>>> 
>>>> Best Regards
>>>>       Sebastian
>>>> 
>>>>> 
>>>>> -Toke
>>>>> 
>>>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>>>> _______________________________________________
>>>>> Cerowrt-devel mailing list
>>>>> Cerowrt-devel at lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>>> 
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat at lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> 
> 
> 
> --
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740