Re: [Bloat] We built a new bufferbloat test and keen for feedback

["Toke Høiland-Jørgensen" <toke@toke.dk>]

Sam Westwood <sam@repeaterstore.com> writes:

> Hi everyone,
>
> My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform
> we provide equipment to help improve cell phone service, and being in the
> industry we've always been interested in all aspects of network
> connectivity. Bufferbloat for us has always been interesting, and while
> there are a few tests out there we never found one that was fantastic. So
> we thought we'd try and build one!
>
> My colleague Arshan has built the test, which we based upon the Cloudflare
> Speedtest template that was discussed earlier in the summer in a previous
> thread.
>
> We measure bufferbloat under two conditions: when downlink is saturated and
> when uplink is saturated. The test involves three stages: Unloaded,
> Downlink Saturated, and Uplink Saturated. In the first stage we simply
> measure latency to a file hosted on a CDN. This is usually around 5ms and
> could vary a bit based on the user's location. We use the webTiming API to
> find the time-to-first-byte, and consider that as the latency. In the
> second stage we run a download, while simultaneously measuring latency. In
> the third stage we do the same but for upload. Both download and upload
> usually take around 5 seconds. We show the median, first quartile and the
> third quartile on distribution charts corresponding to each stage to
> provide a visual representation of the latency variations. For download and
> upload we have used Cloudflare's speedtest backend.

This sounds great, thanks for doing this! It certainly sounds like
you're on the right track here. Some comments below...

> You can find the test here: https://www.waveform.com/apps/dev-arshan
>
> We built testing it on Chrome, but it works on Firefox and mobile too. On
> mobile results may be a little different, as the APIs aren't available and
> so instead we implemented a more manual method, which can be a little
> noisier.
>
> This is a really early alpha, and so we are keen to get any and all
> feedback you have :-). Things that we would particularly like feedback on:
>
>    - How does the bufferbloat measure compare to other tests you may have
>    run on the same connection (e.g. dslreports, fast.com)
>    - How the throughput results (download/upload/latency) look compared to
>    other tools

I'm fortunate enough to have a full Gbps fibre link, which makes it
really hard to saturate the connection from a browser test (or at all,
sometimes). Your test does a decent job and comes pretty close, at least
in Chromium (about 800 Mbps which is not too far off at the application
layer, considering I have a constant 100Mbps flow in the background
taking up some of the bandwidth). Firefox seems way off (one test said
500Mbps the other >1000).

This does mean that I can't say much that's useful about your
bufferbloat scores, unfortunately. The latency measurement puzzled me a
bit (your tool says 16.6ms, but I get half that when I ping the
cloudfront.net CDN, which I think is what you're measuring against?),
but it does seem to stay fairly constant.

How do you calculate the jitter score? It's not obvious how you get from
the percentiles to the jitter.

Link to the test in Chromium:
https://www.waveform.com/apps/dev-arshan?test-id=91a55adc-7513-4b55-b8a6-0fa698ce634e

>    - Any feedback on the user interface of the test itself? We know that
>    before releasing more widely we will put more effort into explaining
>    bufferbloat than we have so far.

Brain dump of thoughts on the UI:

I found it hard to tell whether it was doing anything while the test was
running. Most other tests have some kind of very obvious feedback
(moving graphs of bandwidth-over-time for cloudflare/dslreports, a
honking big number going up and down for fast.com), which I was missing
here. I would also have liked to a measure of bandwidth over time, it
seems a bit suspicious (from a "verify that this is doing something
reasonable" PoV) that it just spits out a number at the end without
telling me how long it ran, or how it got to that number.

It wasn't obvious at first either that the header changes from
"bufferbloat test" to "your bufferbloat grade" once the test is over I
think the stages + result would be better put somewhere else where it's
more obvious (the rest of the page grows downwards, so why isn't the
result at the "end"?).

Also, what are the shields below the grade supposed to mean? Do they
change depending on the result? On which criteria? And it's telling me I
have an A+ grade, so why is there a link to fix my bufferbloat issues?

Smaller nit, I found the up/down arrows in "up saturated" and "down
saturated" a bit hard to grasp at first, I think spelling out
upload/download would be better. Also not sure I like the "saturated"
term in the first place; do people know what that means in a networking
context? And can you be sure the network is actually *being* saturated?

Why is the "before you start" text below the page? Shouldn't it be at
the top? And maybe explain *why* this is important?

>    - Anything else you would like to give feedback on?

As far as the web page itself is concerned, holy cross-domain script
deluge, Batman! I'm one of those people who run uMatrix in my regular
Firefox session, and I disallow all cross-site script loads by default.
I had to allow 15(!) different cross-site requests, *and* whitelist a
few domains in my ad blocker as well to even get the test to run. Please
fix this! I get that you can't completely avoid cross-domain requests
due to the nature of the test, but why is a speedtest pulling in scripts
from 'shopify.com' and three different ad tracking networks?

I realise some of this may just be because you put this thing on your
corporate web site, and thus a lot of it is not directly related to the
speed test itself. But all the same, having to wade through chest-high
piles of javascript just to get to the test doesn't exactly scream
"trustworthy company that just wants to help me fight bufferbloat" :)

(For comparison, dslreports, fast.com and speed.cloudflare.com all pull
in scripts from exactly three domains, all of them directly related to
the test, and none of them fail with my default uBlock blocklist).

-Toke