[Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Toke Høiland-Jørgensen]

> I believe TLS handshake time is not included here. I’m using the
> Resource Timing API
> <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API>
> to measure the time-to-first-byte for a request that I’m sending to
> retrieve a static file. The resource loading phases
> <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API>
> section of the documentation explicitly shows the different stages for
> DNS Lookup, TCP connection establishment, etc. I’m using the
> difference between requestStart and responseStart values. This value
> is deemed to be the same as time-to-first-byte
> <https://stackoverflow.com/questions/6533456/time-to-first-byte-with-javascript>
> seen in the inspector’s network tab.

This does not seem completely ludicrous, at least :)

> We’re using this static file
> <https://fonts.gstatic.com/l/font?kit=KFOmCnqEu92Fr1Me4GZNCzcPK4I&skey=a0a0114a1dcab3ac&v=v20>
> that is hosted on a google CDN. We tried multiple different files, and
> this one had the lowest latency in both locations that we tested it
> (I’m in Toronto, and my colleague Sina is in San Francisco).

Ah, so that's why that request showed up :)

Curious to know why you picked this instead of, say, something from
speed.cloudflare.com (since you're using that for the speed tests anyway)?

> @Toke Høiland-Jørgensen
>> Your test does a decent job and comes pretty close, at least
>> in Chromium (about 800 Mbps which is not too far off at the application
>> layer, considering I have a constant 100Mbps flow in the background
>> taking up some of the bandwidth). Firefox seems way off (one test said
>> 500Mbps the other >1000).
>
>
> The way I’m measuring download is that I make multiple simultaneous
> requests to cloudflare’s backend requesting 100MB files. Their backend
> simply returns a file that has “0”s in the body repeated until 100MB
> of file is generated. Then I use readable streams
> <https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams>
> to make multiple measurements of (total bytes downloaded, timestamp).
> Then I fit a line to the measurements collected, and the slope of that
> line is the calculated bandwidth. For gigabit connections, this
> download happens very quickly, and it may be the case that not a lot
> of points are collected, in which case the fitted line is not accurate
> and one might get overly-huge bandwidths as is the >1000 case in ur
> Firefox browser. I think this might be fixed if we increase the
> download time. Currently it’s 5s, maybe changing that to 10-20s would
> help. I think in general it’d be a good feature to have a "more
> advanced options” feature that allows the user to adjust some
> parameters of the connection (such as number of parallel connections,
> download scenario’s duration, upload scenario’s duration, etc.)

Yeah, I think running the test for longer will help; 5s is not nearly
enough to saturate a connection, especially not as the link speed increases.

> The reason I do this line-fitting is because I want to get rid of the
> bandwidth ramp-up time when the download begins.

Yeah, allowing some ramp-up time before determining the bandwidth seems
reasonable, but it's not generally possible to just pick a static number
of (say) seconds to chop off... Having the graph over time helps
sanity-check things, though.

Also, a continuous graph of latency samples over time (for the whole
duration, including idle/upload/download) is usually very instructive
when plotting such a test.

> Real-time Bandwidth Reporting
> Using readable-streams also allows for instantaneous bandwidth
> reporting (maybe using average of a moving window) similar to what
> fast.com <http://fast.com/> or speedtest.net <http://speedtest.net/>
> do, but I unfortunately am not able to do the same thing with upload,
> since getting progress on http uploads adds some pre-flight OPTIONS
> requests which cloudflare’s speedtest backend
> <https://speed.cloudflare.com/> doesn’t allow those requests. For this
> test we are directly hitting cloudflare’s backend, you can see this in
> the network tab:
>
> Our download is by sending an http GET request to this endpoint:
> https://speed.cloudflare.com/__down?bytes=100000000
> <https://speed.cloudflare.com/__down?bytes=100000000> and our upload
> is done by sending and http POST request to this endpoint:
> https://speed.cloudflare.com/__up <https://speed.cloudflare.com/__up>
>
> Since we are using cloudflare’s backend we are limited by what they
> allow us to do.

The test at speed.cloudflare.com does seem to plot real-time upload
bandwidth; is that a privileged operation for themselves, or something?

> I did try making my own worker which essentially does the same thing
> as cloudflare’s speedtest backend (They do have this template worker
> <https://github.com/cloudflare/worker-speedtest-template> that for the
> most part does the same thing.) I modified that worker a bit so that
> it allows http progress on upload for real-time measurements, but we
> hit another wall with that: we could not saturate gigabit internet
> connections. Turns out that cloudflare has tiers for the workers and
> the bundle tier that we are using doesn’t get the most priority in
> terms of bandwidth, so we could only get up to 600mbps measurements.
> Their own speed test is hosted on an enterprise tier, which is around
> $6-7k USD and is way too expensive. We are however, requesting for a
> demo from them, and it’s an ongoing progress.
>
> So since we can measure instantaneous download speeds but not upload
> speeds, we don’t report it for either one. But I can still make the
> adjustment to report it for download at least.

Sure, better than nothing.

> @Toke Høiland-Jørgensen
>> How do you calculate the jitter score? It's not obvious how you get from
>> the percentiles to the jitter.
>
> Jitter here is the standard deviation of the latency measurements in
> each stage. Is this a good definition?

If you're reporting standard deviation, I'd just label it as such. One
good definition for jitter is IPDV:
https://en.wikipedia.org/wiki/Packet_delay_variation

> @Toke Høiland-Jørgensen
>> Also, what are the shields below the grade supposed to mean? Do they
>> change depending on the result? On which criteria? 
>
>
> They do change! The criteria are listed below. Note that in the criteria below:
> Latency is calculated as the maximum median of latency across all three stages.
> Latency with Jitter is calculated as the maximum  of (median + std) across all three stages.
>
> Web Browsing:
> Downlink: > 1mbps
> Uplink: > 1mbps
>
> Audio Calls:
> Downlink: > 3mbps
> Uplink: > 3mbps
> Latency: < 150ms
> Latency with Jitter: < 200ms
>
> 4K Video Streaming:
> Downlink: > 40mbps
>
> Video Conferencing:
> Downlink: > 2.5mbps
> Uplink: > 2.5mbps
> Latency: < 150ms
> Latency with Jitter: < 200ms
>
> Online Gaming:
> Downlink: > 3mbps
> Uplink: > 0.5mbps
> Latency: < 100ms
> Latency with Jitter: < 150ms
>
> For the bufferbloat grade we use the same criteria as DSL reports
> <http://www.dslreports.com/faq/17930>.

Right, cool; explaining this on the page might be useful ;)

> @Toke Høiland-Jørgensen 
>> As far as the web page itself is concerned, holy cross-domain script
>> deluge, Batman! I'm one of those people who run uMatrix in my regular
>> Firefox session, and I disallow all cross-site script loads by default.
>> I had to allow 15(!) different cross-site requests, *and* whitelist a
>> few domains in my ad blocker as well to even get the test to run. Please
>> fix this! I get that you can't completely avoid cross-domain requests
>> due to the nature of the test, but why is a speedtest pulling in scripts
>> from 'shopify.com' and three different ad tracking networks?
>
>
> Hahahah this is because we’re using Shopify App Proxies
> <https://shopify.dev/tutorials/display-data-on-an-online-store-with-an-application-proxy-app-extension>.
> It’s a technique we use to import assets from our main store, and make
> it appear such that this is part of our main store whereas in reality
> it’s a separately-hosted application. This allows us to bring in the
> header, footer and the chatbot. This is a really good point though, I
> wonder what we can do with that.

Just having the test be on a separate (static) page would be the obvious
fix ;)

-Toke