[Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Stephen Hemminger]

On Thu, 05 Nov 2020 18:25:07 +0100
Toke Høiland-Jørgensen via Bloat <bloat at lists.bufferbloat.net> wrote:

> > I believe TLS handshake time is not included here. I’m using the
> > Resource Timing API
> > <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API>
> > to measure the time-to-first-byte for a request that I’m sending to
> > retrieve a static file. The resource loading phases
> > <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API>
> > section of the documentation explicitly shows the different stages for
> > DNS Lookup, TCP connection establishment, etc. I’m using the
> > difference between requestStart and responseStart values. This value
> > is deemed to be the same as time-to-first-byte
> > <https://stackoverflow.com/questions/6533456/time-to-first-byte-with-javascript>
> > seen in the inspector’s network tab.  
> 
> This does not seem completely ludicrous, at least :)
> 
> > We’re using this static file
> > <https://fonts.gstatic.com/l/font?kit=KFOmCnqEu92Fr1Me4GZNCzcPK4I&skey=a0a0114a1dcab3ac&v=v20>
> > that is hosted on a google CDN. We tried multiple different files, and
> > this one had the lowest latency in both locations that we tested it
> > (I’m in Toronto, and my colleague Sina is in San Francisco).  
> 
> Ah, so that's why that request showed up :)
> 
> Curious to know why you picked this instead of, say, something from
> speed.cloudflare.com (since you're using that for the speed tests anyway)?
> 
> > @Toke Høiland-Jørgensen  
> >> Your test does a decent job and comes pretty close, at least
> >> in Chromium (about 800 Mbps which is not too far off at the application
> >> layer, considering I have a constant 100Mbps flow in the background
> >> taking up some of the bandwidth). Firefox seems way off (one test said
> >> 500Mbps the other >1000).  
> >
> >
> > The way I’m measuring download is that I make multiple simultaneous
> > requests to cloudflare’s backend requesting 100MB files. Their backend
> > simply returns a file that has “0”s in the body repeated until 100MB
> > of file is generated. Then I use readable streams
> > <https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams>
> > to make multiple measurements of (total bytes downloaded, timestamp).
> > Then I fit a line to the measurements collected, and the slope of that
> > line is the calculated bandwidth. For gigabit connections, this
> > download happens very quickly, and it may be the case that not a lot
> > of points are collected, in which case the fitted line is not accurate
> > and one might get overly-huge bandwidths as is the >1000 case in ur
> > Firefox browser. I think this might be fixed if we increase the
> > download time. Currently it’s 5s, maybe changing that to 10-20s would
> > help. I think in general it’d be a good feature to have a "more
> > advanced options” feature that allows the user to adjust some
> > parameters of the connection (such as number of parallel connections,
> > download scenario’s duration, upload scenario’s duration, etc.)  
> 
> Yeah, I think running the test for longer will help; 5s is not nearly
> enough to saturate a connection, especially not as the link speed increases.
> 
> > The reason I do this line-fitting is because I want to get rid of the
> > bandwidth ramp-up time when the download begins.  
> 
> Yeah, allowing some ramp-up time before determining the bandwidth seems
> reasonable, but it's not generally possible to just pick a static number
> of (say) seconds to chop off... Having the graph over time helps
> sanity-check things, though.
> 
> Also, a continuous graph of latency samples over time (for the whole
> duration, including idle/upload/download) is usually very instructive
> when plotting such a test.
> 
> > Real-time Bandwidth Reporting
> > Using readable-streams also allows for instantaneous bandwidth
> > reporting (maybe using average of a moving window) similar to what
> > fast.com <http://fast.com/> or speedtest.net <http://speedtest.net/>
> > do, but I unfortunately am not able to do the same thing with upload,
> > since getting progress on http uploads adds some pre-flight OPTIONS
> > requests which cloudflare’s speedtest backend
> > <https://speed.cloudflare.com/> doesn’t allow those requests. For this
> > test we are directly hitting cloudflare’s backend, you can see this in
> > the network tab:
> >
> > Our download is by sending an http GET request to this endpoint:
> > https://speed.cloudflare.com/__down?bytes=100000000
> > <https://speed.cloudflare.com/__down?bytes=100000000> and our upload
> > is done by sending and http POST request to this endpoint:
> > https://speed.cloudflare.com/__up <https://speed.cloudflare.com/__up>
> >
> > Since we are using cloudflare’s backend we are limited by what they
> > allow us to do.  
> 
> The test at speed.cloudflare.com does seem to plot real-time upload
> bandwidth; is that a privileged operation for themselves, or something?
> 
> > I did try making my own worker which essentially does the same thing
> > as cloudflare’s speedtest backend (They do have this template worker
> > <https://github.com/cloudflare/worker-speedtest-template> that for the
> > most part does the same thing.) I modified that worker a bit so that
> > it allows http progress on upload for real-time measurements, but we
> > hit another wall with that: we could not saturate gigabit internet
> > connections. Turns out that cloudflare has tiers for the workers and
> > the bundle tier that we are using doesn’t get the most priority in
> > terms of bandwidth, so we could only get up to 600mbps measurements.
> > Their own speed test is hosted on an enterprise tier, which is around
> > $6-7k USD and is way too expensive. We are however, requesting for a
> > demo from them, and it’s an ongoing progress.
> >
> > So since we can measure instantaneous download speeds but not upload
> > speeds, we don’t report it for either one. But I can still make the
> > adjustment to report it for download at least.  
> 
> Sure, better than nothing.
> 
> > @Toke Høiland-Jørgensen  
> >> How do you calculate the jitter score? It's not obvious how you get from
> >> the percentiles to the jitter.  
> >
> > Jitter here is the standard deviation of the latency measurements in
> > each stage. Is this a good definition?  
> 
> If you're reporting standard deviation, I'd just label it as such. One
> good definition for jitter is IPDV:
> https://en.wikipedia.org/wiki/Packet_delay_variation
> 
> > @Toke Høiland-Jørgensen  
> >> Also, what are the shields below the grade supposed to mean? Do they
> >> change depending on the result? On which criteria?   
> >
> >
> > They do change! The criteria are listed below. Note that in the criteria below:
> > Latency is calculated as the maximum median of latency across all three stages.
> > Latency with Jitter is calculated as the maximum  of (median + std) across all three stages.
> >
> > Web Browsing:
> > Downlink: > 1mbps
> > Uplink: > 1mbps
> >
> > Audio Calls:
> > Downlink: > 3mbps
> > Uplink: > 3mbps
> > Latency: < 150ms
> > Latency with Jitter: < 200ms
> >
> > 4K Video Streaming:
> > Downlink: > 40mbps
> >
> > Video Conferencing:
> > Downlink: > 2.5mbps
> > Uplink: > 2.5mbps
> > Latency: < 150ms
> > Latency with Jitter: < 200ms
> >
> > Online Gaming:
> > Downlink: > 3mbps
> > Uplink: > 0.5mbps
> > Latency: < 100ms
> > Latency with Jitter: < 150ms
> >
> > For the bufferbloat grade we use the same criteria as DSL reports
> > <http://www.dslreports.com/faq/17930>.  
> 
> Right, cool; explaining this on the page might be useful ;)
> 
> > @Toke Høiland-Jørgensen   
> >> As far as the web page itself is concerned, holy cross-domain script
> >> deluge, Batman! I'm one of those people who run uMatrix in my regular
> >> Firefox session, and I disallow all cross-site script loads by default.
> >> I had to allow 15(!) different cross-site requests, *and* whitelist a
> >> few domains in my ad blocker as well to even get the test to run. Please
> >> fix this! I get that you can't completely avoid cross-domain requests
> >> due to the nature of the test, but why is a speedtest pulling in scripts
> >> from 'shopify.com' and three different ad tracking networks?  
> >
> >
> > Hahahah this is because we’re using Shopify App Proxies
> > <https://shopify.dev/tutorials/display-data-on-an-online-store-with-an-application-proxy-app-extension>.
> > It’s a technique we use to import assets from our main store, and make
> > it appear such that this is part of our main store whereas in reality
> > it’s a separately-hosted application. This allows us to bring in the
> > header, footer and the chatbot. This is a really good point though, I
> > wonder what we can do with that.  
> 
> Just having the test be on a separate (static) page would be the obvious
> fix ;)
> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

The results for me look similar to other previous tests like the old dslreports.
Running on OpenWrt and using Wave cable which has reported 250M down and 10M up.

PS: Why to US providers have such asymmetric bandwidth? Getting something symmetric
requires going to a $$$ business rate.

https://www.waveform.com/apps/dev-arshan?test-id=a77df4a0-73cc-4375-8b0e-58e315aaffcf

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2020-11-06 08-01-30.png
Type: image/png
Size: 103325 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20201106/bdadd6b7/attachment-0001.png>