From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [45.145.95.4]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 620023CB38 for ; Thu, 5 Nov 2020 12:25:10 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1604597109; bh=b03+LZho9WjJl6ZFJ0y65jMn/wT0tle4LKQyUFRkDVo=; h=From:To:Subject:In-Reply-To:References:Date:From; b=aqiJHhVrMv5wtUyTrllOijUVhYYmVCfDZBe8KSqYW1Pmbf1ARRisCPJNpsRlP69Jc Py4lHhBP4qVRdj9WLKWOPT1uFzpGYGObqjn/BwOFRsoyuFHRJTQz6IPu0s62eeZRIE Y5dJZTWGA3n0n4tOUF+ZeJ426o3hEUja6INdZn0r/GIL9XMHU4YhWSpJ0UhZMEAMH6 vwQv8hr1JYGdS9SrLFdkDJDh+jhXl3PMHGBztEf6BLK/qPLwNYQ0g1VpnG70FFuRPM rVAqL+r1Nd3QpjaQuZZzcL8VctBlzRU6tjIgKzzNlCxBH8PTJCVW0S0rszhtXAa8oC qwJJZTrE7paig== To: Arshan Khanifar , bloat@lists.bufferbloat.net In-Reply-To: References: <87mtzwt47a.fsf@toke.dk> <87wnyzswy9.fsf@toke.dk> Date: Thu, 05 Nov 2020 18:25:07 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87h7q3solo.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 17:25:10 -0000 > I believe TLS handshake time is not included here. I=E2=80=99m using the > Resource Timing API > > to measure the time-to-first-byte for a request that I=E2=80=99m sending = to > retrieve a static file. The resource loading phases > > section of the documentation explicitly shows the different stages for > DNS Lookup, TCP connection establishment, etc. I=E2=80=99m using the > difference between requestStart and responseStart values. This value > is deemed to be the same as time-to-first-byte > > seen in the inspector=E2=80=99s network tab. This does not seem completely ludicrous, at least :) > We=E2=80=99re using this static file > > that is hosted on a google CDN. We tried multiple different files, and > this one had the lowest latency in both locations that we tested it > (I=E2=80=99m in Toronto, and my colleague Sina is in San Francisco). Ah, so that's why that request showed up :) Curious to know why you picked this instead of, say, something from speed.cloudflare.com (since you're using that for the speed tests anyway)? > @Toke H=C3=B8iland-J=C3=B8rgensen >> Your test does a decent job and comes pretty close, at least >> in Chromium (about 800 Mbps which is not too far off at the application >> layer, considering I have a constant 100Mbps flow in the background >> taking up some of the bandwidth). Firefox seems way off (one test said >> 500Mbps the other >1000). > > > The way I=E2=80=99m measuring download is that I make multiple simultaneo= us > requests to cloudflare=E2=80=99s backend requesting 100MB files. Their ba= ckend > simply returns a file that has =E2=80=9C0=E2=80=9Ds in the body repeated = until 100MB > of file is generated. Then I use readable streams > > to make multiple measurements of (total bytes downloaded, timestamp). > Then I fit a line to the measurements collected, and the slope of that > line is the calculated bandwidth. For gigabit connections, this > download happens very quickly, and it may be the case that not a lot > of points are collected, in which case the fitted line is not accurate > and one might get overly-huge bandwidths as is the >1000 case in ur > Firefox browser. I think this might be fixed if we increase the > download time. Currently it=E2=80=99s 5s, maybe changing that to 10-20s w= ould > help. I think in general it=E2=80=99d be a good feature to have a "more > advanced options=E2=80=9D feature that allows the user to adjust some > parameters of the connection (such as number of parallel connections, > download scenario=E2=80=99s duration, upload scenario=E2=80=99s duration,= etc.) Yeah, I think running the test for longer will help; 5s is not nearly enough to saturate a connection, especially not as the link speed increases. > The reason I do this line-fitting is because I want to get rid of the > bandwidth ramp-up time when the download begins. Yeah, allowing some ramp-up time before determining the bandwidth seems reasonable, but it's not generally possible to just pick a static number of (say) seconds to chop off... Having the graph over time helps sanity-check things, though. Also, a continuous graph of latency samples over time (for the whole duration, including idle/upload/download) is usually very instructive when plotting such a test. > Real-time Bandwidth Reporting > Using readable-streams also allows for instantaneous bandwidth > reporting (maybe using average of a moving window) similar to what > fast.com or speedtest.net > do, but I unfortunately am not able to do the same thing with upload, > since getting progress on http uploads adds some pre-flight OPTIONS > requests which cloudflare=E2=80=99s speedtest backend > doesn=E2=80=99t allow those requests. For= this > test we are directly hitting cloudflare=E2=80=99s backend, you can see th= is in > the network tab: > > Our download is by sending an http GET request to this endpoint: > https://speed.cloudflare.com/__down?bytes=3D100000000 > and our upload > is done by sending and http POST request to this endpoint: > https://speed.cloudflare.com/__up > > Since we are using cloudflare=E2=80=99s backend we are limited by what th= ey > allow us to do. The test at speed.cloudflare.com does seem to plot real-time upload bandwidth; is that a privileged operation for themselves, or something? > I did try making my own worker which essentially does the same thing > as cloudflare=E2=80=99s speedtest backend (They do have this template wor= ker > that for the > most part does the same thing.) I modified that worker a bit so that > it allows http progress on upload for real-time measurements, but we > hit another wall with that: we could not saturate gigabit internet > connections. Turns out that cloudflare has tiers for the workers and > the bundle tier that we are using doesn=E2=80=99t get the most priority in > terms of bandwidth, so we could only get up to 600mbps measurements. > Their own speed test is hosted on an enterprise tier, which is around > $6-7k USD and is way too expensive. We are however, requesting for a > demo from them, and it=E2=80=99s an ongoing progress. > > So since we can measure instantaneous download speeds but not upload > speeds, we don=E2=80=99t report it for either one. But I can still make t= he > adjustment to report it for download at least. Sure, better than nothing. > @Toke H=C3=B8iland-J=C3=B8rgensen >> How do you calculate the jitter score? It's not obvious how you get from >> the percentiles to the jitter. > > Jitter here is the standard deviation of the latency measurements in > each stage. Is this a good definition? If you're reporting standard deviation, I'd just label it as such. One good definition for jitter is IPDV: https://en.wikipedia.org/wiki/Packet_delay_variation > @Toke H=C3=B8iland-J=C3=B8rgensen >> Also, what are the shields below the grade supposed to mean? Do they >> change depending on the result? On which criteria?=20 > > > They do change! The criteria are listed below. Note that in the criteria = below: > Latency is calculated as the maximum median of latency across all three s= tages. > Latency with Jitter is calculated as the maximum of (median + std) acros= s all three stages. > > Web Browsing: > Downlink: > 1mbps > Uplink: > 1mbps > > Audio Calls: > Downlink: > 3mbps > Uplink: > 3mbps > Latency: < 150ms > Latency with Jitter: < 200ms > > 4K Video Streaming: > Downlink: > 40mbps > > Video Conferencing: > Downlink: > 2.5mbps > Uplink: > 2.5mbps > Latency: < 150ms > Latency with Jitter: < 200ms > > Online Gaming: > Downlink: > 3mbps > Uplink: > 0.5mbps > Latency: < 100ms > Latency with Jitter: < 150ms > > For the bufferbloat grade we use the same criteria as DSL reports > . Right, cool; explaining this on the page might be useful ;) > @Toke H=C3=B8iland-J=C3=B8rgensen=20 >> As far as the web page itself is concerned, holy cross-domain script >> deluge, Batman! I'm one of those people who run uMatrix in my regular >> Firefox session, and I disallow all cross-site script loads by default. >> I had to allow 15(!) different cross-site requests, *and* whitelist a >> few domains in my ad blocker as well to even get the test to run. Please >> fix this! I get that you can't completely avoid cross-domain requests >> due to the nature of the test, but why is a speedtest pulling in scripts >> from 'shopify.com' and three different ad tracking networks? > > > Hahahah this is because we=E2=80=99re using Shopify App Proxies > . > It=E2=80=99s a technique we use to import assets from our main store, and= make > it appear such that this is part of our main store whereas in reality > it=E2=80=99s a separately-hosted application. This allows us to bring in = the > header, footer and the chatbot. This is a really good point though, I > wonder what we can do with that. Just having the test be on a separate (static) page would be the obvious fix ;) -Toke