[Bloat] We built a new bufferbloat test and keen for feedback

[Bloat] We built a new bufferbloat test and keen for feedback

[Sam Westwood]

[-- Attachment #1: Type: text/plain, Size: 2643 bytes --]

Hi everyone,

My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform
we provide equipment to help improve cell phone service, and being in the
industry we've always been interested in all aspects of network
connectivity. Bufferbloat for us has always been interesting, and while
there are a few tests out there we never found one that was fantastic. So
we thought we'd try and build one!

My colleague Arshan has built the test, which we based upon the Cloudflare
Speedtest template that was discussed earlier in the summer in a previous
thread.

We measure bufferbloat under two conditions: when downlink is saturated and
when uplink is saturated. The test involves three stages: Unloaded,
Downlink Saturated, and Uplink Saturated. In the first stage we simply
measure latency to a file hosted on a CDN. This is usually around 5ms and
could vary a bit based on the user's location. We use the webTiming API to
find the time-to-first-byte, and consider that as the latency. In the
second stage we run a download, while simultaneously measuring latency. In
the third stage we do the same but for upload. Both download and upload
usually take around 5 seconds. We show the median, first quartile and the
third quartile on distribution charts corresponding to each stage to
provide a visual representation of the latency variations. For download and
upload we have used Cloudflare's speedtest backend.

You can find the test here: https://www.waveform.com/apps/dev-arshan

We built testing it on Chrome, but it works on Firefox and mobile too. On
mobile results may be a little different, as the APIs aren't available and
so instead we implemented a more manual method, which can be a little
noisier.

This is a really early alpha, and so we are keen to get any and all
feedback you have :-). Things that we would particularly like feedback on:

   - How does the bufferbloat measure compare to other tests you may have
   run on the same connection (e.g. dslreports, fast.com)
   - How the throughput results (download/upload/latency) look compared to
   other tools
   - Any feedback on the user interface of the test itself? We know that
   before releasing more widely we will put more effort into explaining
   bufferbloat than we have so far.
   - Anything else you would like to give feedback on?

We have added a feature to share results via a URL, so please feel free to
share these if you have specific feedback.

Thanks!
Sam and Arshan

*************************
Sam Westwood
Co-Founder & COO | RSRF & Waveform
E   sam@waveform.com
D   (949) 207-3175
T   1-800-761-3041 Ext. 100
W   www.rsrf.com & www.waveform.com

[-- Attachment #2: Type: text/html, Size: 3251 bytes --]

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]


Sam Westwood <sam@repeaterstore.com> wrote:
    > Downlink Saturated, and Uplink Saturated. In the first stage we simply
    > measure latency to a file hosted on a CDN. This is usually around 5ms
    > and could vary a bit based on the user's location. We use the webTiming
    > API to find the time-to-first-byte, and consider that as the
    > latency. In the second stage we run a download, while simultaneously

Just to clarify:
  the latency measurement involves a TCP three-way handshake, with
  the fourth packet providing the end of the process.
  No TLS, I hope?

    > We built testing it on Chrome, but it works on Firefox and mobile
    > too. On mobile results may be a little different, as the APIs aren't
    > available and so instead we implemented a more manual method, which can
    > be a little noisier.

Would webrtc APIs have helped?


--
Michael Richardson <mcr+IETF@sandelman.ca>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide





[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Toke Høiland-Jørgensen]

Sam Westwood <sam@repeaterstore.com> writes:

> Hi everyone,
>
> My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform
> we provide equipment to help improve cell phone service, and being in the
> industry we've always been interested in all aspects of network
> connectivity. Bufferbloat for us has always been interesting, and while
> there are a few tests out there we never found one that was fantastic. So
> we thought we'd try and build one!
>
> My colleague Arshan has built the test, which we based upon the Cloudflare
> Speedtest template that was discussed earlier in the summer in a previous
> thread.
>
> We measure bufferbloat under two conditions: when downlink is saturated and
> when uplink is saturated. The test involves three stages: Unloaded,
> Downlink Saturated, and Uplink Saturated. In the first stage we simply
> measure latency to a file hosted on a CDN. This is usually around 5ms and
> could vary a bit based on the user's location. We use the webTiming API to
> find the time-to-first-byte, and consider that as the latency. In the
> second stage we run a download, while simultaneously measuring latency. In
> the third stage we do the same but for upload. Both download and upload
> usually take around 5 seconds. We show the median, first quartile and the
> third quartile on distribution charts corresponding to each stage to
> provide a visual representation of the latency variations. For download and
> upload we have used Cloudflare's speedtest backend.

This sounds great, thanks for doing this! It certainly sounds like
you're on the right track here. Some comments below...

> You can find the test here: https://www.waveform.com/apps/dev-arshan
>
> We built testing it on Chrome, but it works on Firefox and mobile too. On
> mobile results may be a little different, as the APIs aren't available and
> so instead we implemented a more manual method, which can be a little
> noisier.
>
> This is a really early alpha, and so we are keen to get any and all
> feedback you have :-). Things that we would particularly like feedback on:
>
>    - How does the bufferbloat measure compare to other tests you may have
>    run on the same connection (e.g. dslreports, fast.com)
>    - How the throughput results (download/upload/latency) look compared to
>    other tools

I'm fortunate enough to have a full Gbps fibre link, which makes it
really hard to saturate the connection from a browser test (or at all,
sometimes). Your test does a decent job and comes pretty close, at least
in Chromium (about 800 Mbps which is not too far off at the application
layer, considering I have a constant 100Mbps flow in the background
taking up some of the bandwidth). Firefox seems way off (one test said
500Mbps the other >1000).

This does mean that I can't say much that's useful about your
bufferbloat scores, unfortunately. The latency measurement puzzled me a
bit (your tool says 16.6ms, but I get half that when I ping the
cloudfront.net CDN, which I think is what you're measuring against?),
but it does seem to stay fairly constant.

How do you calculate the jitter score? It's not obvious how you get from
the percentiles to the jitter.

Link to the test in Chromium:
https://www.waveform.com/apps/dev-arshan?test-id=91a55adc-7513-4b55-b8a6-0fa698ce634e

>    - Any feedback on the user interface of the test itself? We know that
>    before releasing more widely we will put more effort into explaining
>    bufferbloat than we have so far.

Brain dump of thoughts on the UI:

I found it hard to tell whether it was doing anything while the test was
running. Most other tests have some kind of very obvious feedback
(moving graphs of bandwidth-over-time for cloudflare/dslreports, a
honking big number going up and down for fast.com), which I was missing
here. I would also have liked to a measure of bandwidth over time, it
seems a bit suspicious (from a "verify that this is doing something
reasonable" PoV) that it just spits out a number at the end without
telling me how long it ran, or how it got to that number.

It wasn't obvious at first either that the header changes from
"bufferbloat test" to "your bufferbloat grade" once the test is over I
think the stages + result would be better put somewhere else where it's
more obvious (the rest of the page grows downwards, so why isn't the
result at the "end"?).

Also, what are the shields below the grade supposed to mean? Do they
change depending on the result? On which criteria? And it's telling me I
have an A+ grade, so why is there a link to fix my bufferbloat issues?

Smaller nit, I found the up/down arrows in "up saturated" and "down
saturated" a bit hard to grasp at first, I think spelling out
upload/download would be better. Also not sure I like the "saturated"
term in the first place; do people know what that means in a networking
context? And can you be sure the network is actually *being* saturated?

Why is the "before you start" text below the page? Shouldn't it be at
the top? And maybe explain *why* this is important?

>    - Anything else you would like to give feedback on?

As far as the web page itself is concerned, holy cross-domain script
deluge, Batman! I'm one of those people who run uMatrix in my regular
Firefox session, and I disallow all cross-site script loads by default.
I had to allow 15(!) different cross-site requests, *and* whitelist a
few domains in my ad blocker as well to even get the test to run. Please
fix this! I get that you can't completely avoid cross-domain requests
due to the nature of the test, but why is a speedtest pulling in scripts
from 'shopify.com' and three different ad tracking networks?

I realise some of this may just be because you put this thing on your
corporate web site, and thus a lot of it is not directly related to the
speed test itself. But all the same, having to wade through chest-high
piles of javascript just to get to the test doesn't exactly scream
"trustworthy company that just wants to help me fight bufferbloat" :)

(For comparison, dslreports, fast.com and speed.cloudflare.com all pull
in scripts from exactly three domains, all of them directly related to
the test, and none of them fail with my default uBlock blocklist).

-Toke

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Y]

https://www.waveform.com/apps/dev-arshan?test-id=a9f994a0-171c-48d3-a40d-1550f13b2319
Great job. This is the result of my slow internees. I would like to know the criteria for the grade.







Le jeudi 5 novembre 2020 à 08:54:57 UTC+9, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> a écrit : 





Sam Westwood <sam@repeaterstore.com> writes:

> Hi everyone,
>
> My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform
> we provide equipment to help improve cell phone service, and being in the
> industry we've always been interested in all aspects of network
> connectivity. Bufferbloat for us has always been interesting, and while
> there are a few tests out there we never found one that was fantastic. So
> we thought we'd try and build one!
>
> My colleague Arshan has built the test, which we based upon the Cloudflare
> Speedtest template that was discussed earlier in the summer in a previous
> thread.
>
> We measure bufferbloat under two conditions: when downlink is saturated and
> when uplink is saturated. The test involves three stages: Unloaded,
> Downlink Saturated, and Uplink Saturated. In the first stage we simply
> measure latency to a file hosted on a CDN. This is usually around 5ms and
> could vary a bit based on the user's location. We use the webTiming API to
> find the time-to-first-byte, and consider that as the latency. In the
> second stage we run a download, while simultaneously measuring latency. In
> the third stage we do the same but for upload. Both download and upload
> usually take around 5 seconds. We show the median, first quartile and the
> third quartile on distribution charts corresponding to each stage to
> provide a visual representation of the latency variations. For download and
> upload we have used Cloudflare's speedtest backend.

This sounds great, thanks for doing this! It certainly sounds like
you're on the right track here. Some comments below...

> You can find the test here: https://www.waveform.com/apps/dev-arshan
>
> We built testing it on Chrome, but it works on Firefox and mobile too. On
> mobile results may be a little different, as the APIs aren't available and
> so instead we implemented a more manual method, which can be a little
> noisier.
>
> This is a really early alpha, and so we are keen to get any and all
> feedback you have :-). Things that we would particularly like feedback on:
>
>    - How does the bufferbloat measure compare to other tests you may have
>    run on the same connection (e.g. dslreports, fast.com)
>    - How the throughput results (download/upload/latency) look compared to
>    other tools

I'm fortunate enough to have a full Gbps fibre link, which makes it
really hard to saturate the connection from a browser test (or at all,
sometimes). Your test does a decent job and comes pretty close, at least
in Chromium (about 800 Mbps which is not too far off at the application
layer, considering I have a constant 100Mbps flow in the background
taking up some of the bandwidth). Firefox seems way off (one test said
500Mbps the other >1000).

This does mean that I can't say much that's useful about your
bufferbloat scores, unfortunately. The latency measurement puzzled me a
bit (your tool says 16.6ms, but I get half that when I ping the
cloudfront.net CDN, which I think is what you're measuring against?),
but it does seem to stay fairly constant.

How do you calculate the jitter score? It's not obvious how you get from
the percentiles to the jitter.

Link to the test in Chromium:
https://www.waveform.com/apps/dev-arshan?test-id=91a55adc-7513-4b55-b8a6-0fa698ce634e

>    - Any feedback on the user interface of the test itself? We know that
>    before releasing more widely we will put more effort into explaining
>    bufferbloat than we have so far.

Brain dump of thoughts on the UI:

I found it hard to tell whether it was doing anything while the test was
running. Most other tests have some kind of very obvious feedback
(moving graphs of bandwidth-over-time for cloudflare/dslreports, a
honking big number going up and down for fast.com), which I was missing
here. I would also have liked to a measure of bandwidth over time, it
seems a bit suspicious (from a "verify that this is doing something
reasonable" PoV) that it just spits out a number at the end without
telling me how long it ran, or how it got to that number.

It wasn't obvious at first either that the header changes from
"bufferbloat test" to "your bufferbloat grade" once the test is over I
think the stages + result would be better put somewhere else where it's
more obvious (the rest of the page grows downwards, so why isn't the
result at the "end"?).

Also, what are the shields below the grade supposed to mean? Do they
change depending on the result? On which criteria? And it's telling me I
have an A+ grade, so why is there a link to fix my bufferbloat issues?

Smaller nit, I found the up/down arrows in "up saturated" and "down
saturated" a bit hard to grasp at first, I think spelling out
upload/download would be better. Also not sure I like the "saturated"
term in the first place; do people know what that means in a networking
context? And can you be sure the network is actually *being* saturated?

Why is the "before you start" text below the page? Shouldn't it be at
the top? And maybe explain *why* this is important?

>    - Anything else you would like to give feedback on?

As far as the web page itself is concerned, holy cross-domain script
deluge, Batman! I'm one of those people who run uMatrix in my regular
Firefox session, and I disallow all cross-site script loads by default.
I had to allow 15(!) different cross-site requests, *and* whitelist a
few domains in my ad blocker as well to even get the test to run. Please
fix this! I get that you can't completely avoid cross-domain requests
due to the nature of the test, but why is a speedtest pulling in scripts
from 'shopify.com' and three different ad tracking networks?

I realise some of this may just be because you put this thing on your
corporate web site, and thus a lot of it is not directly related to the
speed test itself. But all the same, having to wade through chest-high
piles of javascript just to get to the test doesn't exactly scream
"trustworthy company that just wants to help me fight bufferbloat" :)

(For comparison, dslreports, fast.com and speed.cloudflare.com all pull
in scripts from exactly three domains, all of them directly related to
the test, and none of them fail with my default uBlock blocklist).

-Toke

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 5361 bytes --]

Tried it, and I really like the header and use of candle-charts!

I got this:

[cid:part1.81AC21AC.758FE66F@indexexchange.com]

I'd like to be able to explain it to non-techie folks (my grandma, and also my IT team at work (;-)), so I wonder on their behalf...

  *   Why is unloaded a large number, and loaded a small one?
     *   milliseconds sound like delay, so 111.7 ms sounds slower than 0.0 ms
  *   Is bloat and latency something bad? The zeroes are in green, does that mean they're good?
  *   Is max "bad"? In that case I'd call it "worst" and min "best"
  *   Is median the middle or the average? (no kidding, I've been asked that! I'd call it average)
  *   Is 25% twenty-five percent of the packets? (I suspect it's a percentile)
  *   What does this mean in terms of how many Skype calls I can have happening at my house? I have two kids, a wife and a grandmother, all of whom Skype a lot.

Looking at the cool stuff in the banner, it looks like I can browse, do audio calls, video calls (just one, or many?) but not streaming (any or just 4k?) or gaming.  Emphasizing that would be instantly understandable by grandma and the kids.

--dave


On 2020-11-04 4:30 p.m., Sam Westwood wrote:
Hi everyone,

My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform we provide equipment to help improve cell phone service, and being in the industry we've always been interested in all aspects of network connectivity. Bufferbloat for us has always been interesting, and while there are a few tests out there we never found one that was fantastic. So we thought we'd try and build one!

My colleague Arshan has built the test, which we based upon the Cloudflare Speedtest template that was discussed earlier in the summer in a previous thread.

We measure bufferbloat under two conditions: when downlink is saturated and when uplink is saturated. The test involves three stages: Unloaded, Downlink Saturated, and Uplink Saturated. In the first stage we simply measure latency to a file hosted on a CDN. This is usually around 5ms and could vary a bit based on the user's location. We use the webTiming API to find the time-to-first-byte, and consider that as the latency. In the second stage we run a download, while simultaneously measuring latency. In the third stage we do the same but for upload. Both download and upload usually take around 5 seconds. We show the median, first quartile and the third quartile on distribution charts corresponding to each stage to provide a visual representation of the latency variations. For download and upload we have used Cloudflare's speedtest backend.

You can find the test here: https://www.waveform.com/apps/dev-arshan

We built testing it on Chrome, but it works on Firefox and mobile too. On mobile results may be a little different, as the APIs aren't available and so instead we implemented a more manual method, which can be a little noisier.

This is a really early alpha, and so we are keen to get any and all feedback you have :-). Things that we would particularly like feedback on:

  *   How does the bufferbloat measure compare to other tests you may have run on the same connection (e.g. dslreports, fast.com<http://fast.com>)
  *   How the throughput results (download/upload/latency) look compared to other tools
  *   Any feedback on the user interface of the test itself? We know that before releasing more widely we will put more effort into explaining bufferbloat than we have so far.
  *   Anything else you would like to give feedback on?

We have added a feature to share results via a URL, so please feel free to share these if you have specific feedback.

Thanks!
Sam and Arshan

*************************
Sam Westwood
Co-Founder & COO | RSRF & Waveform
E   sam@waveform.com<mailto:sam@waveform.com>
D   (949) 207-3175
T   1-800-761-3041 Ext. 100
W   www.rsrf.com<http://www.rsrf.com> & www.waveform.com<http://www.waveform.com>



_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat


--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2.1: Type: text/html, Size: 7022 bytes --]

[-- Attachment #2.2: mbbhaappkdeojcia.png --]
[-- Type: image/png, Size: 121135 bytes --]

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Toke Høiland-Jørgensen]

Dave Collier-Brown <dave.collier-brown@indexexchange.com> writes:

> Tried it, and I really like the header and use of candle-charts!
>
> I got this:
>
> [cid:part1.81AC21AC.758FE66F@indexexchange.com]
>
> I'd like to be able to explain it to non-techie folks (my grandma, and also my IT team at work (;-)), so I wonder on their behalf...
>
>   *   Why is unloaded a large number, and loaded a small one?
>      *   milliseconds sound like delay, so 111.7 ms sounds slower than 0.0 ms
>   *   Is bloat and latency something bad? The zeroes are in green, does that mean they're good?
>   *   Is max "bad"? In that case I'd call it "worst" and min "best"
>   *   Is median the middle or the average? (no kidding, I've been asked that! I'd call it average)
>   *   Is 25% twenty-five percent of the packets? (I suspect it's a percentile)
>   *   What does this mean in terms of how many Skype calls I can have happening at my house? I have two kids, a wife and a grandmother, all of whom Skype a lot.
>
> Looking at the cool stuff in the banner, it looks like I can browse,
> do audio calls, video calls (just one, or many?) but not streaming
> (any or just 4k?) or gaming.  Emphasizing that would be instantly
> understandable by grandma and the kids.

Also, holy cow, what's going on with your connection? The unloaded
latency says 17/110/200 min/median/max RTT. Is that due to bad
measurements, or do you have a lot of cross traffic and a really bloated
link? :/

-Toke

[Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 2034 bytes --]


On 2020-11-05 6:48 a.m., Toke Høiland-Jørgensen via Bloat wrote:

Also, holy cow, what's going on with your connection? The unloaded
latency says 17/110/200 min/median/max RTT. Is that due to bad
measurements, or do you have a lot of cross traffic and a really bloated
link? :/

-Toke


The tests differ somewhat while looking at an unloaded residential link provided by a local monopoly, Rogers Cable, and mitigated by an IQrouter (my old linksys is long dead (;-))

DSLReports says

  *   144.7 Mb/s down
  *   14.05 MB/s up
  *   bufferbloat A+
  *   downloading lag 40-100 ms

Waveform says:

  *   43.47 Mbps down
  *   16.05 Mbps up
  *   bufferbloat grade A+
  *   unloaded latency 93.5 ms

So we're reporting different speeds and RTTs. Are we using different units or definitions, I wonder?

--dave

--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 2614 bytes --]

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Toke Høiland-Jørgensen]

Dave Collier-Brown <dave.collier-brown@indexexchange.com> writes:

> On 2020-11-05 6:48 a.m., Toke Høiland-Jørgensen via Bloat wrote:
>
> Also, holy cow, what's going on with your connection? The unloaded
> latency says 17/110/200 min/median/max RTT. Is that due to bad
> measurements, or do you have a lot of cross traffic and a really bloated
> link? :/
>
> -Toke
>
>
> The tests differ somewhat while looking at an unloaded residential link provided by a local monopoly, Rogers Cable, and mitigated by an IQrouter (my old linksys is long dead (;-))
>
> DSLReports says
>
>   *   144.7 Mb/s down
>   *   14.05 MB/s up
>   *   bufferbloat A+
>   *   downloading lag 40-100 ms

Still a pretty big span from 40-100ms; how does that turn into an A+
score, I wonder?

> Waveform says:
>
>   *   43.47 Mbps down
>   *   16.05 Mbps up
>   *   bufferbloat grade A+
>   *   unloaded latency 93.5 ms
>
> So we're reporting different speeds and RTTs. Are we using different
> units or definitions, I wonder?

Well either that, or one of the tests is just busted. My immediate guess
would be the not-yet-released prototype is the least accurate ;)
I do wonder why, though...

-Toke

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Arshan Khanifar]

[-- Attachment #1: Type: text/plain, Size: 14027 bytes --]

Hello everyone! 

My name is Arshan and I’m one of the developers of the Bufferbloat project! Firstly thank you so much for your feedbacks! So many good points were raised that we will take into consideration for our later versions. I will attempt to answer your questions to the best of my knowledge so far (I am an intern, nonetheless :) ). 

Caution: Very long email ahead. I apologize in advance for this, I tried to address your individual questions, and I was offline for a while so I’m replying to lots of points here.


@Michael Richardson
> the latency measurement involves a TCP three-way handshake, with
>   the fourth packet providing the end of the process.
>   No TLS, I hope?

@Toke Høiland-Jørgensen
> The latency measurement puzzled me a
> bit (your tool says 16.6ms, but I get half that when I ping the
> cloudfront.net CDN, which I think is what you're measuring against?),
> but it does seem to stay fairly constant.


I believe TLS handshake time is not included here. I’m using the Resource Timing API <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API> to measure the time-to-first-byte for a request that I’m sending to retrieve a static file. The resource loading phases <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API> section of the documentation explicitly shows the different stages for DNS Lookup, TCP connection establishment, etc. I’m using the difference between requestStart and responseStart values. This value is deemed to be the same as time-to-first-byte <https://stackoverflow.com/questions/6533456/time-to-first-byte-with-javascript> seen in the inspector’s network tab.




We’re using this static file <https://fonts.gstatic.com/l/font?kit=KFOmCnqEu92Fr1Me4GZNCzcPK4I&skey=a0a0114a1dcab3ac&v=v20> that is hosted on a google CDN. We tried multiple different files, and this one had the lowest latency in both locations that we tested it (I’m in Toronto, and my colleague Sina is in San Francisco).


@Michael Richardson
> Would webrtc APIs have helped?

We took a look at WebRTC and it would be a really good option as it uses udp so we can even measure things like packetloss (packetlosstest.com <http://packetlosstest.com/> does this). However this requires that we host a WebRTC backend, and we’d have to have multiple deployments globally distributed so that the latency values are consistent elsewhere. Between that and fetching a static file backed by google’s CDN, we chose the latter for simplicity. 

@Toke Høiland-Jørgensen
> Your test does a decent job and comes pretty close, at least
> in Chromium (about 800 Mbps which is not too far off at the application
> layer, considering I have a constant 100Mbps flow in the background
> taking up some of the bandwidth). Firefox seems way off (one test said
> 500Mbps the other >1000).


The way I’m measuring download is that I make multiple simultaneous requests to cloudflare’s backend requesting 100MB files. Their backend simply returns a file that has “0”s in the body repeated until 100MB of file is generated. Then I use readable streams <https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams> to make multiple measurements of (total bytes downloaded, timestamp). Then I fit a line to the measurements collected, and the slope of that line is the calculated bandwidth. For gigabit connections, this download happens very quickly, and it may be the case that not a lot of points are collected, in which case the fitted line is not accurate and one might get overly-huge bandwidths as is the >1000 case in ur Firefox browser. I think this might be fixed if we increase the download time. Currently it’s 5s, maybe changing that to 10-20s would help. I think in general it’d be a good feature to have a "more advanced options” feature that allows the user to adjust some parameters of the connection (such as number of parallel connections, download scenario’s duration, upload scenario’s duration, etc.)

The reason I do this line-fitting is because I want to get rid of the bandwidth ramp-up time when the download begins. 

Real-time Bandwidth Reporting
Using readable-streams also allows for instantaneous bandwidth reporting (maybe using average of a moving window) similar to what fast.com <http://fast.com/> or speedtest.net <http://speedtest.net/> do, but I unfortunately am not able to do the same thing with upload, since getting progress on http uploads adds some pre-flight OPTIONS requests which cloudflare’s speedtest backend <https://speed.cloudflare.com/> doesn’t allow those requests. For this test we are directly hitting cloudflare’s backend, you can see this in the network tab: 

Our download is by sending an http GET request to this endpoint: https://speed.cloudflare.com/__down?bytes=100000000 <https://speed.cloudflare.com/__down?bytes=100000000>
and our upload is done by sending and http POST request to this endpoint: https://speed.cloudflare.com/__up <https://speed.cloudflare.com/__up>

Since we are using cloudflare’s backend we are limited by what they allow us to do. 

I did try making my own worker which essentially does the same thing as cloudflare’s speedtest backend (They do have this template worker <https://github.com/cloudflare/worker-speedtest-template> that for the most part does the same thing.) I modified that worker a bit so that it allows http progress on upload for real-time measurements, but we hit another wall with that: we could not saturate gigabit internet connections. Turns out that cloudflare has tiers for the workers and the bundle tier that we are using doesn’t get the most priority in terms of bandwidth, so we could only get up to 600mbps measurements. Their own speed test is hosted on an enterprise tier, which is around $6-7k USD and is way too expensive. We are however, requesting for a demo from them, and it’s an ongoing progress. 

So since we can  measure instantaneous download speeds  but not upload speeds, we don’t report it for either one. But I can still make the adjustment to report it for download at least. 

@Toke Høiland-Jørgensen
> How do you calculate the jitter score? It's not obvious how you get from
> the percentiles to the jitter.


Jitter here is the standard deviation of the latency measurements in each stage. Is this a good definition?

@Toke Høiland-Jørgensen
> I found it hard to tell whether it was doing anything while the test was
> running. Most other tests have some kind of very obvious feedback
> (moving graphs of bandwidth-over-time for cloudflare/dslreports, a
> honking big number going up and down for fast.com), which I was missing
> here. I would also have liked to a measure of bandwidth over time, it
> seems a bit suspicious (from a "verify that this is doing something
> reasonable" PoV) that it just spits out a number at the end without
> telling me how long it ran, or how it got to that number.

Yeah I think we need to either report real-time bandwidths, or put some sort of animation.

@Toke Høiland-Jørgensen
> It wasn't obvious at first either that the header changes from
> "bufferbloat test" to "your bufferbloat grade" once the test is over I
> think the stages + result would be better put somewhere else where it's
> more obvious (the rest of the page grows downwards, so why isn't the
> result at the "end"?)


Good point!

@Y intruder_tkyf at yahoo.fr <http://yahoo.fr/>
> Great job. This is the result of my slow internees. I would like to know the criteria for the grade.

@Toke Høiland-Jørgensen
> Also, what are the shields below the grade supposed to mean? Do they
> change depending on the result? On which criteria? 


They do change! The criteria are listed below. Note that in the criteria below:
Latency is calculated as the maximum median of latency across all three stages.
Latency with Jitter is calculated as the maximum  of (median + std) across all three stages.

Web Browsing:
Downlink: > 1mbps
Uplink: > 1mbps

Audio Calls:
Downlink: > 3mbps
Uplink: > 3mbps
Latency: < 150ms
Latency with Jitter: < 200ms

4K Video Streaming:
Downlink: > 40mbps

Video Conferencing:
Downlink: > 2.5mbps
Uplink: > 2.5mbps
Latency: < 150ms
Latency with Jitter: < 200ms

Online Gaming:
Downlink: > 3mbps
Uplink: > 0.5mbps
Latency: < 100ms
Latency with Jitter: < 150ms

For the bufferbloat grade we use the same criteria as DSL reports <http://www.dslreports.com/faq/17930>.

@Toke Høiland-Jørgensen
> And it's telling me I have an A+ grade, so why is there a link to fix my bufferbloat issues?

We should hide that for A+ grades. 😬

> Less than 5ms (average of down bloat and up bloat) - A+
> Less than 30ms - A
> Less than 60ms - B
> Less than 200ms - C
> Less than 400ms - D
> 400ms+ - F

@Toke Høiland-Jørgensen 
> Smaller nit, I found the up/down arrows in "up saturated" and "down
> saturated" a bit hard to grasp at first, I think spelling out
> upload/download would be better. Also not sure I like the "saturated"
> term in the first place; do people know what that means in a networking
> context? And can you be sure the network is actually *being* saturated?
> Why is the "before you start" text below the page? Shouldn't it be at
> the top? And maybe explain *why* this is important?


All amazing points! Thanks! 

@Toke Høiland-Jørgensen 
> As far as the web page itself is concerned, holy cross-domain script
> deluge, Batman! I'm one of those people who run uMatrix in my regular
> Firefox session, and I disallow all cross-site script loads by default.
> I had to allow 15(!) different cross-site requests, *and* whitelist a
> few domains in my ad blocker as well to even get the test to run. Please
> fix this! I get that you can't completely avoid cross-domain requests
> due to the nature of the test, but why is a speedtest pulling in scripts
> from 'shopify.com' and three different ad tracking networks?


Hahahah this is because we’re using Shopify App Proxies <https://shopify.dev/tutorials/display-data-on-an-online-store-with-an-application-proxy-app-extension>. It’s a technique we use to import assets from our main store, and make it appear such that this is part of our main store whereas in reality it’s a separately-hosted application. This allows us to bring in the header, footer and the chatbot. This is a really good point though, I wonder what we can do with that. 

@Dave Collier-Brown
>   *   Why is unloaded a large number, and loaded a small one?
>   *   milliseconds sound like delay, so 111.7 ms sounds slower than 0.0 ms

This is a good point! We measure bufferbloat as the “change” in latency, so the value reported as loaded is “relative” to the unloaded, and not an “absolute” value. In case of a good router with small bufferbloat, this value will always be smaller than the unloaded case. We should probably include a text explaining that. 

@Dave Collier-Brown
>   *   Is bloat and latency something bad? The zeroes are in green, does that mean they're good?
>   *   Is max "bad"? In that case I'd call it "worst" and min "best"
>   *   Is median the middle or the average? (no kidding, I've been asked that! I'd call it average)

It’s actually middle, and we meant to report middle since a huge latency spike tends to move the average pretty dramatically in some bad networks. 

@Dave Collier-Brown
>   *   Is 25% twenty-five percent of the packets? (I suspect it's a percentile)
>   *   What does this mean in terms of how many Skype calls I can have happening at my house? I have two kids, a wife and a grandmother, all of whom Skype a lot.
@Dave Collier-Brown
> Looking at the cool stuff in the banner, it looks like I can browse, do audio calls, video calls (just one, or many?) but not streaming (any or just 4k?) or gaming.  Emphasizing that would be instantly understandable by grandma and the kids.


All very good questions. I think we should try and answer them in the FAQ’s below the test.

@Dave Collier-Brown
> DSLReports says
> 
>   *   144.7 Mb/s down
>   *   14.05 MB/s up
>   *   bufferbloat A+
>   *   downloading lag 40-100 ms
> 
> Waveform says:
> 
>   *   43.47 Mbps down
>   *   16.05 Mbps up
>   *   bufferbloat grade A+
>   *   unloaded latency 93.5 ms
> 
> So we're reporting different speeds and RTTs. Are we using different units or definitions, I wonder?

This is most likely an issue with our test. Do you consistently get low downlink values with our test? maybe increasing the download stage duration will help with this.

@Sebastian Moeller
>         [SM] This is a decent starting point. In addition it might be helpful to at least optionally include a test with with bidirectional saturating load, in the past such tests typically were quite successful in detecting bufferbloat sources, that were less obvious in the uni-directional load tests. I am not sure however how well that can work with a browser based test?
>  
> [SM] Mmmh, I like that this is a relevant latency measure, it might make sense though to make sure users realize that this is not the eqivalent number to runing a ICMP eche request against the same endpoint?
> 
> [SM] On heavily bufferbloated links it often takes a considerable amount of time for the bottleneck buffers to drain after a uni-directional test, so it might make sense to separate the two direction test with an additional phase of idle latency measurements. If that latency is like the initial unloaded latency, all is well, but if latency slowly ramps down in that phase you have a smoking gun for bad bufferbloat.

All very good points! I think having a fixed idle-time between stages 2 and 3 would be good. It appears to me that the ICMP ping values are still very close to the measured latency, however. 

Thank you everyone for your feedback! 

Arshan

[-- Attachment #2.1: Type: text/html, Size: 56662 bytes --]

[-- Attachment #2.2: PastedGraphic-1.png --]
[-- Type: image/png, Size: 71429 bytes --]

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Toke Høiland-Jørgensen]

> I believe TLS handshake time is not included here. I’m using the
> Resource Timing API
> <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API>
> to measure the time-to-first-byte for a request that I’m sending to
> retrieve a static file. The resource loading phases
> <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API>
> section of the documentation explicitly shows the different stages for
> DNS Lookup, TCP connection establishment, etc. I’m using the
> difference between requestStart and responseStart values. This value
> is deemed to be the same as time-to-first-byte
> <https://stackoverflow.com/questions/6533456/time-to-first-byte-with-javascript>
> seen in the inspector’s network tab.

This does not seem completely ludicrous, at least :)

> We’re using this static file
> <https://fonts.gstatic.com/l/font?kit=KFOmCnqEu92Fr1Me4GZNCzcPK4I&skey=a0a0114a1dcab3ac&v=v20>
> that is hosted on a google CDN. We tried multiple different files, and
> this one had the lowest latency in both locations that we tested it
> (I’m in Toronto, and my colleague Sina is in San Francisco).

Ah, so that's why that request showed up :)

Curious to know why you picked this instead of, say, something from
speed.cloudflare.com (since you're using that for the speed tests anyway)?

> @Toke Høiland-Jørgensen
>> Your test does a decent job and comes pretty close, at least
>> in Chromium (about 800 Mbps which is not too far off at the application
>> layer, considering I have a constant 100Mbps flow in the background
>> taking up some of the bandwidth). Firefox seems way off (one test said
>> 500Mbps the other >1000).
>
>
> The way I’m measuring download is that I make multiple simultaneous
> requests to cloudflare’s backend requesting 100MB files. Their backend
> simply returns a file that has “0”s in the body repeated until 100MB
> of file is generated. Then I use readable streams
> <https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams>
> to make multiple measurements of (total bytes downloaded, timestamp).
> Then I fit a line to the measurements collected, and the slope of that
> line is the calculated bandwidth. For gigabit connections, this
> download happens very quickly, and it may be the case that not a lot
> of points are collected, in which case the fitted line is not accurate
> and one might get overly-huge bandwidths as is the >1000 case in ur
> Firefox browser. I think this might be fixed if we increase the
> download time. Currently it’s 5s, maybe changing that to 10-20s would
> help. I think in general it’d be a good feature to have a "more
> advanced options” feature that allows the user to adjust some
> parameters of the connection (such as number of parallel connections,
> download scenario’s duration, upload scenario’s duration, etc.)

Yeah, I think running the test for longer will help; 5s is not nearly
enough to saturate a connection, especially not as the link speed increases.

> The reason I do this line-fitting is because I want to get rid of the
> bandwidth ramp-up time when the download begins.

Yeah, allowing some ramp-up time before determining the bandwidth seems
reasonable, but it's not generally possible to just pick a static number
of (say) seconds to chop off... Having the graph over time helps
sanity-check things, though.

Also, a continuous graph of latency samples over time (for the whole
duration, including idle/upload/download) is usually very instructive
when plotting such a test.

> Real-time Bandwidth Reporting
> Using readable-streams also allows for instantaneous bandwidth
> reporting (maybe using average of a moving window) similar to what
> fast.com <http://fast.com/> or speedtest.net <http://speedtest.net/>
> do, but I unfortunately am not able to do the same thing with upload,
> since getting progress on http uploads adds some pre-flight OPTIONS
> requests which cloudflare’s speedtest backend
> <https://speed.cloudflare.com/> doesn’t allow those requests. For this
> test we are directly hitting cloudflare’s backend, you can see this in
> the network tab:
>
> Our download is by sending an http GET request to this endpoint:
> https://speed.cloudflare.com/__down?bytes=100000000
> <https://speed.cloudflare.com/__down?bytes=100000000> and our upload
> is done by sending and http POST request to this endpoint:
> https://speed.cloudflare.com/__up <https://speed.cloudflare.com/__up>
>
> Since we are using cloudflare’s backend we are limited by what they
> allow us to do.

The test at speed.cloudflare.com does seem to plot real-time upload
bandwidth; is that a privileged operation for themselves, or something?

> I did try making my own worker which essentially does the same thing
> as cloudflare’s speedtest backend (They do have this template worker
> <https://github.com/cloudflare/worker-speedtest-template> that for the
> most part does the same thing.) I modified that worker a bit so that
> it allows http progress on upload for real-time measurements, but we
> hit another wall with that: we could not saturate gigabit internet
> connections. Turns out that cloudflare has tiers for the workers and
> the bundle tier that we are using doesn’t get the most priority in
> terms of bandwidth, so we could only get up to 600mbps measurements.
> Their own speed test is hosted on an enterprise tier, which is around
> $6-7k USD and is way too expensive. We are however, requesting for a
> demo from them, and it’s an ongoing progress.
>
> So since we can measure instantaneous download speeds but not upload
> speeds, we don’t report it for either one. But I can still make the
> adjustment to report it for download at least.

Sure, better than nothing.

> @Toke Høiland-Jørgensen
>> How do you calculate the jitter score? It's not obvious how you get from
>> the percentiles to the jitter.
>
> Jitter here is the standard deviation of the latency measurements in
> each stage. Is this a good definition?

If you're reporting standard deviation, I'd just label it as such. One
good definition for jitter is IPDV:
https://en.wikipedia.org/wiki/Packet_delay_variation

> @Toke Høiland-Jørgensen
>> Also, what are the shields below the grade supposed to mean? Do they
>> change depending on the result? On which criteria? 
>
>
> They do change! The criteria are listed below. Note that in the criteria below:
> Latency is calculated as the maximum median of latency across all three stages.
> Latency with Jitter is calculated as the maximum  of (median + std) across all three stages.
>
> Web Browsing:
> Downlink: > 1mbps
> Uplink: > 1mbps
>
> Audio Calls:
> Downlink: > 3mbps
> Uplink: > 3mbps
> Latency: < 150ms
> Latency with Jitter: < 200ms
>
> 4K Video Streaming:
> Downlink: > 40mbps
>
> Video Conferencing:
> Downlink: > 2.5mbps
> Uplink: > 2.5mbps
> Latency: < 150ms
> Latency with Jitter: < 200ms
>
> Online Gaming:
> Downlink: > 3mbps
> Uplink: > 0.5mbps
> Latency: < 100ms
> Latency with Jitter: < 150ms
>
> For the bufferbloat grade we use the same criteria as DSL reports
> <http://www.dslreports.com/faq/17930>.

Right, cool; explaining this on the page might be useful ;)

> @Toke Høiland-Jørgensen 
>> As far as the web page itself is concerned, holy cross-domain script
>> deluge, Batman! I'm one of those people who run uMatrix in my regular
>> Firefox session, and I disallow all cross-site script loads by default.
>> I had to allow 15(!) different cross-site requests, *and* whitelist a
>> few domains in my ad blocker as well to even get the test to run. Please
>> fix this! I get that you can't completely avoid cross-domain requests
>> due to the nature of the test, but why is a speedtest pulling in scripts
>> from 'shopify.com' and three different ad tracking networks?
>
>
> Hahahah this is because we’re using Shopify App Proxies
> <https://shopify.dev/tutorials/display-data-on-an-online-store-with-an-application-proxy-app-extension>.
> It’s a technique we use to import assets from our main store, and make
> it appear such that this is part of our main store whereas in reality
> it’s a separately-hosted application. This allows us to bring in the
> header, footer and the chatbot. This is a really good point though, I
> wonder what we can do with that.

Just having the test be on a separate (static) page would be the obvious
fix ;)

-Toke

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Stephen Hemminger]

[-- Attachment #1: Type: text/plain, Size: 9662 bytes --]

On Thu, 05 Nov 2020 18:25:07 +0100
Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:

> > I believe TLS handshake time is not included here. I’m using the
> > Resource Timing API
> > <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API>
> > to measure the time-to-first-byte for a request that I’m sending to
> > retrieve a static file. The resource loading phases
> > <https://developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/Using_the_Resource_Timing_API>
> > section of the documentation explicitly shows the different stages for
> > DNS Lookup, TCP connection establishment, etc. I’m using the
> > difference between requestStart and responseStart values. This value
> > is deemed to be the same as time-to-first-byte
> > <https://stackoverflow.com/questions/6533456/time-to-first-byte-with-javascript>
> > seen in the inspector’s network tab.  
> 
> This does not seem completely ludicrous, at least :)
> 
> > We’re using this static file
> > <https://fonts.gstatic.com/l/font?kit=KFOmCnqEu92Fr1Me4GZNCzcPK4I&skey=a0a0114a1dcab3ac&v=v20>
> > that is hosted on a google CDN. We tried multiple different files, and
> > this one had the lowest latency in both locations that we tested it
> > (I’m in Toronto, and my colleague Sina is in San Francisco).  
> 
> Ah, so that's why that request showed up :)
> 
> Curious to know why you picked this instead of, say, something from
> speed.cloudflare.com (since you're using that for the speed tests anyway)?
> 
> > @Toke Høiland-Jørgensen  
> >> Your test does a decent job and comes pretty close, at least
> >> in Chromium (about 800 Mbps which is not too far off at the application
> >> layer, considering I have a constant 100Mbps flow in the background
> >> taking up some of the bandwidth). Firefox seems way off (one test said
> >> 500Mbps the other >1000).  
> >
> >
> > The way I’m measuring download is that I make multiple simultaneous
> > requests to cloudflare’s backend requesting 100MB files. Their backend
> > simply returns a file that has “0”s in the body repeated until 100MB
> > of file is generated. Then I use readable streams
> > <https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams>
> > to make multiple measurements of (total bytes downloaded, timestamp).
> > Then I fit a line to the measurements collected, and the slope of that
> > line is the calculated bandwidth. For gigabit connections, this
> > download happens very quickly, and it may be the case that not a lot
> > of points are collected, in which case the fitted line is not accurate
> > and one might get overly-huge bandwidths as is the >1000 case in ur
> > Firefox browser. I think this might be fixed if we increase the
> > download time. Currently it’s 5s, maybe changing that to 10-20s would
> > help. I think in general it’d be a good feature to have a "more
> > advanced options” feature that allows the user to adjust some
> > parameters of the connection (such as number of parallel connections,
> > download scenario’s duration, upload scenario’s duration, etc.)  
> 
> Yeah, I think running the test for longer will help; 5s is not nearly
> enough to saturate a connection, especially not as the link speed increases.
> 
> > The reason I do this line-fitting is because I want to get rid of the
> > bandwidth ramp-up time when the download begins.  
> 
> Yeah, allowing some ramp-up time before determining the bandwidth seems
> reasonable, but it's not generally possible to just pick a static number
> of (say) seconds to chop off... Having the graph over time helps
> sanity-check things, though.
> 
> Also, a continuous graph of latency samples over time (for the whole
> duration, including idle/upload/download) is usually very instructive
> when plotting such a test.
> 
> > Real-time Bandwidth Reporting
> > Using readable-streams also allows for instantaneous bandwidth
> > reporting (maybe using average of a moving window) similar to what
> > fast.com <http://fast.com/> or speedtest.net <http://speedtest.net/>
> > do, but I unfortunately am not able to do the same thing with upload,
> > since getting progress on http uploads adds some pre-flight OPTIONS
> > requests which cloudflare’s speedtest backend
> > <https://speed.cloudflare.com/> doesn’t allow those requests. For this
> > test we are directly hitting cloudflare’s backend, you can see this in
> > the network tab:
> >
> > Our download is by sending an http GET request to this endpoint:
> > https://speed.cloudflare.com/__down?bytes=100000000
> > <https://speed.cloudflare.com/__down?bytes=100000000> and our upload
> > is done by sending and http POST request to this endpoint:
> > https://speed.cloudflare.com/__up <https://speed.cloudflare.com/__up>
> >
> > Since we are using cloudflare’s backend we are limited by what they
> > allow us to do.  
> 
> The test at speed.cloudflare.com does seem to plot real-time upload
> bandwidth; is that a privileged operation for themselves, or something?
> 
> > I did try making my own worker which essentially does the same thing
> > as cloudflare’s speedtest backend (They do have this template worker
> > <https://github.com/cloudflare/worker-speedtest-template> that for the
> > most part does the same thing.) I modified that worker a bit so that
> > it allows http progress on upload for real-time measurements, but we
> > hit another wall with that: we could not saturate gigabit internet
> > connections. Turns out that cloudflare has tiers for the workers and
> > the bundle tier that we are using doesn’t get the most priority in
> > terms of bandwidth, so we could only get up to 600mbps measurements.
> > Their own speed test is hosted on an enterprise tier, which is around
> > $6-7k USD and is way too expensive. We are however, requesting for a
> > demo from them, and it’s an ongoing progress.
> >
> > So since we can measure instantaneous download speeds but not upload
> > speeds, we don’t report it for either one. But I can still make the
> > adjustment to report it for download at least.  
> 
> Sure, better than nothing.
> 
> > @Toke Høiland-Jørgensen  
> >> How do you calculate the jitter score? It's not obvious how you get from
> >> the percentiles to the jitter.  
> >
> > Jitter here is the standard deviation of the latency measurements in
> > each stage. Is this a good definition?  
> 
> If you're reporting standard deviation, I'd just label it as such. One
> good definition for jitter is IPDV:
> https://en.wikipedia.org/wiki/Packet_delay_variation
> 
> > @Toke Høiland-Jørgensen  
> >> Also, what are the shields below the grade supposed to mean? Do they
> >> change depending on the result? On which criteria?   
> >
> >
> > They do change! The criteria are listed below. Note that in the criteria below:
> > Latency is calculated as the maximum median of latency across all three stages.
> > Latency with Jitter is calculated as the maximum  of (median + std) across all three stages.
> >
> > Web Browsing:
> > Downlink: > 1mbps
> > Uplink: > 1mbps
> >
> > Audio Calls:
> > Downlink: > 3mbps
> > Uplink: > 3mbps
> > Latency: < 150ms
> > Latency with Jitter: < 200ms
> >
> > 4K Video Streaming:
> > Downlink: > 40mbps
> >
> > Video Conferencing:
> > Downlink: > 2.5mbps
> > Uplink: > 2.5mbps
> > Latency: < 150ms
> > Latency with Jitter: < 200ms
> >
> > Online Gaming:
> > Downlink: > 3mbps
> > Uplink: > 0.5mbps
> > Latency: < 100ms
> > Latency with Jitter: < 150ms
> >
> > For the bufferbloat grade we use the same criteria as DSL reports
> > <http://www.dslreports.com/faq/17930>.  
> 
> Right, cool; explaining this on the page might be useful ;)
> 
> > @Toke Høiland-Jørgensen   
> >> As far as the web page itself is concerned, holy cross-domain script
> >> deluge, Batman! I'm one of those people who run uMatrix in my regular
> >> Firefox session, and I disallow all cross-site script loads by default.
> >> I had to allow 15(!) different cross-site requests, *and* whitelist a
> >> few domains in my ad blocker as well to even get the test to run. Please
> >> fix this! I get that you can't completely avoid cross-domain requests
> >> due to the nature of the test, but why is a speedtest pulling in scripts
> >> from 'shopify.com' and three different ad tracking networks?  
> >
> >
> > Hahahah this is because we’re using Shopify App Proxies
> > <https://shopify.dev/tutorials/display-data-on-an-online-store-with-an-application-proxy-app-extension>.
> > It’s a technique we use to import assets from our main store, and make
> > it appear such that this is part of our main store whereas in reality
> > it’s a separately-hosted application. This allows us to bring in the
> > header, footer and the chatbot. This is a really good point though, I
> > wonder what we can do with that.  
> 
> Just having the test be on a separate (static) page would be the obvious
> fix ;)
> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

The results for me look similar to other previous tests like the old dslreports.
Running on OpenWrt and using Wave cable which has reported 250M down and 10M up.

PS: Why to US providers have such asymmetric bandwidth? Getting something symmetric
requires going to a $$$ business rate.

https://www.waveform.com/apps/dev-arshan?test-id=a77df4a0-73cc-4375-8b0e-58e315aaffcf


[-- Attachment #2: Screenshot from 2020-11-06 08-01-30.png --]
[-- Type: image/png, Size: 103325 bytes --]

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Toke Høiland-Jørgensen]

Stephen Hemminger <stephen@networkplumber.org> writes:

> PS: Why to US providers have such asymmetric bandwidth? Getting something symmetric
> requires going to a $$$ business rate.

For Cable, the DOCSIS standard is asymmetric by design, but not *that*
asymmetric. I *think* the rest is because providers have to assign
channels independently for upstream and downstream, and if they just
assign them all to downstream they can advertise a bigger number...

-Toke

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Sebastian Moeller]

Hi Toke,

> Gesendet: Freitag, 06. November 2020 um 17:17 Uhr
> Von: "Toke Høiland-Jørgensen via Bloat" <bloat@lists.bufferbloat.net>
> An: "Stephen Hemminger" <stephen@networkplumber.org>, "Toke Høiland-Jørgensen via Bloat" <bloat@lists.bufferbloat.net>
> Betreff: Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)
>
> Stephen Hemminger <stephen@networkplumber.org> writes:
> 
> > PS: Why to US providers have such asymmetric bandwidth? Getting something symmetric
> > requires going to a $$$ business rate.
> 
> For Cable, the DOCSIS standard is asymmetric by design, but not *that*
> asymmetric. 

       Unfortunately is is that bad: DOCSIS 3.0 Downstream 108 MHz to 1002 MHz Upstream 30 MHz to 85 MHz, so (1002-108)/(85-30) 16:1, but not all cable co, have matching upstream filters for 85MHz. Then again, the two ACK per two full segment rule puts a lower end in, with what an ISP can get away, if the customer is expected to at least see the downstream rate in speedtests, I can never remember whether that is essentially 20:1 or 40:1, but since then GRO?GSO and friends, as well as ACK filtering has reduced the ACK traffic somewhat...


> I *think* the rest is because providers have to assign
> channels independently for upstream and downstream, and if they just
> assign them all to downstream they can advertise a bigger number...

       They wished, once they deploy upstream amplifiers these have a fixed frequency split and need to be replaced if the spilt is changed, that gets expensive quickly... or so I have heard.

Best Regards
        Sebastian

> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Arshan Khanifar]

Hello Toke!

> Curious to know why you picked this instead of, say, something from
> speed.cloudflare.com (since you're using that for the speed tests anyway)?

The reason we chose the static file instead of cloudflare was because the ping to Cloudflare from where I am reported a pretty high number (around 20ms), and we also noticed that there was an issue with the network path to cloudflare in general. We were getting some packet losses when doing a normal ping, and there were some latency spikes that would happen regularly. 

> Yeah, allowing some ramp-up time before determining the bandwidth seems
> reasonable, but it's not generally possible to just pick a static number
> of (say) seconds to chop off... Having the graph over time helps
> sanity-check things, though.

Yeah it’d be useful to have a real-time graph. 

> Also, a continuous graph of latency samples over time (for the whole
> duration, including idle/upload/download) is usually very instructive
> when plotting such a test.

I had that in the earlier version of the test that I sent to you previously. It would make the test look a bit complicated though, so we should maybe have an advanced view, where the user can see these charts.

> The test at speed.cloudflare.com does seem to plot real-time upload
> bandwidth; is that a privileged operation for themselves, or something?

It’s not “that” real time, they start from a small upload size, and progressively increase the file size. They will calculate the average bandwidth for each upload and show it on the chart that they have. I want to have the test continuously do a big upload, and have real-time measurements.

> If you're reporting standard deviation, I'd just label it as such. One
> good definition for jitter is IPDV:
> https://en.wikipedia.org/wiki/Packet_delay_variation

Thanks! This is great! 

> 
>> @Toke Høiland-Jørgensen
>>> Also, what are the shields below the grade supposed to mean? Do they
>>> change depending on the result? On which criteria? 
> 
> Right, cool; explaining this on the page might be useful ;)

Yeah we’ll change the texts and add more explanations in the end. 

> Just having the test be on a separate (static) page would be the obvious
> fix ;)

Correct hahah. I’ll bring that up with the team. 

Thank you so much for all the help!

Arshan

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Dave Taht]

I just wanted to note on this enormous thread that my primary beef
with most speedtests is the too short duration.

cable's "Powerboost" was optimized for speedtests' 20 sec duration.
The internet as a whole ended up optimized for a ~20sec download as a
result.... Run a test for 21 seconds, and boom.

flent defaults to 1minute tests by default and you can clearly see in
many results we've published, the long, slow, miserable, ramp.

A scientific way of compensating for the balance of user attention and
an accurate test would be to always run some
subset of the tests initiated by users for this longer duration and
cross check those results.

Re: [Bloat] Comparing bufferbloat tests (was: We built a new bufferbloat test and keen for feedback)

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]


Dave Taht <dave.taht@gmail.com> wrote:
    > I just wanted to note on this enormous thread that my primary beef
    > with most speedtests is the too short duration.

    > cable's "Powerboost" was optimized for speedtests' 20 sec duration.
    > The internet as a whole ended up optimized for a ~20sec download as a
    > result.... Run a test for 21 seconds, and boom.

What did they optimize for that duration?
"Tell me how your measure me, and I'll tell you have I'll behave"
(I was told Demming said this, but the quote sites disagree)

    > A scientific way of compensating for the balance of user attention and
    > an accurate test would be to always run some
    > subset of the tests initiated by users for this longer duration and
    > cross check those results.

Yes... and/or give them some feedback after 20s, and if they stay on the site,
keep running the test.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Sebastian Moeller]

Hi Sam,
 
first thanks, always good to see more dedicated tools to asses latency under load, especially tools that are easy to use and do not require the user to maintain her/his own dedicated endpoints!
More below in-line, prefixed [SM].
 
 

Gesendet: Mittwoch, 04. November 2020 um 22:30 Uhr
Von: "Sam Westwood" <sam@repeaterstore.com>
An: bloat@lists.bufferbloat.net
Betreff: [Bloat] We built a new bufferbloat test and keen for feedback

Hi everyone, 
My name is Sam and I'm the co-founder and COO of Waveform.com. At Waveform we provide equipment to help improve cell phone service, and being in the industry we've always been interested in all aspects of network connectivity. Bufferbloat for us has always been interesting, and while there are a few tests out there we never found one that was fantastic. So we thought we'd try and build one!
My colleague Arshan has built the test, which we based upon the Cloudflare Speedtest template that was discussed earlier in the summer in a previous thread.
 
We measure bufferbloat under two conditions: when downlink is saturated and when uplink is saturated.
 
        [SM] This is a decent starting point. In addition it might be helpful to at least optionally include a test with with bidirectional saturating load, in the past such tests typically were quite successful in detecting bufferbloat sources, that were less obvious in the uni-directional load tests. I am not sure however how well that can work with a browser based test?
 
 
The test involves three stages: Unloaded, Downlink Saturated, and Uplink Saturated. In the first stage we simply measure latency to a file hosted on a CDN. This is usually around 5ms and could vary a bit based on the user's location. We use the webTiming API to find the time-to-first-byte, and consider that as the latency. 

        [SM] Mmmh, I like that this is a relevant latency measure, it might make sense though to make sure users realize that this is not the eqivalent number to runing a ICMP eche request against the same endpoint?


In the second stage we run a download, while simultaneously measuring latency. In the third stage we do the same but for upload. Both download and upload usually take around 5 seconds. 

        [SM] On heavily bufferbloated links it often takes a considerable amount of time for the bottleneck buffers to drain after a uni-directional test, so it might make sense to separate the two direction test with an additional phase of idle latency measurements. If that latency is like the initial unloaded latency, all is well, but if latency slowly ramps down in that phase you have a smoking gun for bad bufferbloat.
         Also, there are link technologies and scheduler techniques that can prioritise relative short flows (e.g. Comcast's powerboost) to avoid just measuring the properties of these short duration special modes, it might make sense to optionally and considerably lengthen the duration of the test durations to say 30 seconds (empirically powerboost does not engage for a full 30second perid at full rate, but that might be arms race). Also to assess possible root causes for latency and rate issues, it is very helpful to show time resolved plots, that show the development of rate and latency over the duration of all phases of the test. For example, using longer running flent tests I could pinpoint the cyclic channel scanning of my laptop's wifi as a source of repeated bufferbloat with a period of ~10 seconds, by seeing evenly spaced latency spikes and rate dips every 10 seconds then went away when switching to wired ethernet...

We show the median, first quartile and the third quartile on distribution charts corresponding to each stage to provide a visual representation of the latency variations. For download and upload we have used Cloudflare's speedtest backend.

        [SM] This is great, it would be nice though to also add a graphical representation, be it a histogram or a cumulative density plot of latencies (split out for idle, download, upload and the idle period between down- and upload).


Best Regards
        Sebastian


 
You can find the test here: https://www.waveform.com/apps/dev-arshan
 
We built testing it on Chrome, but it works on Firefox and mobile too. On mobile results may be a little different, as the APIs aren't available and so instead we implemented a more manual method, which can be a little noisier.
 
This is a really early alpha, and so we are keen to get any and all feedback you have :-). Things that we would particularly like feedback on:
How does the bufferbloat measure compare to other tests you may have run on the same connection (e.g. dslreports, fast.com[http://fast.com])How the throughput results (download/upload/latency) look compared to other toolsAny feedback on the user interface of the test itself? We know that before releasing more widely we will put more effort into explaining bufferbloat than we have so far.  Anything else you would like to give feedback on?We have added a feature to share results via a URL, so please feel free to share these if you have specific feedback. 
Thanks!
Sam and Arshan
 
*************************
Sam Westwood
Co-Founder & COO | RSRF & Waveform
E   sam@waveform.com[mailto:sam@waveform.com]
D   (949) 207-3175  
T   1-800-761-3041 Ext. 100
W   www.rsrf.com[http://www.rsrf.com] & www.waveform.com[http://www.waveform.com]

 _______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat[https://lists.bufferbloat.net/listinfo/bloat]

Re: [Bloat] We built a new bufferbloat test and keen for feedback

[Sam]

[-- Attachment #1: Type: text/plain, Size: 3297 bytes --]

On 11/4/20 3:30 PM, Sam Westwood wrote:
> Hi everyone,
> 
> My name is Sam and I'm the co-founder and COO of Waveform.com. At 
> Waveform we provide equipment to help improve cell phone service, and 
> being in the industry we've always been interested in all aspects of 
> network connectivity. Bufferbloat for us has always been interesting, 
> and while there are a few tests out there we never found one that was 
> fantastic. So we thought we'd try and build one!
> 
> My colleague Arshan has built the test, which we based upon the 
> Cloudflare Speedtest template that was discussed earlier in the summer 
> in a previous thread.
> 
> We measure bufferbloat under two conditions: when downlink is saturated 
> and when uplink is saturated. The test involves three stages: Unloaded, 
> Downlink Saturated, and Uplink Saturated. In the first stage we simply 
> measure latency to a file hosted on a CDN. This is usually around 5ms 
> and could vary a bit based on the user's location. We use the webTiming 
> API to find the time-to-first-byte, and consider that as the latency. In 
> the second stage we run a download, while simultaneously measuring 
> latency. In the third stage we do the same but for upload. Both download 
> and upload usually take around 5 seconds. We show the median, first 
> quartile and the third quartile on distribution charts corresponding to 
> each stage to provide a visual representation of the latency variations. 
> For download and upload we have used Cloudflare's speedtest backend.
> 
> You can find the test here: https://www.waveform.com/apps/dev-arshan 
> <https://www.waveform.com/apps/dev-arshan>
> 
> We built testing it on Chrome, but it works on Firefox and mobile too. 
> On mobile results may be a little different, as the APIs aren't 
> available and so instead we implemented a more manual method, which can 
> be a little noisier.
> 
> This is a really early alpha, and so we are keen to get any and all 
> feedback you have :-). Things that we would particularly like feedback on:
> 
>   * How does the bufferbloat measure compare to other tests you may have
>     run on the same connection (e.g. dslreports, fast.com <http://fast.com>)
>   * How the throughput results (download/upload/latency) look compared
>     to other tools
>   * Any feedback on the user interface of the test itself? We know that
>     before releasing more widely we will put more effort into explaining
>     bufferbloat than we have so far.
>   * Anything else you would like to give feedback on?
> 
> We have added a feature to share results via a URL, so please feel free 
> to share these if you have specific feedback.
> 
> Thanks!
> Sam and Arshan
> 
> *************************
> Sam Westwood
> Co-Founder & COO | RSRF & Waveform
> E sam@waveform.com <mailto:sam@waveform.com>
> D   (949) 207-3175
> T   1-800-761-3041 Ext. 100
> W www.rsrf.com <http://www.rsrf.com> & www.waveform.com 
> <http://www.waveform.com>
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 


Looks pretty identical to what fast.com gave me. I'm on 50/50 fiber and 
firefox 82.
https://www.waveform.com/apps/dev-arshan?test-id=58dfa326-23d4-44a3-9059-b6011b104ccb

--Sam

[-- Attachment #2: Screenshot_20201105_152751.png --]
[-- Type: image/png, Size: 32907 bytes --]

[-- Attachment #3: Screenshot_20201105_152830.png --]
[-- Type: image/png, Size: 41976 bytes --]