[Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Livingood, Jason]

FWIW, we at Comcast just announced public beta of a new soon-to-be-open-source web-based speed test (see http://labs.comcast.com/beta-testing-a-new-open-source-speed-test). We plan to have a hackathon at Princeton in early November (see https://citp.princeton.edu/event/speedtest-hackathon/). One of the items in the backlog and suggested for one of the hackathon teams to work on is a buffer bloat test. 

If anyone here is interested in participating, ping me off-list (we have a limited number of spots available).

Thanks!
Jason


On 8/17/16, 7:33 AM, "Bloat on behalf of Rich Brown" <bloat-bounces@lists.bufferbloat.net on behalf of richb.hanover@gmail.com> wrote:

    I just saw Netflix's Fast.com site. They (may) suffer from non-neutral treatment by various carriers, so they set up their own testing suite so that their customers can see if Netflix servers are getting full treatment.
    
    Netflix make interesting choices in the design of the test. They only measure download speed (since that's their customer's major beef). It's a really attractive page, and it's pretty clear what they're doing.
    
    This blog posting tells how they designed it and some of the challenges. http://techblog.netflix.com/2016/08/building-fastcom.html
    
    Cheers!
    
    Rich
    _______________________________________________
    Bloat mailing list
    Bloat@lists.bufferbloat.net
    https://lists.bufferbloat.net/listinfo/bloat
    

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Stephen Hemminger]

On Tue, 23 Aug 2016 18:42:07 +0000
"Livingood, Jason" <Jason_Livingood@comcast.com> wrote:

> FWIW, we at Comcast just announced public beta of a new soon-to-be-open-source web-based speed test (see http://labs.comcast.com/beta-testing-a-new-open-source-speed-test). We plan to have a hackathon at Princeton in early November (see https://citp.princeton.edu/event/speedtest-hackathon/). One of the items in the backlog and suggested for one of the hackathon teams to work on is a buffer bloat test. 
> 
> If anyone here is interested in participating, ping me off-list (we have a limited number of spots available).
> 
> Thanks!
> Jason
> 

Why doesn't the test measure bufferbloat like FLENT or dslreports test?

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Livingood, Jason]

> Why doesn't the test measure bufferbloat like FLENT or dslreports test?
   
We have not gotten to it yet; it is in our backlog. We’re hoping folks might help prior to the hackathon or at the hackathon…  ☺

Jason 
    

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Kathleen Nichols]

I think it might be useful to say these tests measure the maximum
*potential* for
bufferbloat. That is, they plumb the depths of the buffers in the path.
I tried running
dslreports while I was running a video and though dslreports ramps
delays up to 700ms,
before and after that peak delay is more like 45ms. I don't think large
buffers are going
to go away, what matters is whether they are getting filled up.

So, is "bufferbloat" the existence of large buffers or the existence of
large queues? I think
the latter.

	Kathie

On 8/24/16 10:28 AM, Livingood, Jason wrote:
>> Why doesn't the test measure bufferbloat like FLENT or dslreports test?
>    
> We have not gotten to it yet; it is in our backlog. We’re hoping folks might help prior to the hackathon or at the hackathon…  ☺
> 
> Jason 
>     
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[David Lang]

On Fri, 26 Aug 2016, Kathleen Nichols wrote:

> I think it might be useful to say these tests measure the maximum
> *potential* for
> bufferbloat. That is, they plumb the depths of the buffers in the path.
> I tried running
> dslreports while I was running a video and though dslreports ramps
> delays up to 700ms,
> before and after that peak delay is more like 45ms. I don't think large
> buffers are going
> to go away, what matters is whether they are getting filled up.
>
> So, is "bufferbloat" the existence of large buffers or the existence of
> large queues? I think
> the latter.

large buffers that never fill up may as well be small buffers.

it's the fact that the large buffers fill that's the problem.

so you can call it large queues instead of large buffers, but the result is that 
packets end up being 'in transit' for a long time.

David Lang

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Kathleen Nichols]

in-line

On 8/26/16 4:20 PM, David Lang wrote:
> On Fri, 26 Aug 2016, Kathleen Nichols wrote:
> 
>> I think it might be useful to say these tests measure the maximum
>> *potential* for
>> bufferbloat. That is, they plumb the depths of the buffers in the path.
>> I tried running
>> dslreports while I was running a video and though dslreports ramps
>> delays up to 700ms,
>> before and after that peak delay is more like 45ms. I don't think large
>> buffers are going
>> to go away, what matters is whether they are getting filled up.
>>
>> So, is "bufferbloat" the existence of large buffers or the existence of
>> large queues? I think
>> the latter.
> 
> large buffers that never fill up may as well be small buffers.
> 
> it's the fact that the large buffers fill that's the problem.

Yes, that's the point.
> 
> so you can call it large queues instead of large buffers, but the result
> is that packets end up being 'in transit' for a long time.

No, a large queue is a bunch of packets waiting in a queue (which is
contained in
a buffer). A large buffer with zero or a small number of packets in it
is not going
to result in packets being in transit for a long time.
> 
> David Lang

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]

On Fri, Aug 26, 2016 at 6:13 PM, Kathleen Nichols <nichols@pollere.com>
wrote:

>
> I think it might be useful to say these tests measure the maximum
> *potential* for
> bufferbloat. That is, they plumb the depths of the buffers in the path.
> I tried running
> dslreports while I was running a video and though dslreports ramps
> delays up to 700ms,
> before and after that peak delay is more like 45ms. I don't think large
> buffers are going
>
I would like to know how my ISP is configured. I only know two things. They
are using Calix for the last mile and don't do any rate limiting in the
last mile. They use a Cisco core router where they do all of their rate
limiting. Prior to their Cisco core router, they use the rate limiting on
the ONT. It was strict and created the normal TCP saw-tooth pattern. After
the change over, it was no longer a saw-tooth and more like a high
frequency sine-wave

Here are two speedtests with no traffic shaping on my end,
https://www.dslreports.com/speedtest/4703254
https://www.dslreports.com/speedtest/4703265

This is what it looks like when I enabled Codel on my firewall
https://www.dslreports.com/speedtest/4698506

This is what it looks like when I'm seeding about 20Mb/s and my wife is
watching Netflix while I do a speedtest with Codel
https://www.dslreports.com/speedtest/4702550

Speaking of large buffers. I have been able to get my ISP's AQM to have
massive bursts of latency, but it required my to have my connection DDOSd.
Since the rate limiting is done in the core router for all customers, it
must have a huge buffer. If I have a service DDOS me, used to be 100Mb
connection, with 200Mb/s of UDP, going from perfectly idle to fully
saturated with no ramp-up, I was able to get brief pings into the 4k ms
range before the AQM just started dropping nearly all packets.

What was interesting is after the AQM kicked in, even though I was getting
something like 80% packetloss, my ping stayed around 30ms above idle.

Level 3 comm also seems to have some sort of AQM or they do with my ISP. At
one point my ISP was getting about 15% loss between them and their Level 3
upstream hop. I called up and I was told they were getting DDOSd. Even
during this attack, the ping never spiked more than 20ms to Level 3. Or
Level 3 uses really really small buffers. During this time the ping to my
ISP was its normal healthy stable low value, so I know all of the jitter
was the trunk.

There is hope. It's making its way into hardware, but I would like to know
what hardware and what implementation and why isn't this being more used?

to go away, what matters is whether they are getting filled up.
>
> So, is "bufferbloat" the existence of large buffers or the existence of
> large queues? I think
> the latter.
>
>         Kathie
>
> On 8/24/16 10:28 AM, Livingood, Jason wrote:
> >> Why doesn't the test measure bufferbloat like FLENT or dslreports test?
> >
> > We have not gotten to it yet; it is in our backlog. We’re hoping folks
> might help prior to the hackathon or at the hackathon…  ☺
> >
> > Jason
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 4845 bytes --]

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[David Lang]

On Fri, 26 Aug 2016, Kathleen Nichols wrote:

> On 8/26/16 4:20 PM, David Lang wrote:
>> On Fri, 26 Aug 2016, Kathleen Nichols wrote:
>>
>>> I think it might be useful to say these tests measure the maximum
>>> *potential* for
>>> bufferbloat. That is, they plumb the depths of the buffers in the path.
>>> I tried running
>>> dslreports while I was running a video and though dslreports ramps
>>> delays up to 700ms,
>>> before and after that peak delay is more like 45ms. I don't think large
>>> buffers are going
>>> to go away, what matters is whether they are getting filled up.
>>>
>>> So, is "bufferbloat" the existence of large buffers or the existence of
>>> large queues? I think
>>> the latter.
>>
>> large buffers that never fill up may as well be small buffers.
>>
>> it's the fact that the large buffers fill that's the problem.
>
> Yes, that's the point.
>>
>> so you can call it large queues instead of large buffers, but the result
>> is that packets end up being 'in transit' for a long time.
>
> No, a large queue is a bunch of packets waiting in a queue (which is contained 
> in a buffer). A large buffer with zero or a small number of packets in it is 
> not going to result in packets being in transit for a long time.

Is a large buffer that is never used really a large buffer? or does whatever 
prevents it from being used really turn it into a small buffer?

The problem has never been a matter of the number of bytes the buffers hold, but 
rather the number of bytes relative to the data rate (also known as the time 
that data can wait in the buffer). A buffer that's the right size for a 1Gb/s 
connection is horribly bloated if the data rate is only 10Mb/s

We've found that the solution isn't just to shrink the size of the buffers (even 
if we first change from X packets to X bytes), and instead the proper solution 
is some form of active queue management that has the side effect of keeping 
the queues small.

I don't understand what you are trying to call out by trying to change the 
terminology.

David Lang

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[grenville armitage]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]



On 08/27/2016 11:06, David Lang wrote:
> On Fri, 26 Aug 2016, Kathleen Nichols wrote:
>
     [..]
>>> so you can call it large queues instead of large buffers, but the result
>>> is that packets end up being 'in transit' for a long time.
>>
>> No, a large queue is a bunch of packets waiting in a queue (which is contained in a buffer). A large buffer with zero or a small number of packets in it is not going to result in packets being in transit for a long time.
     [..]
>
> I don't understand what you are trying to call out by trying to change the terminology.

I think you're almost in violent agreement, except that Kathy is differentiating between the space set aside for holding between 0 and N packets (or bytes) of data for delivery (a /buffer/) and an instance of packets queued up to a particular depth in a buffer (a /queue/) . Given that terminology, a bottleneck may implement a large buffer, but with proper congestion signals (or eg. delay-based congestion inference by end points) there might only ever be small queues build up in the (large) buffer.

cheers,
gja


[-- Attachment #2: Type: text/html, Size: 2082 bytes --]

Re: [Bloat] Open Source Speed Test (was fast.com - Netflix's speed monitoring)

[Jonathan Morton]

> On 27 Aug, 2016, at 04:06, David Lang <david@lang.hm> wrote:
> 
>>> so you can call it large queues instead of large buffers, but the result
>>> is that packets end up being 'in transit' for a long time.
>> 
>> No, a large queue is a bunch of packets waiting in a queue (which is contained in a buffer). A large buffer with zero or a small number of packets in it is not going to result in packets being in transit for a long time.
> 
> Is a large buffer that is never used really a large buffer? or does whatever prevents it from being used really turn it into a small buffer?

> I don't understand what you are trying to call out by trying to change the terminology.

If we’re talking terminology, I think we have to make better distinctions to avoid confusion.  There is a qualitative difference between a managed queue and both a large and small unmanaged queue; none of them behave similarly to each other.

A managed queue tries to keep itself empty, but does so by means of congestion signalling (marking or dropping a relatively small proportion of traffic), rather than placing a hard limit on queue length.  It *can* therefore fill up in various circumstances, including where the traffic simply ignores those signals, so its *peak* induced delay can be large; this is true of both flow-isolating and flow-blind queues.  However, the managed queue can achieve lower *average* induced delay than the large queue, and lower packet loss and higher link utilisation than the small queue, when given the buffer space of the large queue to work with.

Flow-isolation is an orthogonal property here; both managed and unmanaged implementations exist.  A flow-isolating queue aims to keep the induced delay to sparse flows, which use less than their fair share of the link, lower than to bulk flows; also to minimise the impact of unresponsive flows on responsive traffic.  The peak induced latency to any given bulk flow thus becomes less important than the peak induced latency to sparse flows.  With a flow-blind queue, all traffic suffers the same induced delay at any given moment, so the overall peak induced latency remains important.

Where I think the confusion arises here is between the *capacity* of the queue, which is static and often large for a managed queue, and the dynamic *backlog* of that queue, which is what a managed queue attempts to actively control.  In the case of a flow-isolating queue, there is also a distinction between the overall backlog of the queue, and the backlog of an individual subqueue.

I have noticed that some bufferbloat tests employ an unresponsive traffic burst as the latency measurement stream - particularly Netalyzr’s.  This does capture the difference between a large and small capacity queue, but is incapable of detecting AQM or flow-isolation, which are the preferred solutions to bufferbloat, unless the AQM is extremely aggressive when faced with unresponsive traffic.  A sufficiently aggressive response to satisfy such a test would however hurt goodput and packet loss on normal traffic, and would thus be counterproductive.

 - Jonathan Morton