[Bloat] I am unable to pinpoint the source of bufferbloat

[Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hi everyone,

Can anyone give some tips on how to diagnose the sources of bufferbloat?
According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
7 laptop:

- For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
know). I also disabled interrupt moderation.
Result? Still 550ms.
- Then I connected my laptop directly to my cable modem, bypassing my
Mikrotik 450G router. Result? Still 550ms of bufferbloat.
- Then I put a 100 megabit switch between the cable modem an the laptop (as
both cable modem and Intel NIC are gigabit). Result? Still 550ms of upload
bufferbloat.

I'm out of ideas now. It seems I can't do anything at all to lower
bufferbloat. Or the Netalyzr test is broken?:-)

many thanks for your advice,
Jeroen

[-- Attachment #2: Type: text/html, Size: 1023 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

What's the uplink speed from your modem onwards? If that's less than
100Mbit then your bottleneck is still there after all you've done. That's
where your bloat will be too.

- Jonathan Morton
 On Feb 9, 2013 11:53 AM, "Forums1000" <forums1000@gmail.com> wrote:

> Hi everyone,
>
> Can anyone give some tips on how to diagnose the sources of bufferbloat?
> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
> 7 laptop:
>
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
> know). I also disabled interrupt moderation.
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my
> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> - Then I put a 100 megabit switch between the cable modem an the laptop
> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
> upload bufferbloat.
>
> I'm out of ideas now. It seems I can't do anything at all to lower
> bufferbloat. Or the Netalyzr test is broken?:-)
>
> many thanks for your advice,
> Jeroen
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 1866 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

It's in your cablemodem.

Put a rate limiter between your cablemodem and the universe and you'll be
able to see it, and control it)

(in my case that's cerowrt's simple_qos script, but most of that's in
openwrt now)

On Sat, Feb 9, 2013 at 1:52 AM, Forums1000 <forums1000@gmail.com> wrote:

> Hi everyone,
>
> Can anyone give some tips on how to diagnose the sources of bufferbloat?
> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
> 7 laptop:
>
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
> know). I also disabled interrupt moderation.
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my
> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> - Then I put a 100 megabit switch between the cable modem an the laptop
> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
> upload bufferbloat.
>
> I'm out of ideas now. It seems I can't do anything at all to lower
> bufferbloat. Or the Netalyzr test is broken?:-)
>
> many thanks for your advice,
> Jeroen
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: Type: text/html, Size: 2113 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

Hi Jonathan and Dave

My entire LAN-network is gigabit. My cable subscription is 60 megabit down
and 4 megabit up.
Now, both my routers' WAN-port and the cable modems' LAN port are also
gigabit. The router can route LAN to WAN and the other way around (with NAT
and connection tracking enabled) in excess of 100 megabit.

Now my cable modem is a Motorola Surfboard SV6120E and hers is a Motorola
Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms. Moreover,
at her place there are Powerplugs in the path limiting her download to 30
megabit instead of 60 megabit. Yet, the upload lag is much lower than mine.
There, it also did not matter where I ran Natalyzr, the result was always
220ms of bufferbload.

Could this still be only the modem?


On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com> wrote:

> Hi everyone,
>
> Can anyone give some tips on how to diagnose the sources of bufferbloat?
> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
> 7 laptop:
>
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
> know). I also disabled interrupt moderation.
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my
> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> - Then I put a 100 megabit switch between the cable modem an the laptop
> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
> upload bufferbloat.
>
> I'm out of ideas now. It seems I can't do anything at all to lower
> bufferbloat. Or the Netalyzr test is broken?:-)
>
> many thanks for your advice,
> Jeroen
>
>

[-- Attachment #2: Type: text/html, Size: 2170 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 3176 bytes --]

Latency caused by bufferbloat always appears at the bottleneck device.
Usually that is the modem, and you've given no alternative that it could
plausibly be. The modems you mention are slightly different model numbers,
but that can hide substantial differences in internal configuration.

For a typical drop-tail queue, the induced latency under load is the size
of the buffer divided by the speed of the link draining it. Assuming both
modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer, and
220ms is consistent with a 48KB buffer - neither of which would seem
excessively large to a modem builder who hasn't heard of bufferbloat.
However with a shared cable infrastructure, it is possible that the uplink
is constrained by other users on the same segment, which will skew this
calculation.

To cure it without modifying the modem, you need to move the bottleneck to
a point where you can control the buffer. You do this by introducing
traffic shaping at slightly below the advertised modem uplink speed on one
of your own machines and directing all upstream traffic through it.

- Jonathan Morton
 On Feb 9, 2013 7:27 PM, "Forums1000" <forums1000@gmail.com> wrote:

> Hi Jonathan and Dave
>
> My entire LAN-network is gigabit. My cable subscription is 60 megabit down
> and 4 megabit up.
> Now, both my routers' WAN-port and the cable modems' LAN port are also
> gigabit. The router can route LAN to WAN and the other way around (with NAT
> and connection tracking enabled) in excess of 100 megabit.
>
> Now my cable modem is a Motorola Surfboard SV6120E and hers is a Motorola
> Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms. Moreover,
> at her place there are Powerplugs in the path limiting her download to 30
> megabit instead of 60 megabit. Yet, the upload lag is much lower than mine.
> There, it also did not matter where I ran Natalyzr, the result was always
> 220ms of bufferbload.
>
> Could this still be only the modem?
>
>
> On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Can anyone give some tips on how to diagnose the sources of bufferbloat?
>> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
>> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
>> 7 laptop:
>>
>> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
>> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
>> know). I also disabled interrupt moderation.
>> Result? Still 550ms.
>> - Then I connected my laptop directly to my cable modem, bypassing my
>> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
>> - Then I put a 100 megabit switch between the cable modem an the laptop
>> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
>> upload bufferbloat.
>>
>> I'm out of ideas now. It seems I can't do anything at all to lower
>> bufferbloat. Or the Netalyzr test is broken?:-)
>>
>> many thanks for your advice,
>> Jeroen
>>
>>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 3940 bytes --]

[Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 4282 bytes --]

The newer modem supports the bonding of more download channels (8 vs. 4 for
the 6120) and is also used in conjunction with telephony (whereas I have a
pure Internet subscription without telephony). Anyway, all of that should
not matter as you also indicate: both can handle the respective up- and
download speeds with ease.

Yes, both she and I have the same subscription (minus the telephony): 60 Mb
down and 4 Mb up. We also only live 25 kilometres apart (small country:-))
The uplink speed is mostly consistent around 3.76 megabit. I've never
caught our cable company going lower than 3.50 megabit in the upload
direction and I've done a lot of speedtests. The only limit I have been
able to observe is in the download direction when their network is really
busy.

I guess I will have to venture into traffic shaping. (I'll probably loose a
lot of sleep setting this up lol). Is there any way to determine the buffer
size reliably?

On Sat, Feb 9, 2013 at 7:15 PM, Jonathan Morton <chromatix99@gmail.com>wrote:

> Latency caused by bufferbloat always appears at the bottleneck device.
> Usually that is the modem, and you've given no alternative that it could
> plausibly be. The modems you mention are slightly different model numbers,
> but that can hide substantial differences in internal configuration.
>
> For a typical drop-tail queue, the induced latency under load is the size
> of the buffer divided by the speed of the link draining it. Assuming both
> modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer, and
> 220ms is consistent with a 48KB buffer - neither of which would seem
> excessively large to a modem builder who hasn't heard of bufferbloat.
> However with a shared cable infrastructure, it is possible that the uplink
> is constrained by other users on the same segment, which will skew this
> calculation.
>
> To cure it without modifying the modem, you need to move the bottleneck to
> a point where you can control the buffer. You do this by introducing
> traffic shaping at slightly below the advertised modem uplink speed on one
> of your own machines and directing all upstream traffic through it.
>
> - Jonathan Morton
>  On Feb 9, 2013 7:27 PM, "Forums1000" <forums1000@gmail.com> wrote:
>
>> Hi Jonathan and Dave
>>
>> My entire LAN-network is gigabit. My cable subscription is 60 megabit
>> down and 4 megabit up.
>> Now, both my routers' WAN-port and the cable modems' LAN port are also
>> gigabit. The router can route LAN to WAN and the other way around (with NAT
>> and connection tracking enabled) in excess of 100 megabit.
>>
>> Now my cable modem is a Motorola Surfboard SV6120E and hers is a Motorola
>> Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms. Moreover,
>> at her place there are Powerplugs in the path limiting her download to 30
>> megabit instead of 60 megabit. Yet, the upload lag is much lower than mine.
>> There, it also did not matter where I ran Natalyzr, the result was always
>> 220ms of bufferbload.
>>
>> Could this still be only the modem?
>>
>>
>> On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Can anyone give some tips on how to diagnose the sources of bufferbloat?
>>> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
>>> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
>>> 7 laptop:
>>>
>>> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
>>> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
>>> know). I also disabled interrupt moderation.
>>> Result? Still 550ms.
>>> - Then I connected my laptop directly to my cable modem, bypassing my
>>> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
>>> - Then I put a 100 megabit switch between the cable modem an the laptop
>>> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
>>> upload bufferbloat.
>>>
>>> I'm out of ideas now. It seems I can't do anything at all to lower
>>> bufferbloat. Or the Netalyzr test is broken?:-)
>>>
>>> many thanks for your advice,
>>> Jeroen
>>>
>>>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>

[-- Attachment #2: Type: text/html, Size: 5461 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[this_is_not_my_name nor_is_this]

Hi Jeroen,

even though the experts already chimed in, let me try to elucidate what I learned about netalyzr's latency probe.


On Feb 9, 2013, at 01:52 , Forums1000 wrote:

> Hi everyone,
> 
> Can anyone give some tips on how to diagnose the sources of bufferbloat? According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows 7 laptop:
> 
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't know). I also disabled interrupt moderation. 
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my Mikrotik 450G router. Result? Still 550ms of bufferbloat. 
> - Then I put a 100 megabit switch between the cable modem an the laptop (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of upload buffer bloat.

	Netalyzr will only measure the latency of the slowest component of the path between the test machine and the netalyzr servers, typically that will be either a cable modem or a del modem. And indeed that component was always constant in your measurements as was the latency. So this is quite conclusive that uplink device has 550ms worth of worst case buffering.


> 
> I'm out of ideas now. It seems I can't do anything at all to lower bufferbloat. Or the Netalyzr test is broken?:-)

	Unlike typical real traffic, netalyzr uses an inelastic unrelenting UDP flood to measure the absolute worst case buffering. Real traffic typically will try to adjust itself to the existing bandwidth.
	If you follow Dave's advise and put a decent rate limiter and modern queue management in your router (which by all means you should), netalyzr will then measure the routers worst case buffering (which will typically will be even worse than your modem's. 550ms actually is relative decent for home equipment). But for real world cases modern queue management will make all the difference in the world. So unless your typical usage includes UDP floods your actual latency is (almost) guaranteed to be much better. It seems your router is capable of running the current OpenWRT release candidate (ar71xx I would guess, see https://openwrt.org).


best regards
	Sebastian

> 
> many thanks for your advice,
> Jeroen
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

[Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 3529 bytes --]

Hi Sebastian,

Thanks for the information. I really appreciate everyone's efforts to
educate me on this (first time I've used a mailing list so I apologise for
the long quote in my previous reply).
When I rate limit my routers uplink to the cable modem (traffic shaping),
I'm moving the bottleneck to the router, correct?

So a rate limiter alone may make things even worse if the router's TX
buffer is larger than the one in the modem. However, queue management is
not the same as rate limiting I assume? You need both working together?
What I'm asking here, will (let's say CoDel) solve the problem in the
router *entirely* or are there still some hidden hardware buffers that the
router's OS cannot "unbloat"? Mikrotik's OS (called RouterOS) is just some
flavour of Linux by the way.

Regards,
Jeroen

On Sat, Feb 9, 2013 at 7:48 PM, this_is_not_my_name nor_is_this <
moeller0@gmx.de> wrote:

> Hi Jeroen,
>
> even though the experts already chimed in, let me try to elucidate what I
> learned about netalyzr's latency probe.
>
>
> On Feb 9, 2013, at 01:52 , Forums1000 wrote:
>
> > Hi everyone,
> >
> > Can anyone give some tips on how to diagnose the sources of bufferbloat?
> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
> 7 laptop:
> >
> > - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
> know). I also disabled interrupt moderation.
> > Result? Still 550ms.
> > - Then I connected my laptop directly to my cable modem, bypassing my
> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> > - Then I put a 100 megabit switch between the cable modem an the laptop
> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
> upload buffer bloat.
>
>         Netalyzr will only measure the latency of the slowest component of
> the path between the test machine and the netalyzr servers, typically that
> will be either a cable modem or a del modem. And indeed that component was
> always constant in your measurements as was the latency. So this is quite
> conclusive that uplink device has 550ms worth of worst case buffering.
>
>
> >
> > I'm out of ideas now. It seems I can't do anything at all to lower
> bufferbloat. Or the Netalyzr test is broken?:-)
>
>         Unlike typical real traffic, netalyzr uses an inelastic
> unrelenting UDP flood to measure the absolute worst case buffering. Real
> traffic typically will try to adjust itself to the existing bandwidth.
>         If you follow Dave's advise and put a decent rate limiter and
> modern queue management in your router (which by all means you should),
> netalyzr will then measure the routers worst case buffering (which will
> typically will be even worse than your modem's. 550ms actually is relative
> decent for home equipment). But for real world cases modern queue
> management will make all the difference in the world. So unless your
> typical usage includes UDP floods your actual latency is (almost)
> guaranteed to be much better. It seems your router is capable of running
> the current OpenWRT release candidate (ar71xx I would guess, see
> https://openwrt.org).
>
>
> best regards
>         Sebastian
>
> >
> > many thanks for your advice,
> > Jeroen
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 4379 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[this_is_not_my_name nor_is_this]

Hi Jonathan,


On Feb 9, 2013, at 10:15 , Jonathan Morton wrote:

> Latency caused by bufferbloat always appears at the bottleneck device. Usually that is the modem, and you've given no alternative that it could plausibly be. The modems you mention are slightly different model numbers, but that can hide substantial differences in internal configuration.
> 
> For a typical drop-tail queue, the induced latency under load is the size of the buffer divided by the speed of the link draining it. Assuming both modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer, and 220ms is consistent with a 48KB buffer - neither of which would seem excessively large to a modem builder who hasn't heard of bufferbloat. However with a shared cable infrastructure, it is possible that the uplink is constrained by other users on the same segment, which will skew this calculation.
> 
> To cure it without modifying the modem, you need to move the bottleneck to a point where you can control the buffer. You do this by introducing traffic shaping at slightly below the advertised modem uplink speed on one of your own machines and directing all upstream traffic through it.

	This is great advise, I just want to mention that both openWRT's QOS settings as well as cerowrt's simple_qos.sh still show some buffering in netalyzr (in my case accidentally 550ms on the uplink with a 4000Kbit/s cable connection shaped down to 3880Kbit/s using cerowrt's (3.7.4-3) simple_qos script). But real measurements with real TCP traffic (opening 60 webpages simultaneously while saturating the uplink with a big upload) showed that ping times are only marginally affected by the cable connection running constantly close to full saturation. My point netalyzr's worst case scenario is quite a bit detached from real world performance, and good queue management (using one of the codel derivates) will give excellent real world performance while still showing excessive buffering in netalyzr. Make what you will out of that… (my not-even-started-yet pet project is to separate tcp from udp traffic (or even better responsive/elastic from unresponsive/inesastic traffic)) and treat both to an according dropping strategy, i.e. drop UDP much harder, but given my time constraints that will happen in the next decade….

best
	Sebastian

> 
> - Jonathan Morton
> On Feb 9, 2013 7:27 PM, "Forums1000" <forums1000@gmail.com> wrote:
> Hi Jonathan and Dave
> 
> My entire LAN-network is gigabit. My cable subscription is 60 megabit down and 4 megabit up. 
> Now, both my routers' WAN-port and the cable modems' LAN port are also gigabit. The router can route LAN to WAN and the other way around (with NAT and connection tracking enabled) in excess of 100 megabit. 
> 
> Now my cable modem is a Motorola Surfboard SV6120E and hers is a Motorola Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms. Moreover, at her place there are Powerplugs in the path limiting her download to 30 megabit instead of 60 megabit. Yet, the upload lag is much lower than mine. There, it also did not matter where I ran Natalyzr, the result was always 220ms of bufferbload.
> 
> Could this still be only the modem?
> 
> 
> On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com> wrote:
> Hi everyone,
> 
> Can anyone give some tips on how to diagnose the sources of bufferbloat? According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows 7 laptop:
> 
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't know). I also disabled interrupt moderation. 
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my Mikrotik 450G router. Result? Still 550ms of bufferbloat. 
> - Then I put a 100 megabit switch between the cable modem an the laptop (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of upload bufferbloat.
> 
> I'm out of ideas now. It seems I can't do anything at all to lower bufferbloat. Or the Netalyzr test is broken?:-)
> 
> many thanks for your advice,
> Jeroen
> 
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 6339 bytes --]

On Sat, Feb 9, 2013 at 11:10 AM, this_is_not_my_name nor_is_this <
moeller0@gmx.de> wrote:

> Hi Jonathan,
>
>
> On Feb 9, 2013, at 10:15 , Jonathan Morton wrote:
>
> > Latency caused by bufferbloat always appears at the bottleneck device.
> Usually that is the modem, and you've given no alternative that it could
> plausibly be. The modems you mention are slightly different model numbers,
> but that can hide substantial differences in internal configuration.
> >
> > For a typical drop-tail queue, the induced latency under load is the
> size of the buffer divided by the speed of the link draining it. Assuming
> both modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer,
> and 220ms is consistent with a 48KB buffer - neither of which would seem
> excessively large to a modem builder who hasn't heard of bufferbloat.
> However with a shared cable infrastructure, it is possible that the uplink
> is constrained by other users on the same segment, which will skew this
> calculation.
> >
> > To cure it without modifying the modem, you need to move the bottleneck
> to a point where you can control the buffer. You do this by introducing
> traffic shaping at slightly below the advertised modem uplink speed on one
> of your own machines and directing all upstream traffic through it.
>
>         This is great advise, I just want to mention that both openWRT's
> QOS settings as well as cerowrt's simple_qos.sh still show some buffering
> in netalyzr (in my case accidentally 550ms on the uplink with a 4000Kbit/s
> cable connection shaped down to 3880Kbit/s using cerowrt's (3.7.4-3)
> simple_qos script).


A big problem with netanalyzer (presently) is that it measures a single UDP
flow in its buffering analysis. It is unaware of packet scheduling (such as
what fq_codel and sfq and qfq do). So it will show large buffers in a rate
limited fq_codel case... *but they don't matter*, because of the fair
queuing component.

I would rather like them to change their test to try and explicitly
identify where some form of fair queuing is actually in use. SFQ for
example, is widely deployed in free.fr's ISP network, and I would hope a
few other places.

The way to do that, is easy - send one high rate flow and one low rate
flow, with a different tuple for the sent addresses, and observe what
happens to both streams.

We do the same thing with high rate flows + ping in the rrul tests.

You observe large buffers with a single udp flood against fq_codel, with a
single stream, for a while, but all other flows get through it very
rapidly.  So it's totally ok to have a large buffer for a single flow for a
while!

I tried to talk to this in the stanford talk, I need to work on describing
this behavior better.

http://netseminar.stanford.edu/







> But real measurements with real TCP traffic (opening 60 webpages
> simultaneously while saturating the uplink with a big upload) showed that
> ping times are only marginally affected by the cable connection running
> constantly close to full saturation. My point netalyzr's worst case
> scenario is quite a bit detached from real world performance, and good
> queue management (using one of the codel derivates) will give excellent
> real world performance while still showing excessive buffering in netalyzr.
> Make what you will out of that… (my not-even-started-yet pet project is to
> separate tcp from udp traffic (or even better responsive/elastic from
> unresponsive/inesastic traffic)) and treat both to an according dropping
> strategy, i.e. drop UDP much harder, but given my time constraints that
> will happen in the next decade….
>
> best
>         Sebastian
>
> >
> > - Jonathan Morton
> > On Feb 9, 2013 7:27 PM, "Forums1000" <forums1000@gmail.com> wrote:
> > Hi Jonathan and Dave
> >
> > My entire LAN-network is gigabit. My cable subscription is 60 megabit
> down and 4 megabit up.
> > Now, both my routers' WAN-port and the cable modems' LAN port are also
> gigabit. The router can route LAN to WAN and the other way around (with NAT
> and connection tracking enabled) in excess of 100 megabit.
> >
> > Now my cable modem is a Motorola Surfboard SV6120E and hers is a
> Motorola Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms.
> Moreover, at her place there are Powerplugs in the path limiting her
> download to 30 megabit instead of 60 megabit. Yet, the upload lag is much
> lower than mine. There, it also did not matter where I ran Natalyzr, the
> result was always 220ms of bufferbload.
> >
> > Could this still be only the modem?
> >
> >
> > On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com>
> wrote:
> > Hi everyone,
> >
> > Can anyone give some tips on how to diagnose the sources of bufferbloat?
> According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I
> have 550ms of upload bufferbloat. I tried all kinds of stuff on my Windows
> 7 laptop:
> >
> > - For the Intel(R) 82567LF Gigabit Network Connection, I put receive and
> transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't
> know). I also disabled interrupt moderation.
> > Result? Still 550ms.
> > - Then I connected my laptop directly to my cable modem, bypassing my
> Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> > - Then I put a 100 megabit switch between the cable modem an the laptop
> (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of
> upload bufferbloat.
> >
> > I'm out of ideas now. It seems I can't do anything at all to lower
> bufferbloat. Or the Netalyzr test is broken?:-)
> >
> > many thanks for your advice,
> > Jeroen
> >
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: Type: text/html, Size: 7709 bytes --]

[Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

Excellent information. So an AQM-algorithm will sort things on the OS level
of the router and should make things considerably better. However, from
reading around on the matter, it seems drivers for the network device and
the hardware itself also contain buffers. Since, Dave (and respect for
that) is developing CeroWRT, is there anything that can be done about that?
Do we have any idea on how severe the buffering in drivers and hardware is?

A little test I just performed using Windows XP now, indeed shows that
Netalyzr is showing me a worst case scenario:

- a continuous ping (1 ping per second) between 2 routers under my control
has an RTT of 20ms (give or take). The remote router I'm pinging sits
pretty much idle and has nothing better to do than answering the ping.
- uploading a large file to Google drive (thereby saturating my uplink
bandwidth) adds +-10ms of additional latency. Sure it varies a bit between
20 and 30ms and goes to 35ms or even 40ms regularly. Moreover, every now
and then I get a spike to 70-80ms but that spike never lasts more than 3
pings.

All in all considerably lower bloat than the 550ms Netalyzr is indicating.
In order to mimic the worst case scenario, I'd have to transfer using UDP
then?

[-- Attachment #2: Type: text/html, Size: 1285 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2720 bytes --]

On Sat, Feb 9, 2013 at 2:06 PM, Forums1000 <forums1000@gmail.com> wrote:

> Excellent information. So an AQM-algorithm will sort things on the OS
> level of the router and should make things considerably better. However,
> from reading around on the matter, it seems drivers for the network device
> and the hardware itself also contain buffers. Since, Dave (and respect for
> that) is developing CeroWRT, is there anything that can be done about that?
> Do we have any idea on how severe the buffering in drivers and hardware is?
>
>
In linux, it's got a lot better in the past 2 years. Work continues.

I have some data on OSX now, and some on windows, but not a lot.

In APs, routers, and switches, it's not looking very good.


However I note that the second biggest place we see the issue is on the
edge home gateways, dslams, cable head-ends, and many of those use software
rate limiting, which generally has 1 buffer or less native to it, and the
underlying buffering doesn't matter.

(then on top of the software rate limiters are big fat dumb drop tails
queues. currently. sigh)




> A little test I just performed using Windows XP now, indeed shows that
> Netalyzr is showing me a worst case scenario:
>
>
Meh. Try something with window scaling. Or, try a netanalyzer test from
some other machine at the same time you do this one.


> - a continuous ping (1 ping per second) between 2 routers under my control
> has an RTT of 20ms (give or take). The remote router I'm pinging sits
> pretty much idle and has nothing better to do than answering the ping.
> - uploading a large file to Google drive (thereby saturating my uplink
> bandwidth) adds +-10ms of additional latency.
>

I think this is an invalid assumption without actually measuring your
transfer rate to the the gdrive. It would not surprise me if their TCP was
pretty sensitive to latency swings, however, they are very on top of the
bufferbloat issue.

Wouldn't mind a packet capture of that upload while doing the above test.



> Sure it varies a bit between 20 and 30ms and goes to 35ms or even 40ms
> regularly. Moreover, every now and then I get a spike to 70-80ms but that
> spike never lasts more than 3 pings.
>
> All in all considerably lower bloat than the 550ms Netalyzr is indicating.
> In order to mimic the worst case scenario, I'd have to transfer using UDP
> then?
>

Just run a more modern OS....


>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: Type: text/html, Size: 3959 bytes --]

[Bloat] I am unable to pinpoint the source of bufferbloat

[Forums1000]

[-- Attachment #1: Type: text/plain, Size: 4478 bytes --]

Firstly, I can confirm uploading to Google Drive saturates my upload (I
checked the outgoing rate in the router) and  also tried Dropbox and got
the same result.

Anyway, performing an upload with Windows 7 and then pinging the remote
router with windows XP, added 50ms to the "unbloated" 20ms roundtrip times.
So the lowest "most represented" RTT numbers I was getting were around
70ms, the worst well over 100ms.
I said "most represented" in the previous sentence, as there was also a
significant delay variance ("jitter") in the RTT's. The RTT's regularly
dived under 70ms, sometimes even hitting an odd 30ms. The was also a large
swing in the other direction with regular ventures to 110-120ms. Needless
to say, the balance of those ventures tilted more towards the higher
numbers.

So Windows 7, being more modern definitely induces more bufferbloat (thanks
to TCP window scaling:)). I'll be repeating this when running Wireshark as
it slipped my mind to fire it up.

As a sidenote to the above:
I still don't see how an AQM-algorithm will combat the buffering in drivers
and hardware (actually, are the issues pertaining to "drivers" and
"hardware' distinct or referring to the same?). I understand that feeding
less packets to a device will prevent the hardware buffer from filling up
as fast as without AQM, but we cannot actually influence its (the HW
buffer) size, can we? If so, it can still introduce a bottleneck that
cannot be prevented...
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

> On Sat, Feb 9, 2013 at 2:06 PM, Forums1000 <forums1000@gmail.com> wrote:
>
>> Excellent information. So an AQM-algorithm will sort things on the OS
>> level of the router and should make things considerably better. However,
>> from reading around on the matter, it seems drivers for the network device
>> and the hardware itself also contain buffers. Since, Dave (and respect for
>> that) is developing CeroWRT, is there anything that can be done about that?
>> Do we have any idea on how severe the buffering in drivers and hardware is?
>>
>>
> In linux, it's got a lot better in the past 2 years. Work continues.
>
> I have some data on OSX now, and some on windows, but not a lot.
>
> In APs, routers, and switches, it's not looking very good.
>
>
> However I note that the second biggest place we see the issue is on the
> edge home gateways, dslams, cable head-ends, and many of those use software
> rate limiting, which generally has 1 buffer or less native to it, and the
> underlying buffering doesn't matter.
>
> (then on top of the software rate limiters are big fat dumb drop tails
> queues. currently. sigh)
>
>
>
>
>>  A little test I just performed using Windows XP now, indeed shows that
>> Netalyzr is showing me a worst case scenario:
>>
>>
> Meh. Try something with window scaling. Or, try a netanalyzer test from
> some other machine at the same time you do this one.
>
>
>> - a continuous ping (1 ping per second) between 2 routers under my
>> control has an RTT of 20ms (give or take). The remote router I'm pinging
>> sits pretty much idle and has nothing better to do than answering the ping.
>> - uploading a large file to Google drive (thereby saturating my uplink
>> bandwidth) adds +-10ms of additional latency.
>>
>
> I think this is an invalid assumption without actually measuring your
> transfer rate to the the gdrive. It would not surprise me if their TCP was
> pretty sensitive to latency swings, however, they are very on top of the
> bufferbloat issue.
>
> Wouldn't mind a packet capture of that upload while doing the above test.
>
>
>
>>  Sure it varies a bit between 20 and 30ms and goes to 35ms or even 40ms
>> regularly. Moreover, every now and then I get a spike to 70-80ms but that
>> spike never lasts more than 3 pings.
>>
>> All in all considerably lower bloat than the 550ms Netalyzr is
>> indicating. In order to mimic the worst case scenario, I'd have to transfer
>> using UDP then?
>>
>
> Just run a more modern OS....
>
>
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
>

[-- Attachment #2: Type: text/html, Size: 5968 bytes --]

Re: [Bloat] I am unable to pinpoint the source of bufferbloat

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 5387 bytes --]

If you are traffic shaping - and thus limiting traffic to a level that the
hardware can always deal with immediately - then the hardware/driver queues
will always be empty except immediately after a packet has been queued. So
then their size doesn't matter.

There is an initiative within Linux called BQL (Byte Queue Limits) which
aims to deal with the driver queue problem (which for a lot of hardware,
but not all, is synonymous with the hardware queue). This requires small
edits to each driver though.

- Jonathan Morton
 On Feb 10, 2013 6:43 PM, "Forums1000" <forums1000@gmail.com> wrote:

> Firstly, I can confirm uploading to Google Drive saturates my upload (I
> checked the outgoing rate in the router) and  also tried Dropbox and got
> the same result.
>
> Anyway, performing an upload with Windows 7 and then pinging the remote
> router with windows XP, added 50ms to the "unbloated" 20ms roundtrip times.
> So the lowest "most represented" RTT numbers I was getting were around
> 70ms, the worst well over 100ms.
> I said "most represented" in the previous sentence, as there was also a
> significant delay variance ("jitter") in the RTT's. The RTT's regularly
> dived under 70ms, sometimes even hitting an odd 30ms. The was also a large
> swing in the other direction with regular ventures to 110-120ms. Needless
> to say, the balance of those ventures tilted more towards the higher
> numbers.
>
> So Windows 7, being more modern definitely induces more bufferbloat
> (thanks to TCP window scaling:)). I'll be repeating this when running
> Wireshark as it slipped my mind to fire it up.
>
> As a sidenote to the above:
> I still don't see how an AQM-algorithm will combat the buffering in
> drivers and hardware (actually, are the issues pertaining to "drivers" and
> "hardware' distinct or referring to the same?). I understand that feeding
> less packets to a device will prevent the hardware buffer from filling up
> as fast as without AQM, but we cannot actually influence its (the HW
> buffer) size, can we? If so, it can still introduce a bottleneck that
> cannot be prevented...
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>> On Sat, Feb 9, 2013 at 2:06 PM, Forums1000 <forums1000@gmail.com> wrote:
>>
>>> Excellent information. So an AQM-algorithm will sort things on the OS
>>> level of the router and should make things considerably better. However,
>>> from reading around on the matter, it seems drivers for the network device
>>> and the hardware itself also contain buffers. Since, Dave (and respect for
>>> that) is developing CeroWRT, is there anything that can be done about that?
>>> Do we have any idea on how severe the buffering in drivers and hardware is?
>>>
>>>
>> In linux, it's got a lot better in the past 2 years. Work continues.
>>
>> I have some data on OSX now, and some on windows, but not a lot.
>>
>> In APs, routers, and switches, it's not looking very good.
>>
>>
>> However I note that the second biggest place we see the issue is on the
>> edge home gateways, dslams, cable head-ends, and many of those use software
>> rate limiting, which generally has 1 buffer or less native to it, and the
>> underlying buffering doesn't matter.
>>
>> (then on top of the software rate limiters are big fat dumb drop tails
>> queues. currently. sigh)
>>
>>
>>
>>
>>>  A little test I just performed using Windows XP now, indeed shows that
>>> Netalyzr is showing me a worst case scenario:
>>>
>>>
>> Meh. Try something with window scaling. Or, try a netanalyzer test from
>> some other machine at the same time you do this one.
>>
>>
>>> - a continuous ping (1 ping per second) between 2 routers under my
>>> control has an RTT of 20ms (give or take). The remote router I'm pinging
>>> sits pretty much idle and has nothing better to do than answering the ping.
>>> - uploading a large file to Google drive (thereby saturating my uplink
>>> bandwidth) adds +-10ms of additional latency.
>>>
>>
>> I think this is an invalid assumption without actually measuring your
>> transfer rate to the the gdrive. It would not surprise me if their TCP was
>> pretty sensitive to latency swings, however, they are very on top of the
>> bufferbloat issue.
>>
>> Wouldn't mind a packet capture of that upload while doing the above test.
>>
>>
>>
>>>  Sure it varies a bit between 20 and 30ms and goes to 35ms or even 40ms
>>> regularly. Moreover, every now and then I get a spike to 70-80ms but that
>>> spike never lasts more than 3 pings.
>>>
>>> All in all considerably lower bloat than the 550ms Netalyzr is
>>> indicating. In order to mimic the worst case scenario, I'd have to transfer
>>> using UDP then?
>>>
>>
>> Just run a more modern OS....
>>
>>
>>>
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt:
>> http://www.teklibre.com/cerowrt/subscribe.html
>>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 7066 bytes --]