From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-x22f.google.com (ie-in-x022f.1e100.net [IPv6:2607:f8b0:4001:c03::22f]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id D9604202102 for ; Sat, 9 Feb 2013 11:58:41 -0800 (PST) Received: by mail-ie0-f175.google.com with SMTP id c12so6234859ieb.20 for ; Sat, 09 Feb 2013 11:58:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=oTTpaH4zv7bgM2Kp/3hRT2oN21/mRQKvkGbFx36dhKs=; b=p7rGzsI0mW5ydfw5r79UL3BpcPoK7i9MzOJxzj7HUB9Hwa/kn3M9a6FgWFsatAEu7T QaSGpXVnN9UcEBSXvmerF/TqiUGtObJ9TWofz7i9H8LLhS16QOybi8mQskbTbkbpS16M sF0ohogITDwxmartVkeOamhIQQB5MryLyEVpX/kSIVrNpvEQM3QiMc+9Qtw/tz3OJ2Vj Q+iYJKDLny2F3ubF28JkPyZRHPlyRT0Ne1TuRBGwoDuYISBiAk2EMCQq9q0RRnb5T4YG pmFmNxNrMGIP0roN44dokqkl+CRkaETF210LFwMx9Fvs+x+pZZyuW/zzCcYpF9WKDFiN dsgA== MIME-Version: 1.0 X-Received: by 10.42.54.5 with SMTP id p5mr14017363icg.49.1360439921092; Sat, 09 Feb 2013 11:58:41 -0800 (PST) Received: by 10.64.135.39 with HTTP; Sat, 9 Feb 2013 11:58:40 -0800 (PST) In-Reply-To: References: Date: Sat, 9 Feb 2013 11:58:40 -0800 Message-ID: From: Dave Taht To: this_is_not_my_name nor_is_this Content-Type: multipart/alternative; boundary=90e6ba3fd72d3c27eb04d5501a3a Cc: bloat Subject: Re: [Bloat] I am unable to pinpoint the source of bufferbloat X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2013 19:58:42 -0000 --90e6ba3fd72d3c27eb04d5501a3a Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Sat, Feb 9, 2013 at 11:10 AM, this_is_not_my_name nor_is_this < moeller0@gmx.de> wrote: > Hi Jonathan, > > > On Feb 9, 2013, at 10:15 , Jonathan Morton wrote: > > > Latency caused by bufferbloat always appears at the bottleneck device. > Usually that is the modem, and you've given no alternative that it could > plausibly be. The modems you mention are slightly different model numbers= , > but that can hide substantial differences in internal configuration. > > > > For a typical drop-tail queue, the induced latency under load is the > size of the buffer divided by the speed of the link draining it. Assuming > both modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer, > and 220ms is consistent with a 48KB buffer - neither of which would seem > excessively large to a modem builder who hasn't heard of bufferbloat. > However with a shared cable infrastructure, it is possible that the uplin= k > is constrained by other users on the same segment, which will skew this > calculation. > > > > To cure it without modifying the modem, you need to move the bottleneck > to a point where you can control the buffer. You do this by introducing > traffic shaping at slightly below the advertised modem uplink speed on on= e > of your own machines and directing all upstream traffic through it. > > This is great advise, I just want to mention that both openWRT's > QOS settings as well as cerowrt's simple_qos.sh still show some buffering > in netalyzr (in my case accidentally 550ms on the uplink with a 4000Kbit/= s > cable connection shaped down to 3880Kbit/s using cerowrt's (3.7.4-3) > simple_qos script). A big problem with netanalyzer (presently) is that it measures a single UDP flow in its buffering analysis. It is unaware of packet scheduling (such as what fq_codel and sfq and qfq do). So it will show large buffers in a rate limited fq_codel case... *but they don't matter*, because of the fair queuing component. I would rather like them to change their test to try and explicitly identify where some form of fair queuing is actually in use. SFQ for example, is widely deployed in free.fr's ISP network, and I would hope a few other places. The way to do that, is easy - send one high rate flow and one low rate flow, with a different tuple for the sent addresses, and observe what happens to both streams. We do the same thing with high rate flows + ping in the rrul tests. You observe large buffers with a single udp flood against fq_codel, with a single stream, for a while, but all other flows get through it very rapidly. So it's totally ok to have a large buffer for a single flow for a while! I tried to talk to this in the stanford talk, I need to work on describing this behavior better. http://netseminar.stanford.edu/ > But real measurements with real TCP traffic (opening 60 webpages > simultaneously while saturating the uplink with a big upload) showed that > ping times are only marginally affected by the cable connection running > constantly close to full saturation. My point netalyzr's worst case > scenario is quite a bit detached from real world performance, and good > queue management (using one of the codel derivates) will give excellent > real world performance while still showing excessive buffering in netalyz= r. > Make what you will out of that=85 (my not-even-started-yet pet project is= to > separate tcp from udp traffic (or even better responsive/elastic from > unresponsive/inesastic traffic)) and treat both to an according dropping > strategy, i.e. drop UDP much harder, but given my time constraints that > will happen in the next decade=85. > > best > Sebastian > > > > > - Jonathan Morton > > On Feb 9, 2013 7:27 PM, "Forums1000" wrote: > > Hi Jonathan and Dave > > > > My entire LAN-network is gigabit. My cable subscription is 60 megabit > down and 4 megabit up. > > Now, both my routers' WAN-port and the cable modems' LAN port are also > gigabit. The router can route LAN to WAN and the other way around (with N= AT > and connection tracking enabled) in excess of 100 megabit. > > > > Now my cable modem is a Motorola Surfboard SV6120E and hers is a > Motorola Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms= . > Moreover, at her place there are Powerplugs in the path limiting her > download to 30 megabit instead of 60 megabit. Yet, the upload lag is much > lower than mine. There, it also did not matter where I ran Natalyzr, the > result was always 220ms of bufferbload. > > > > Could this still be only the modem? > > > > > > On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 > wrote: > > Hi everyone, > > > > Can anyone give some tips on how to diagnose the sources of bufferbloat= ? > According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I > have 550ms of upload bufferbloat. I tried all kinds of stuff on my Window= s > 7 laptop: > > > > - For the Intel(R) 82567LF Gigabit Network Connection, I put receive an= d > transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don't > know). I also disabled interrupt moderation. > > Result? Still 550ms. > > - Then I connected my laptop directly to my cable modem, bypassing my > Mikrotik 450G router. Result? Still 550ms of bufferbloat. > > - Then I put a 100 megabit switch between the cable modem an the laptop > (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of > upload bufferbloat. > > > > I'm out of ideas now. It seems I can't do anything at all to lower > bufferbloat. Or the Netalyzr test is broken?:-) > > > > many thanks for your advice, > > Jeroen > > > > > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --=20 Dave T=E4ht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html --90e6ba3fd72d3c27eb04d5501a3a Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable

On Sat, Feb 9, 2013 at 11:10 AM, this_is= _not_my_name nor_is_this <moeller0@gmx.de> wrote:
Hi Jonathan,


On Feb 9, 2013, at 10:15 , Jonathan Morton wrote:

> Latency caused by bufferbloat always appears at the bottleneck device.= Usually that is the modem, and you've given no alternative that it cou= ld plausibly be. The modems you mention are slightly different model number= s, but that can hide substantial differences in internal configuration.
>
> For a typical drop-tail queue, the induced latency under load is the s= ize of the buffer divided by the speed of the link draining it. Assuming bo= th modems have a 4Mbit uplink, 550ms is consistent with a 256KB buffer, and= 220ms is consistent with a 48KB buffer - neither of which would seem exces= sively large to a modem builder who hasn't heard of bufferbloat. Howeve= r with a shared cable infrastructure, it is possible that the uplink is con= strained by other users on the same segment, which will skew this calculati= on.
>
> To cure it without modifying the modem, you need to move the bottlenec= k to a point where you can control the buffer. You do this by introducing t= raffic shaping at slightly below the advertised modem uplink speed on one o= f your own machines and directing all upstream traffic through it.

=A0 =A0 =A0 =A0 This is great advise, I just want to mention that bot= h openWRT's QOS settings as well as cerowrt's simple_qos.sh still s= how some buffering in netalyzr (in my case accidentally 550ms on the uplink= with a 4000Kbit/s cable connection shaped down to 3880Kbit/s using cerowrt= 's (3.7.4-3) simple_qos script).

A big problem with netanalyzer (presently) is that it measures a s= ingle UDP flow in its buffering analysis. It is unaware of packet schedulin= g (such as what fq_codel and sfq and qfq do). So it will show large buffers= in a rate limited fq_codel case... *but they don't matter*, because of= the fair queuing component.

I would rather like them to change their test to try and explicitly ide= ntify where some form of fair queuing is actually in use. SFQ for example, = is widely deployed in free.fr's ISP netw= ork, and I would hope a few other places.

The way to do that, is easy - send one high rate flow and one low rate = flow, with a different tuple for the sent addresses, and observe what happe= ns to both streams.

We do the same thing with high rate flows + pin= g in the rrul tests.

You observe large buffers with a single udp flood against fq_codel, wit= h a single stream, for a while, but all other flows get through it very rap= idly.=A0 So it's totally ok to have a large buffer for a single flow fo= r a while!

I tried to talk to this in the stanford talk, I need to work on describ= ing this behavior better.

http://netseminar.stanford.edu/





=A0
But real measurements with real TCP traffic (opening 60 webpages simultaneo= usly while saturating the uplink with a big upload) showed that ping times = are only marginally affected by the cable connection running constantly clo= se to full saturation. My point netalyzr's worst case scenario is quite= a bit detached from real world performance, and good queue management (usi= ng one of the codel derivates) will give excellent real world performance w= hile still showing excessive buffering in netalyzr. Make what you will out = of that=85 (my not-even-started-yet pet project is to separate tcp from udp= traffic (or even better responsive/elastic from unresponsive/inesastic tra= ffic)) and treat both to an according dropping strategy, i.e. drop UDP much= harder, but given my time constraints that will happen in the next decade= =85.

best
=A0 =A0 =A0 =A0 Sebastian

>
> - Jonathan Morton
> On Feb 9, 2013 7:27 PM, "Forums1000" <forums1000@gmail.com> wrote:
> Hi Jonathan and Dave
>
> My entire LAN-network is gigabit. My cable subscription is 60 megabit = down and 4 megabit up.
> Now, both my routers' WAN-port and the cable modems' LAN port = are also gigabit. The router can route LAN to WAN and the other way around = (with NAT and connection tracking enabled) in excess of 100 megabit.
>
> Now my cable modem is a Motorola Surfboard SV6120E and hers is a Motor= ola Surfboard CV6181E. My upload lag is 550ms and hers is only 220ms. Moreo= ver, at her place there are Powerplugs in the path limiting her download to= 30 megabit instead of 60 megabit. Yet, the upload lag is much lower than m= ine. There, it also did not matter where I ran Natalyzr, the result was alw= ays 220ms of bufferbload.
>
> Could this still be only the modem?
>
>
> On Sat, Feb 9, 2013 at 10:52 AM, Forums1000 <forums1000@gmail.com> wrote:
> Hi everyone,
>
> Can anyone give some tips on how to diagnose the sources of bufferbloa= t? According to the Netalyzr test at http://netalyzr.icsi.berkeley.edu/, I have 5= 50ms of upload bufferbloat. I tried all kinds of stuff on my Windows 7 lapt= op:
>
> - For the Intel(R) 82567LF Gigabit Network Connection, I put receive a= nd transmit buffers to the lowest value of 80 (80 bytes? 80 packets? I don&= #39;t know). I also disabled interrupt moderation.
> Result? Still 550ms.
> - Then I connected my laptop directly to my cable modem, bypassing my = Mikrotik 450G router. Result? Still 550ms of bufferbloat.
> - Then I put a 100 megabit switch between the cable modem an the lapto= p (as both cable modem and Intel NIC are gigabit). Result? Still 550ms of u= pload bufferbloat.
>
> I'm out of ideas now. It seems I can't do anything at all to l= ower bufferbloat. Or the Netalyzr test is broken?:-)
>
> many thanks for your advice,
> Jeroen
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat= .net
> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat= .net
> https://lists.bufferbloat.net/listinfo/bloat

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat



--
Dave T=E4ht=

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/= subscribe.html=20 --90e6ba3fd72d3c27eb04d5501a3a--