From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-x22f.google.com (mail-oa0-x22f.google.com [IPv6:2607:f8b0:4003:c02::22f]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id DA1FF21F2BD for ; Wed, 21 May 2014 10:47:07 -0700 (PDT) Received: by mail-oa0-f47.google.com with SMTP id i7so2686826oag.6 for ; Wed, 21 May 2014 10:47:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=e8b6xwyBAgZAlSr9JiRyE7jHmcA2OvXM9Z/hJSVJDcA=; b=062hVBuNomQhuWSO4SDh54cHPBBDq+kFXjoamQ7W6hC+P0f8XD9bBnPmYHWaUj4c8V TdUhYcTJqtczXTFczqSZY5/2iK8x87RdsKRXgtBgUmAR/fF+caaWvXpF7qGABokGf9WQ du6nog/DhZErXHwBzVjPNaOTsx8+eC1mVBwyLYLhaKfdB2SyNXydPZC2NTfDzfGTCgcr TpYqPBLmePELm4EfMzd27zcj6U977IGIYAUm3g5bwICOCT0aKicG6oPh9/+o1i4i2LjZ fUDUothB+4M/Z1PJMKPssxTss7ysBHwdVBTgyL1ZD97OVyHD6Px0ezuu8i7fuDtaO6fI 5EDw== MIME-Version: 1.0 X-Received: by 10.60.149.233 with SMTP id ud9mr25785855oeb.66.1400694426920; Wed, 21 May 2014 10:47:06 -0700 (PDT) Sender: gettysjim@gmail.com Received: by 10.76.73.100 with HTTP; Wed, 21 May 2014 10:47:06 -0700 (PDT) In-Reply-To: <1400688188.27216078@apps.rackspace.com> References: <006001cf7478$804951e0$80dbf5a0$@riepnet.com> <006b01cf74e9$c9edf190$5dc9d4b0$@com> <1400683893.25854395@apps.rackspace.com> <1400688188.27216078@apps.rackspace.com> Date: Wed, 21 May 2014 13:47:06 -0400 X-Google-Sender-Auth: eXSGBaI5WdG1e4Y24vO14wYoLeY Message-ID: From: Jim Gettys To: David P Reed Content-Type: multipart/alternative; boundary=047d7b41c0ccc170a804f9ec9509 Cc: Frits Riep , "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration. X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 17:47:08 -0000 --047d7b41c0ccc170a804f9ec9509 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, May 21, 2014 at 12:03 PM, wrote: > In reality we don't disagree on this: > > > > On Wednesday, May 21, 2014 11:19am, "Dave Taht" > said: > > > > > > Well, I disagree somewhat. The downstream shaper we use works quite > > well, until we run out of cpu at 50mbits. Testing on the ubnt edgeroute= r > > has had the inbound shaper work up a little past 100mbits. So there is > > no need (theoretically) to upgrade the big fat head ends if your cpe is > > powerful enough to do the job. It would be better if the head ends did > it, > > of course.... > > > > > > There is an advantage for the head-ends doing it, to the extent that each > edge device has no clarity about what is happening with all the other cpe > that are sharing that head-end. When there is bloat in the head-end even = if > all cpe's sharing an upward path are shaping themselves to the "up to" > speed the provider sells, they can go into serious congestion if the > head-end queues can grow to 1 second or more of sustained queueing delay. > My understanding is that head-end queues have more than that. They > certainly do in LTE access networks. > =E2=80=8BI have measured 200ms on a 28Mbps LTE quadrant to a single station= . This was using the simplest possible test on an idle cell. Easy to see how that can grow to the second range. Similarly, Dave Taht and I took data recently that showed a large downstream buffer at the CMTS end (line card?), IIRC, it was something like .25 megabyte, using a UDP flooding tool. As always, there may be multiple different buffers lurking in these complex devices, which may only come into play when different parts of them "bottleneck", just as we found many different buffering locations inside of Linux. In fact, some of these devices include Linux boxes (though I do not know if they are on the packet forwarding path or not). Bandwidth shaping downstream of those bottlenecks can help, but only to a degree, and I believe primarily for "well behaved" long lived elephant flows. Offload engines on servers and coalescing acks in various equipment makes the degree of help, particularly for transient behavior such as opening a bunch of TCP connections simultaneously and downloading the elements of a web page I believe are likely to put large bursts of packets into these queues, causing transient poor latency. I think we'll get a bit of help out of the packet pacing code that recently went into Linux (for well behaved servers) as it deploys. Thanks to Eric Dumazet for that work! Ironically, servers get updated much more frequently than these middle boxes, as far as I can tell. Somehow we gotta get the bottlenecks in these devices (broadband & cellular) to behave better. - Jim > > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel > > --047d7b41c0ccc170a804f9ec9509 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Wed= , May 21, 2014 at 12:03 PM, <dpreed@reed.com> wrote:

In reality we don't disagree on this:

=C2=A0

On Wednesday, May 21, 2014 11:19am, "Dave Taht&quo= t; <dave.taht@g= mail.com> said:

>

> Well, I disagree somewhat. The downstr= eam shaper we use works quite
> well, until we run out of cpu at 50mb= its. Testing on the ubnt edgerouter
> has had the inbound shaper work= up a little past 100mbits. So there is
> no need (theoretically) to upgrade the big fat head ends if your cpe i= s
> powerful enough to do the job. It would be better if the head end= s did it,
> of course....
>

=C2=A0

There is an advantage for the head-en= ds doing it, to the extent that each edge device has no clarity about what = is happening with all the other cpe that are sharing that head-end. When th= ere is bloat in the head-end even if all cpe's sharing an upward path a= re shaping themselves to the "up to" speed the provider sells, th= ey can go into serious congestion if the head-end queues can grow to 1 seco= nd or more of sustained queueing delay. =C2=A0My understanding is that head= -end queues have more than that. =C2=A0They certainly do in LTE access netw= orks.


=E2=80=8BI have measured 200ms on a 28Mbps LTE quadran= t to a single station. =C2=A0This was using the simplest possible test on a= n idle cell. =C2=A0Easy to see how that can grow to the second range.

Similarly, Dave Taht and I too= k data recently that showed a large downstream buffer at the CMTS end (line= card?), IIRC, it was something like .25 megabyte, using a UDP flooding too= l. =C2=A0

As always, there may be multip= le different buffers lurking in these complex devices, which may only come = into play when different parts of them "bottleneck", just as we f= ound many different buffering locations inside of Linux. =C2=A0In fact, som= e of these devices include Linux boxes (though I do not know if they are on= the packet forwarding path or not).

Bandwidth shaping downstream o= f those bottlenecks can help, but only to a degree, and I believe primarily= for "well behaved" long lived elephant flows. =C2=A0Offload engi= nes on servers and coalescing acks in various equipment makes the degree of= help, particularly for transient behavior such as opening a bunch of TCP c= onnections simultaneously and downloading the elements of a web page I beli= eve are likely to put large bursts of packets into these queues, causing tr= ansient poor latency. =C2=A0I think we'll get a bit of help out of the = packet pacing code that recently went into Linux (for well behaved servers)= as it deploys. =C2=A0Thanks to Eric Dumazet for that work! =C2=A0Ironicall= y, servers get updated much more frequently than these middle boxes, as far= as I can tell.

Somehow we gotta get the bottl= enecks in these devices (broadband & cellular) to behave better.
<= div class=3D"gmail_default" style=3D"font-size:small"> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 - Jim

=C2=A0


_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.= bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel


--047d7b41c0ccc170a804f9ec9509--