From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 725493B2A3 for ; Sat, 24 Dec 2016 16:15:13 -0500 (EST) Received: by mail-lf0-x22b.google.com with SMTP id c13so156156289lfg.0 for ; Sat, 24 Dec 2016 13:15:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pgJCEcAMpAy9A3mIX/gTpa0kxS8jm/CQQYGUaAJsc/8=; b=EiSClOLy+UqV+plGtf17gRiires0fXJr63O6dFvQbb1Zxsa1p4KghRe7rHsTLuPUvN TKFZMmh+MvGc61AwFg75NUDiEjp7oP4FNM6o5V3o8MpMpEph5Gx3UCVWYb2CrkkoG0iH Y/cnio92vy/12Zz5vGM82n1OJQWPnmZKt4YJtwzT9ATzMPoiwQgqKv1hH+auSNafbLsl 4y7gx/pYMZoyd2+rQBtWYTuMRNh7YMLPEYHTIiBSV2giLo6if/EG//1ncVpJ5pvzpDSP oKEycNgh5t1TQlN5GaaqdPNJOKJP1BXK6pH2t5H2zvD78X2tBb5RxU3zc/MnXQaElA1L CgHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pgJCEcAMpAy9A3mIX/gTpa0kxS8jm/CQQYGUaAJsc/8=; b=eGTqIiT40q31scaKVa485oq4UQBMgyKzS57NreQ1RfjCpyuMF0foYSfajwfPBNvbM/ cWpVEvo8S8oTXOUWT1nUVVGU96uj4JQQK/QYtnmp2eiuqeIom7HLkF1zBsxd5m9VcjkD FJ3VDEH29lka7GIMHLKQ9Kr+/1+uzpbVY9rkbXb88Vj/2ufUm3VY93hWr+mIZbHzgaAN 5djH6JmXC75dWlmMiMLyzZC7Uw1tlPEpGyF3f64Povd5+nVj7kl8vzEgZHM38Wk9+rEr xTqDxVzStGzjO1URhyYnoXq8c7m3nzrGLTi5Sq8279TBp8tE7KLsOlUz0s1M7IYIwZwX QzFQ== X-Gm-Message-State: AIkVDXLjpf3QI3vLGEtZxVfqHOHUdGKHQxOEM/35fD4Yh1kN00cqfC4Xhw/A0SqQmFXT26cRv+L8N/YjcnUmLQ== X-Received: by 10.46.70.17 with SMTP id t17mr8998961lja.38.1482614111775; Sat, 24 Dec 2016 13:15:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.198.212 with HTTP; Sat, 24 Dec 2016 13:15:11 -0800 (PST) In-Reply-To: <76C6FEF2-4939-4601-9F78-43B3077C39E4@gmail.com> References: <20161222174349.5bd5dffd@xeon-e3> <9A8ACB3B-8411-414E-B2C3-ABE2C276B351@gmail.com> <606D54F6-59FB-49D6-A969-F26434EF9292@gmx.de> <76C6FEF2-4939-4601-9F78-43B3077C39E4@gmail.com> From: Benjamin Cronce Date: Sat, 24 Dec 2016 15:15:11 -0600 Message-ID: To: Jonathan Morton Cc: Sebastian Moeller , cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=f403045f789e78b31c05446dff5c Subject: Re: [Cake] upstreaming cake in 2017? X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Dec 2016 21:15:13 -0000 --f403045f789e78b31c05446dff5c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sat, Dec 24, 2016 at 11:22 AM, Jonathan Morton wrote: > > > On 24 Dec, 2016, at 17:55, Benjamin Cronce wrote: > > > > What was also interesting is the flows consuming the majority of the > buffer were always in flux. You would think the same few flows that were > consuming the buffer at one moment would continue to, but that is not the > case, TCP keeps them alternating. > > That sounds like the links are not actually congested on average, and the > flows which temporarily collect in the buffer are due to transitory burst= s > - which is what you=E2=80=99d expect from a competently-managed backbone.= A > flow-isolating AQM doesn=E2=80=99t really help there, though Cake should = be capable > of scaling up to 10Gbps on a modern CPU. > > Conversely, there have been well-publicised instances of congestion at > peering points, which have had substantial impacts on performance. I > imagine the relevant flow counts there would be very much higher, > definitely in the thousands. Even well-managed networks occasionally > experience congestion due to exceptional loads. > At least in my experience, most of the issues of congested peering is just bufferbloat. You can get a 10Gb switch with 4GiB of buffer, which is like 4 seconds of buffer. Nearly every time I see someone talking about congestion, you see pings increasing by 200ms+, many times in the thousands of milliseconds. A "congested" link should show maybe 50ms of latency increase with an increase in packetloss, but everyone knows how bad loss is and bloats the buffers. My argument is that an unbloated buffer has very few states in the buffer at any given time. I would like to see some numbers from fq_Codel or Cake about actual unique states at any given moment. > > The workings of DRR++ are also somewhat more subtle than simply counting > the flows instantaneously in the buffer. Each queue has a deficit, and i= n > Cake an empty queue is not normally released from tracking a particular > flow until the deficit has been repaid (by cycling through all the other > flows and probably servicing them) and decaying the AQM state to rest, > which may often take long enough for another packet to arrive for that fl= ow. > > The number of active bulk flows can therefore exceed the number of packet= s > actually in the queue. This is especially true if the AQM is working > optimally and keeping the queue almost empty on average. > > While fq_codel does not explicitly assign queues to specific flows (ie. t= o > avoid hash collisions), the effects of hash collisions are similarly felt > under the same circumstances, resulting in the colliding flows failing to > receive their theoretical fair share of the link, even if they never have > packets physically in the queue at the same time. > > With that said, both fq_codel and Cake should work okay with statistical > multiplexing to handle exceptional flow counts. In such cases, Cake=E2= =80=99s > triple-isolate feature should be turned off, by selecting either =E2=80= =9Chosts=E2=80=9D or > =E2=80=9Cflows=E2=80=9D modes. I could run an analysis to show how even = the multiplexing > should be. > > - Jonathan Morton > > --f403045f789e78b31c05446dff5c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Sat, Dec 24, 2016 at 11:22 AM, Jonathan Morton <= ;chromatix99@gma= il.com> wrote:

> On 24 Dec, 2016, at 17:55, Benjamin Cronce <bcronce@gmail.com> wrote:
>
> What was also interesting is the flows consuming the majority of the b= uffer were always in flux. You would think the same few flows that were con= suming the buffer at one moment would continue to, but that is not the case= , TCP keeps them alternating.

That sounds like the links are not actually congested on average, an= d the flows which temporarily collect in the buffer are due to transitory b= ursts - which is what you=E2=80=99d expect from a competently-managed backb= one.=C2=A0 A flow-isolating AQM doesn=E2=80=99t really help there, though C= ake should be capable of scaling up to 10Gbps on a modern CPU.

Conversely, there have been well-publicised instances of congestion at peer= ing points, which have had substantial impacts on performance.=C2=A0 I imag= ine the relevant flow counts there would be very much higher, definitely in= the thousands.=C2=A0 Even well-managed networks occasionally experience co= ngestion due to exceptional loads.

At l= east in my experience, most of the issues of congested peering is just buff= erbloat. You can get a 10Gb switch with 4GiB of buffer, which is like 4 sec= onds of buffer. Nearly every time I see someone talking about congestion, y= ou see pings increasing by 200ms+, many times in the thousands of milliseco= nds. A "congested" link should show maybe 50ms of latency increas= e with an increase in packetloss, but everyone knows how bad loss is and bl= oats the buffers.

My argument is that an unbloated= buffer has very few states in the buffer at any given time. I would like t= o see some numbers from fq_Codel or Cake about actual unique states at any = given moment.
=C2=A0

The workings of DRR++ are also somewhat more subtle than simply counting th= e flows instantaneously in the buffer.=C2=A0 Each queue has a deficit, and = in Cake an empty queue is not normally released from tracking a particular = flow until the deficit has been repaid (by cycling through all the other fl= ows and probably servicing them) and decaying the AQM state to rest, which = may often take long enough for another packet to arrive for that flow.

The number of active bulk flows can therefore exceed the number of packets = actually in the queue.=C2=A0 This is especially true if the AQM is working = optimally and keeping the queue almost empty on average.

While fq_codel does not explicitly assign queues to specific flows (ie. to = avoid hash collisions), the effects of hash collisions are similarly felt u= nder the same circumstances, resulting in the colliding flows failing to re= ceive their theoretical fair share of the link, even if they never have pac= kets physically in the queue at the same time.

With that said, both fq_codel and Cake should work okay with statistical mu= ltiplexing to handle exceptional flow counts.=C2=A0 In such cases, Cake=E2= =80=99s triple-isolate feature should be turned off, by selecting either = =E2=80=9Chosts=E2=80=9D or =E2=80=9Cflows=E2=80=9D modes.=C2=A0 I could run= an analysis to show how even the multiplexing should be.

=C2=A0- Jonathan Morton


--f403045f789e78b31c05446dff5c--