From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qv1-xf2f.google.com (mail-qv1-xf2f.google.com [IPv6:2607:f8b0:4864:20::f2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 424B63B29E for ; Wed, 25 Mar 2020 11:57:57 -0400 (EDT) Received: by mail-qv1-xf2f.google.com with SMTP id q73so1306329qvq.2 for ; Wed, 25 Mar 2020 08:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QLaNyjwJpFHveqbuKZmCKBdpYRxmaPH7NifGB5oEBxA=; b=NlwGKFOgf0ojmYggQjXcsUH2VNniP2Ej5OxNvInqvHA5xfGfsctk5HcxRm+2f219le P7X1yj465g24h4XD/XMt+S3XNckRNbKMju9svisgzLHDSg4NzdZkKuGLdX7c6AZiR0iR UWiaL8RxoOjkjELYnCL00j0mRwuOR19SIkJViyV/ZjaeayelYuKVEN65cJ7VCnaBEt9G EpBlcGEElFC8cBQOcpn2zWPL29MVAEj9/uhvu/arP4DVWlAfTj3Spj7dVYRlazi6vYrm z4q81nyvx6gQ0LjqsfMPBIphTtsr6wbImkhY71wxou63aSEQWEllGZwfpwJhTQTlntY9 /CfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QLaNyjwJpFHveqbuKZmCKBdpYRxmaPH7NifGB5oEBxA=; b=HByHdikgxQjPL4+4nvTbl3Y8SFnFfHqI5Ia7HcLoalpqSktx7ZhfP79ARKa0IfO2CC S07JVmz1gAS5SkylJnBJ9R3kgz4vpCxP/zQb2wSvjB/7KxLSl48wep5gh40JgnPpY7oX 0nf/EbTIy6SFXo2rLFfvXoHHmrfAeqYjEIkozlmi5JLZEEFlfiAdldbisJR4MlyOljA0 3gIjOZAV5TQUdrWP8rkofKu32VJKw7prVK2Ng2CjpEGinSfHfR7FYUp3981vk2tfhEL2 I4DnXTfyvBF+DQ6eTroZIyqrsbupSWNTjMEblTt5SA+p59FPvDDCYWiLrdTPoc83WNZn qJcQ== X-Gm-Message-State: ANhLgQ1lv3JIPCpDSKpP+A33kJ0MhIxrpKfbRBat4XZUSc1N7EOp3toa MR3ng4nlVBwmtpX+i9facdZcrdH8hQ0iL62Ycbo= X-Google-Smtp-Source: ADFU+vtl8r5eghnDGuQgqTjgwhDn1dAMUVnG+0vv40nAsgLUuZfQ49RLZgtlyXEPUuUfodfifew6z+S9l6yhqUeTu+U= X-Received: by 2002:a0c:c389:: with SMTP id o9mr3872190qvi.232.1585151876806; Wed, 25 Mar 2020 08:57:56 -0700 (PDT) MIME-Version: 1.0 References: <875zesret5.fsf@toke.dk> <87r1xgpuhm.fsf@toke.dk> In-Reply-To: <87r1xgpuhm.fsf@toke.dk> From: Aaron Wood Date: Wed, 25 Mar 2020 08:57:44 -0700 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Sebastian Moeller , bloat Content-Type: multipart/alternative; boundary="000000000000883ea405a1afee8f" Subject: Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Mar 2020 15:57:57 -0000 --000000000000883ea405a1afee8f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable One other thought I've had with this, is that the apu2 is multi-core, and the i210 is multi-queue. Cake/htb aren't, iirc, setup to run on multiple cores (as the rate limiters then don't talk to each other). But with the correct tuple hashing in the i210, I _should_ be able to split things and do two cores at 500Mbps each (with lots of compute left over). Obviously, that puts a limit on single-connection rates, but as the number of connections climb, they should more or less even out (I remember Dave Taht showing the oddities that happen with say 4 streams and 2 cores, where it's common to end up with 3 streams on the same core). But assuming that the hashing function results in even sharing of streams, it should be fairly balanced (after plotting some binomial distributions with higher "n" values). Still not perfect, especially since streams aren't likely to all be elephants. On Wed, Mar 25, 2020 at 4:03 AM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Sebastian Moeller writes: > > > Hi Toke, > > > > > >> On Mar 25, 2020, at 09:58, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> > >> Aaron Wood writes: > >> > >>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabi= t > >>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it= . > >>> > >>> Flent test results are here: > >>> > https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit= -with.html > >>> > >>> tl/dr; 1000ms of upstream bufferbloat > >>> > >>> But it's DOCSIS 3.1, so why isn't PIE working? Theory: It's in > DOCSIS 3.0 > >>> upstream mode based on the status LEDs. Hopefully it will go away if > I can > >>> convince it to run in DOCSIS 3.1 mode. > >> > >> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the > >> ISP still has to turn it on? So maybe yelling at them will work? (ha!) > >> > >>> At the moment, however, my WRT1900AC isn't up to the task of dealing > with > >>> these sorts of downstream rates. > >>> > >>> So I'm looking at the apu2, which from this post: > >>> > https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-= sqm-wireguard-and-openvpn/44724 > >>> > >>> Will certainly get most of the way there. > >> > >> My Turris Omnia is doing fine on my 1Gbps connection (although that > >> hardly suffers from bloat, so I'm not doing any shaping; did try it > >> though, and it has no problem with running CAKE at 1Gbps). > > > > Well, doing local network flent RRUL stress tests indicated that > > my omnia (at that time with TOS4/Openwrt18) only allowed up to > > 500/500 Mbps shaping with bi directionally saturating traffic > > with full MTU-sized packets. So I undirectional CAKE at 1Gbps > > can work, but under full load, I did not manage that, what did I > > wrong? > > Hmm, not sure I've actually done full bidirectional shaping. And trying > it now, it does seem to be struggling... > > -Toke > --000000000000883ea405a1afee8f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
One other thought I've had with this, is that the apu2= is multi-core, and the i210 is multi-queue.

Cake/htb ar= en't, iirc, setup to run on multiple cores (as the rate limiters then d= on't talk to each other).=C2=A0 But with the correct tuple hashing in t= he i210, I _should_ be able to split things and do two cores at 500Mbps eac= h (with lots of compute left over). =C2=A0

Obvious= ly, that puts a limit on single-connection rates, but as the number of conn= ections climb, they should more or less even out (I remember Dave Taht show= ing the oddities that happen with say 4 streams and 2 cores, where it's= common to end up with 3 streams on the same core).=C2=A0 But assuming that= the hashing function results in even sharing of streams, it should be fair= ly balanced (after plotting some binomial distributions with=C2=A0higher &q= uot;n" values).=C2=A0 Still not perfect, especially since streams aren= 't likely to all be elephants.

On Wed, Mar 25, 2020 at 4:03 AM Tok= e H=C3=B8iland-J=C3=B8rgensen <toke@toke= .dk> wrote:
Sebastian Moeller <moeller0@gmx.de> write= s:

> Hi Toke,
>
>
>> On Mar 25, 2020, at 09:58, Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk> wrote:=
>>
>> Aaron Wood <woody77@gmail.com> writes:
>>
>>> I recently upgraded service from 150up, 10dn Mbps to xfinity&#= 39;s gigabit
>>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go = with it.
>>>
>>> Flent test results are here:
>>> http= s://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.= html
>>>
>>> tl/dr;=C2=A0 1000ms of upstream bufferbloat
>>>
>>> But it's DOCSIS 3.1, so why isn't PIE working?=C2=A0 T= heory:=C2=A0 It's in DOCSIS 3.0
>>> upstream mode based on the status LEDs.=C2=A0 Hopefully it wil= l go away if I can
>>> convince it to run in DOCSIS 3.1 mode.
>>
>> I think that while PIE is "mandatory to implement" in DO= CSIS 3.1, the
>> ISP still has to turn it on? So maybe yelling at them will work? (= ha!)
>>
>>> At the moment, however, my WRT1900AC isn't up to the task = of dealing with
>>> these sorts of downstream rates.
>>>
>>> So I'm looking at the apu2, which from this post:
>>> https://forum.openwrt.org/t/comparative-throughput-testing= -including-nat-sqm-wireguard-and-openvpn/44724
>>>
>>> Will certainly get most of the way there.
>>
>> My Turris Omnia is doing fine on my 1Gbps connection (although tha= t
>> hardly suffers from bloat, so I'm not doing any shaping; did t= ry it
>> though, and it has no problem with running CAKE at 1Gbps).
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0Well, doing local network flent RRUL stress = tests indicated that
>=C2=A0 =C2=A0 =C2=A0 =C2=A0my omnia (at that time with TOS4/Openwrt18) = only allowed up to
>=C2=A0 =C2=A0 =C2=A0 =C2=A0500/500 Mbps shaping with bi directionally s= aturating traffic
>=C2=A0 =C2=A0 =C2=A0 =C2=A0with full MTU-sized packets. So I undirectio= nal CAKE at 1Gbps
>=C2=A0 =C2=A0 =C2=A0 =C2=A0can work, but under full load, I did not man= age that, what did I
>=C2=A0 =C2=A0 =C2=A0 =C2=A0wrong?

Hmm, not sure I've actually done full bidirectional shaping. And trying=
it now, it does seem to be struggling...

-Toke
--000000000000883ea405a1afee8f--