[Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
(with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.

Flent test results are here:
https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html

tl/dr;  1000ms of upstream bufferbloat

But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
upstream mode based on the status LEDs.  Hopefully it will go away if I can
convince it to run in DOCSIS 3.1 mode.

At the moment, however, my WRT1900AC isn't up to the task of dealing with
these sorts of downstream rates.

So I'm looking at the apu2, which from this post:
https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724

Will certainly get most of the way there.

Although

[-- Attachment #2: Type: text/html, Size: 1336 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

(hit send early, somehow)...

Although this thread makes we wonder if perhaps not:
https://lists.bufferbloat.net/pipermail/cake/2018-August/004285.html



On Tue, Mar 24, 2020 at 10:01 PM Aaron Wood <woody77@gmail.com> wrote:

> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>
> Flent test results are here:
>
> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>
> tl/dr;  1000ms of upstream bufferbloat
>
> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS
> 3.0 upstream mode based on the status LEDs.  Hopefully it will go away if I
> can convince it to run in DOCSIS 3.1 mode.
>
> At the moment, however, my WRT1900AC isn't up to the task of dealing with
> these sorts of downstream rates.
>
> So I'm looking at the apu2, which from this post:
>
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>
> Will certainly get most of the way there.
>
> Although
>

[-- Attachment #2: Type: text/html, Size: 2176 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Matt Taggart]

On 3/24/20 10:01 PM, Aaron Wood wrote:
[snip]
> At the moment, however, my WRT1900AC isn't up to the task of dealing 
> with these sorts of downstream rates.
> 
> So I'm looking at the apu2, which from this post:
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724

I recently got CenturyLink gig fiber and bought one of these:

Qotom Q355G4
https://www.amazon.com/gp/product/B077ZWR8Q9

And it's running OpenWRT 19.07 just fine, boots from a small USB thumbdrive.
It has no problem with CAKE + piece_of_cake up to 1gbit

Here is a table I made comparing the Qotom models

https://we.riseup.net/lackof/x86-router-candidates#qotom

(compiled last December, so prices may have changed)

-- 
Matt Taggart
matt@lackof.org

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

So, for higher bandwidth plans people started using raspberry pi4bs with an additional usb3 Ethernet dongle. Its WiFi is not really up for the task but it does seem to make a mean wired only router, the quad A76 cores seem to be capable to reliably shape up to 1 gigabit with cpu cycles to spare.
So maybe get one of those and change your old wifi router into a wifi AP?

Disclaimer, I have not tried that myself, as I am on a 100/40 plan and already have a router well capable of that speed.

Best Regards
        Sebastian

On March 25, 2020 6:29:17 AM GMT+01:00, Matt Taggart <matt@lackof.org> wrote:
>On 3/24/20 10:01 PM, Aaron Wood wrote:
>[snip]
>> At the moment, however, my WRT1900AC isn't up to the task of dealing 
>> with these sorts of downstream rates.
>> 
>> So I'm looking at the apu2, which from this post:
>>
>https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>
>I recently got CenturyLink gig fiber and bought one of these:
>
>Qotom Q355G4
>https://www.amazon.com/gp/product/B077ZWR8Q9
>
>And it's running OpenWRT 19.07 just fine, boots from a small USB
>thumbdrive.
>It has no problem with CAKE + piece_of_cake up to 1gbit
>
>Here is a table I made comparing the Qotom models
>
>https://we.riseup.net/lackof/x86-router-candidates#qotom
>
>(compiled last December, so prices may have changed)
>
>-- 
>Matt Taggart
>matt@lackof.org
>_______________________________________________
>Bloat mailing list
>Bloat@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/bloat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 2129 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Toke Høiland-Jørgensen]

Aaron Wood <woody77@gmail.com> writes:

> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>
> Flent test results are here:
> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>
> tl/dr;  1000ms of upstream bufferbloat
>
> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
> upstream mode based on the status LEDs.  Hopefully it will go away if I can
> convince it to run in DOCSIS 3.1 mode.

I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
ISP still has to turn it on? So maybe yelling at them will work? (ha!)

> At the moment, however, my WRT1900AC isn't up to the task of dealing with
> these sorts of downstream rates.
>
> So I'm looking at the apu2, which from this post:
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>
> Will certainly get most of the way there.

My Turris Omnia is doing fine on my 1Gbps connection (although that
hardly suffers from bloat, so I'm not doing any shaping; did try it
though, and it has no problem with running CAKE at 1Gbps).

-Toke

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Sebastian Moeller]

Hi Toke,


> On Mar 25, 2020, at 09:58, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Aaron Wood <woody77@gmail.com> writes:
> 
>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>> 
>> Flent test results are here:
>> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>> 
>> tl/dr;  1000ms of upstream bufferbloat
>> 
>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
>> upstream mode based on the status LEDs.  Hopefully it will go away if I can
>> convince it to run in DOCSIS 3.1 mode.
> 
> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
> 
>> At the moment, however, my WRT1900AC isn't up to the task of dealing with
>> these sorts of downstream rates.
>> 
>> So I'm looking at the apu2, which from this post:
>> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>> 
>> Will certainly get most of the way there.
> 
> My Turris Omnia is doing fine on my 1Gbps connection (although that
> hardly suffers from bloat, so I'm not doing any shaping; did try it
> though, and it has no problem with running CAKE at 1Gbps).

	Well, doing local network flent RRUL stress tests indicated that my omnia (at that time with TOS4/Openwrt18) only allowed up to 500/500 Mbps shaping with bi directionally saturating traffic with full MTU-sized packets. So I undirectional CAKE at 1Gbps can work, but under full load, I did not manage that, what did I wrong?

Best Regards
	Sebastian

> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
>
>> On Mar 25, 2020, at 09:58, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Aaron Wood <woody77@gmail.com> writes:
>> 
>>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
>>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>>> 
>>> Flent test results are here:
>>> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>>> 
>>> tl/dr;  1000ms of upstream bufferbloat
>>> 
>>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
>>> upstream mode based on the status LEDs.  Hopefully it will go away if I can
>>> convince it to run in DOCSIS 3.1 mode.
>> 
>> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
>> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
>> 
>>> At the moment, however, my WRT1900AC isn't up to the task of dealing with
>>> these sorts of downstream rates.
>>> 
>>> So I'm looking at the apu2, which from this post:
>>> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>>> 
>>> Will certainly get most of the way there.
>> 
>> My Turris Omnia is doing fine on my 1Gbps connection (although that
>> hardly suffers from bloat, so I'm not doing any shaping; did try it
>> though, and it has no problem with running CAKE at 1Gbps).
>
> 	Well, doing local network flent RRUL stress tests indicated that
> 	my omnia (at that time with TOS4/Openwrt18) only allowed up to
> 	500/500 Mbps shaping with bi directionally saturating traffic
> 	with full MTU-sized packets. So I undirectional CAKE at 1Gbps
> 	can work, but under full load, I did not manage that, what did I
> 	wrong?

Hmm, not sure I've actually done full bidirectional shaping. And trying
it now, it does seem to be struggling...

-Toke

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

>
> >>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in
> DOCSIS 3.0
> >>> upstream mode based on the status LEDs.  Hopefully it will go away if
> I can
> >>> convince it to run in DOCSIS 3.1 mode.
> >>
> >> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
> >> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
>

I've chatted with someone about it, and they seemed to think it's
suspicious, but I'm not going to push it further until I have modem showing
that it's in DOCSIS 3.1 for upstream.

I do need to see if I can sort out what the SB8200's status messages are:

CM-STATUS message sent. Event Type Code: 24; Chan ID: 48; DSID: N/A; MAC
Addr: N/A; OFDM/OFDMA Profile ID: 2
3.;CM-MAC=xx:xx:xx:xx:xx:xx;CMTS-MAC=xx:xx:xx:xx:xx:xx;CM-QOS=1.1;CM-VER=3.1;


> >>> At the moment, however, my WRT1900AC isn't up to the task of dealing
> with
> >>> these sorts of downstream rates.
> >>>
> >>> So I'm looking at the apu2, which from this post:
> >>>
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
> >>>
> >>> Will certainly get most of the way there.
> >>
> >> My Turris Omnia is doing fine on my 1Gbps connection (although that
> >> hardly suffers from bloat, so I'm not doing any shaping; did try it
> >> though, and it has no problem with running CAKE at 1Gbps).
> >
> >       Well, doing local network flent RRUL stress tests indicated that
> >       my omnia (at that time with TOS4/Openwrt18) only allowed up to
> >       500/500 Mbps shaping with bi directionally saturating traffic
> >       with full MTU-sized packets. So I undirectional CAKE at 1Gbps
> >       can work, but under full load, I did not manage that, what did I
> >       wrong?
>
> Hmm, not sure I've actually done full bidirectional shaping. And trying
> it now, it does seem to be struggling...


That's definitely an option for me, as I don't have to worry about a 2Gbps
total traffic, only about 1.03Gbps (since cable is so asymmetric).

But I'm also not sure I want to go with another ARM box.  The small x64
boxes are looking like a much better long-term option.

[-- Attachment #2: Type: text/html, Size: 3216 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

On Tue, Mar 24, 2020 at 11:19 PM Sebastian Moeller <moeller0@gmx.de> wrote:

> So, for higher bandwidth plans people started using raspberry pi4bs with
> an additional usb3 Ethernet dongle. Its WiFi is not really up for the task
> but it does seem to make a mean wired only router, the quad A76 cores seem
> to be capable to reliably shape up to 1 gigabit with cpu cycles to spare.
> So maybe get one of those and change your old wifi router into a wifi AP?
>

Using the existing router as an AP is my plan, so this would definitely
work (I'm looking to use this as an excuse to split the router and the APs
up).


> On March 25, 2020 6:29:17 AM GMT+01:00, Matt Taggart <matt@lackof.org>
> wrote:
>>
>> On 3/24/20 10:01 PM, Aaron Wood wrote:
>>
>> I recently got CenturyLink gig fiber and bought one of these:
>>
>> Qotom Q355G4
>> https://www.amazon.com/gp/product/B077ZWR8Q9
>>
>> And it's running OpenWRT 19.07 just fine, boots from a small USB thumbdrive.
>> It has no problem with CAKE + piece_of_cake up to 1gbit
>>
>> Here is a table I made comparing the Qotom models
>>
>> https://we.riseup.net/lackof/x86-router-candidates#qotom
>>
>>
Thanks!  I'll definitely look into those as well.

[-- Attachment #2: Type: text/html, Size: 2185 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

One other thought I've had with this, is that the apu2 is multi-core, and
the i210 is multi-queue.

Cake/htb aren't, iirc, setup to run on multiple cores (as the rate limiters
then don't talk to each other).  But with the correct tuple hashing in the
i210, I _should_ be able to split things and do two cores at 500Mbps each
(with lots of compute left over).

Obviously, that puts a limit on single-connection rates, but as the number
of connections climb, they should more or less even out (I remember Dave
Taht showing the oddities that happen with say 4 streams and 2 cores, where
it's common to end up with 3 streams on the same core).  But assuming that
the hashing function results in even sharing of streams, it should be
fairly balanced (after plotting some binomial distributions with higher "n"
values).  Still not perfect, especially since streams aren't likely to all
be elephants.

On Wed, Mar 25, 2020 at 4:03 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Sebastian Moeller <moeller0@gmx.de> writes:
>
> > Hi Toke,
> >
> >
> >> On Mar 25, 2020, at 09:58, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >>
> >> Aaron Wood <woody77@gmail.com> writes:
> >>
> >>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
> >>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
> >>>
> >>> Flent test results are here:
> >>>
> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
> >>>
> >>> tl/dr;  1000ms of upstream bufferbloat
> >>>
> >>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in
> DOCSIS 3.0
> >>> upstream mode based on the status LEDs.  Hopefully it will go away if
> I can
> >>> convince it to run in DOCSIS 3.1 mode.
> >>
> >> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
> >> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
> >>
> >>> At the moment, however, my WRT1900AC isn't up to the task of dealing
> with
> >>> these sorts of downstream rates.
> >>>
> >>> So I'm looking at the apu2, which from this post:
> >>>
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
> >>>
> >>> Will certainly get most of the way there.
> >>
> >> My Turris Omnia is doing fine on my 1Gbps connection (although that
> >> hardly suffers from bloat, so I'm not doing any shaping; did try it
> >> though, and it has no problem with running CAKE at 1Gbps).
> >
> >       Well, doing local network flent RRUL stress tests indicated that
> >       my omnia (at that time with TOS4/Openwrt18) only allowed up to
> >       500/500 Mbps shaping with bi directionally saturating traffic
> >       with full MTU-sized packets. So I undirectional CAKE at 1Gbps
> >       can work, but under full load, I did not manage that, what did I
> >       wrong?
>
> Hmm, not sure I've actually done full bidirectional shaping. And trying
> it now, it does seem to be struggling...
>
> -Toke
>

[-- Attachment #2: Type: text/html, Size: 4313 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Birdies have told me that it is possible for DOCSIS 3.1 modems to be
running in 3.0 mode.

Bitch at your ISP.
                                              - Jim

On Wed, Mar 25, 2020 at 1:01 AM Aaron Wood <woody77@gmail.com> wrote:

> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>
> Flent test results are here:
>
> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>
> tl/dr;  1000ms of upstream bufferbloat
>
> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS
> 3.0 upstream mode based on the status LEDs.  Hopefully it will go away if I
> can convince it to run in DOCSIS 3.1 mode.
>
> At the moment, however, my WRT1900AC isn't up to the task of dealing with
> these sorts of downstream rates.
>
> So I'm looking at the apu2, which from this post:
>
> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>
> Will certainly get most of the way there.
>
> Although
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 2448 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Dave Taht]

On Wed, Mar 25, 2020 at 8:58 AM Aaron Wood <woody77@gmail.com> wrote:
>
> One other thought I've had with this, is that the apu2 is multi-core, and the i210 is multi-queue.
>
> Cake/htb aren't, iirc, setup to run on multiple cores (as the rate limiters then don't talk to each other).  But with the correct tuple hashing in the i210, I _should_ be able to split things and do two cores at 500Mbps each (with lots of compute left over).
>
> Obviously, that puts a limit on single-connection rates, but as the number of connections climb, they should more or less even out (I remember Dave Taht showing the oddities that happen with say 4 streams and 2 cores, where it's common to end up with 3 streams on the same core).  But assuming that the hashing function results in even sharing of streams, it should be fairly balanced (after plotting some binomial distributions with higher "n" values).  Still not perfect, especially since streams aren't likely to all be elephants.

We live with imperfect per core tcp flow behavior already.

What I wanted to happen was the "list" ingress improvement to become
more generally available ( I can't find the lwn link at the moment).
It has. I thought that then we could express a syntax of tc qdisc add
dev eth0 ingress cake-mq bandwidth whatever, and it would rock.

I figured getting rid of the cost of the existing ifb and tc mirred,
and having a fast path preserving each hardware queue, then using
rcu to do a sloppy allocate atomic lock for shaped bandwidth and merge
every ms or so might be then low-cost enough. Certainly folding
everything into a single queue has a cost!

I was (before money ran out) prototyping adding a shared shaper to mq
at one point (no rcu, just  There have been so many other things
toss around (bpf?)

As for load balancing better, google "RSS++", if you must.


> On Wed, Mar 25, 2020 at 4:03 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>>
>> > Hi Toke,
>> >
>> >
>> >> On Mar 25, 2020, at 09:58, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> >>
>> >> Aaron Wood <woody77@gmail.com> writes:
>> >>
>> >>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
>> >>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>> >>>
>> >>> Flent test results are here:
>> >>> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>> >>>
>> >>> tl/dr;  1000ms of upstream bufferbloat
>> >>>
>> >>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
>> >>> upstream mode based on the status LEDs.  Hopefully it will go away if I can
>> >>> convince it to run in DOCSIS 3.1 mode.
>> >>
>> >> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
>> >> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
>> >>
>> >>> At the moment, however, my WRT1900AC isn't up to the task of dealing with
>> >>> these sorts of downstream rates.
>> >>>
>> >>> So I'm looking at the apu2, which from this post:
>> >>> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>> >>>
>> >>> Will certainly get most of the way there.
>> >>
>> >> My Turris Omnia is doing fine on my 1Gbps connection (although that
>> >> hardly suffers from bloat, so I'm not doing any shaping; did try it
>> >> though, and it has no problem with running CAKE at 1Gbps).
>> >
>> >       Well, doing local network flent RRUL stress tests indicated that
>> >       my omnia (at that time with TOS4/Openwrt18) only allowed up to
>> >       500/500 Mbps shaping with bi directionally saturating traffic
>> >       with full MTU-sized packets. So I undirectional CAKE at 1Gbps
>> >       can work, but under full load, I did not manage that, what did I
>> >       wrong?
>>
>> Hmm, not sure I've actually done full bidirectional shaping. And trying
>> it now, it does seem to be struggling...
>>
>> -Toke
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



--
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 4136 bytes --]

On Wed, Mar 25, 2020 at 12:18 PM Dave Taht <dave.taht@gmail.com> wrote:

> On Wed, Mar 25, 2020 at 8:58 AM Aaron Wood <woody77@gmail.com> wrote:
> >
> > One other thought I've had with this, is that the apu2 is multi-core,
> and the i210 is multi-queue.
> >
> > Cake/htb aren't, iirc, setup to run on multiple cores (as the rate
> limiters then don't talk to each other).  But with the correct tuple
> hashing in the i210, I _should_ be able to split things and do two cores at
> 500Mbps each (with lots of compute left over).
> >
> > Obviously, that puts a limit on single-connection rates, but as the
> number of connections climb, they should more or less even out (I remember
> Dave Taht showing the oddities that happen with say 4 streams and 2 cores,
> where it's common to end up with 3 streams on the same core).  But assuming
> that the hashing function results in even sharing of streams, it should be
> fairly balanced (after plotting some binomial distributions with higher "n"
> values).  Still not perfect, especially since streams aren't likely to all
> be elephants.
>
> We live with imperfect per core tcp flow behavior already.
>

Do you think this idea would make it worse, or better?  (I couldn't tell
from your comment how, exactly, you meant that)

OTOH, any gains I'd get over 500Mbps would just be gravy, as my current
router can't do more than that downstream on a single core, so even if the
sharing I have is uneven, it's better than what I have now (~400Mbps and
not pretty), so even if all fat streams landed on one core (unlikely
starting at 4+ streams), so I think I'd see overall gains (in my situation,
others might not).


> What I wanted to happen was the "list" ingress improvement to become
> more generally available ( I can't find the lwn link at the moment).
> It has. I thought that then we could express a syntax of tc qdisc add
> dev eth0 ingress cake-mq bandwidth whatever, and it would rock.
>
> I figured getting rid of the cost of the existing ifb and tc mirred,
> and having a fast path preserving each hardware queue, then using
> rcu to do a sloppy allocate atomic lock for shaped bandwidth and merge
> every ms or so might be then low-cost enough. Certainly folding
> everything into a single queue has a cost!
>

Sharing the tracked state by each cake-mq "thread" and updating it every so
often?

Or doing the rate limiting in one core, and the fq'ing on another? (I don't
think this is what you meant?)


> I was (before money ran out) prototyping adding a shared shaper to mq
> at one point (no rcu, just  There have been so many other things
> toss around (bpf?)
>
> As for load balancing better, google "RSS++", if you must.


I few years ago (before my current job ate all my brain cycles), I was
toying around with taking the ideas from OpenFlow/OpenVSwitch and RSS and
using them for parallelizing tasks like this:

- have N worker threads (say N=real cores, or real cores-1, or somesuch),
each fed by RSS / RPS / multiqueue etc.
- have a single controller thread (like the OpenFlow "controller")

Each worker publishes state/observations to the controller, as well as
forwarding "decisions to make", while the controller publishes each
worker's operating parameters to them individually.

The workers then just move packets as fast as they can, using their simple
rules, with no shared state between the workers, or need to access global
tables like connection tracking (e.g. NAT tables to map NAT'd tuples to lan
address tuples).

The controller deals with the decisions and the balancing of params (such
as dynamic configuration of the policer to keep things "fair").

I never got much farther than sketches on paper, and laying out how I'd do
it in a heavily multi-threaded userspace app (workers would use select() to
receive the control messages in-band, instead of needing to do shared
memory access).

I was also hoping that it would generalize to the hardware packet
accelerators, but I think to really take advantage of that, they would need
to be able to implement.

And, I never seem to have the time to try to stand up a rough framework for
this, to try it out...

[-- Attachment #2: Type: text/html, Size: 5370 bytes --]

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Dave Taht]

I just finished doing my first openwrt build in a couple years. (with
AQL) Trying to summon up the moxie to try it. Found my soldiering iron
and usb to serial interfaces....


On Wed, Mar 25, 2020 at 8:58 AM Aaron Wood <woody77@gmail.com> wrote:
>
> One other thought I've had with this, is that the apu2 is multi-core, and the i210 is multi-queue.
>
> Cake/htb aren't, iirc, setup to run on multiple cores (as the rate limiters then don't talk to each other).  But with the correct tuple hashing in the i210, I _should_ be able to split things and do two cores at 500Mbps each (with lots of compute left over).

A good test might be sch_mq + cake bandwidth whatever for each hw
queue. irqbalancing also may or may not help.

> Obviously, that puts a limit on single-connection rates, but as the number of connections climb, they should more or less even out (I remember Dave Taht showing the oddities that happen with say 4 streams and 2 cores, where it's common to end up with 3 streams on the same core).  But assuming that the hashing function results in even sharing of streams, it should be fairly balanced (after plotting some binomial distributions with higher "n" values).  Still not perfect, especially since streams aren't likely to all be elephants.

One reason why we are seeing "tcp rack" pushed so hard is due to cable
modems having multiple channels, and thus ooo packets are probable
when you try to push a stream across those channels.

Me, I'm reasonably confident we've hit the age of "peak bandwidth" for
most things at up/dl rates above 40Mbit.

And in the real world at home, a couple hash collissions and unequal
distribution really don't matter for real traffic.

>
> On Wed, Mar 25, 2020 at 4:03 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Sebastian Moeller <moeller0@gmx.de> writes:
>>
>> > Hi Toke,
>> >
>> >
>> >> On Mar 25, 2020, at 09:58, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> >>
>> >> Aaron Wood <woody77@gmail.com> writes:
>> >>
>> >>> I recently upgraded service from 150up, 10dn Mbps to xfinity's gigabit
>> >>> (with 35Mbps up) tier, and picked up a DOCSIS 3.1 modem to go with it.
>> >>>
>> >>> Flent test results are here:
>> >>> https://burntchrome.blogspot.com/2020/03/bufferbloat-with-comcast-gigabit-with.html
>> >>>
>> >>> tl/dr;  1000ms of upstream bufferbloat
>> >>>
>> >>> But it's DOCSIS 3.1, so why isn't PIE working?  Theory:  It's in DOCSIS 3.0
>> >>> upstream mode based on the status LEDs.  Hopefully it will go away if I can
>> >>> convince it to run in DOCSIS 3.1 mode.
>> >>
>> >> I think that while PIE is "mandatory to implement" in DOCSIS 3.1, the
>> >> ISP still has to turn it on? So maybe yelling at them will work? (ha!)
>> >>
>> >>> At the moment, however, my WRT1900AC isn't up to the task of dealing with
>> >>> these sorts of downstream rates.
>> >>>
>> >>> So I'm looking at the apu2, which from this post:
>> >>> https://forum.openwrt.org/t/comparative-throughput-testing-including-nat-sqm-wireguard-and-openvpn/44724
>> >>>
>> >>> Will certainly get most of the way there.
>> >>
>> >> My Turris Omnia is doing fine on my 1Gbps connection (although that
>> >> hardly suffers from bloat, so I'm not doing any shaping; did try it
>> >> though, and it has no problem with running CAKE at 1Gbps).
>> >
>> >       Well, doing local network flent RRUL stress tests indicated that
>> >       my omnia (at that time with TOS4/Openwrt18) only allowed up to
>> >       500/500 Mbps shaping with bi directionally saturating traffic
>> >       with full MTU-sized packets. So I undirectional CAKE at 1Gbps
>> >       can work, but under full load, I did not manage that, what did I
>> >       wrong?
>>
>> Hmm, not sure I've actually done full bidirectional shaping. And trying
>> it now, it does seem to be struggling...
>>
>> -Toke
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

On Sun, Mar 29, 2020 at 12:58 PM Dave Taht <dave.taht@gmail.com> wrote:

> I just finished doing my first openwrt build in a couple years. (with
> AQL) Trying to summon up the moxie to try it. Found my soldiering iron
> and usb to serial interfaces....
>

That's kept me from rolling my own...  I have the interfaces, but not the
energy to deal with the troubleshooting.  I think I still have an old
WNDR3700 in a box somewhere that I could prep as a backup, but I'd rather
not go through the hassle.


> On Wed, Mar 25, 2020 at 8:58 AM Aaron Wood <woody77@gmail.com> wrote:
> >
> > One other thought I've had with this, is that the apu2 is multi-core,
> and the i210 is multi-queue.
> >
> > Cake/htb aren't, iirc, setup to run on multiple cores (as the rate
> limiters then don't talk to each other).  But with the correct tuple
> hashing in the i210, I _should_ be able to split things and do two cores at
> 500Mbps each (with lots of compute left over).
>
> A good test might be sch_mq + cake bandwidth whatever for each hw
> queue. irqbalancing also may or may not help.
>

Bandwidth = 1Gbps or 500Mbps?  (I was thinking 500Mbps for that test setup).


> > Obviously, that puts a limit on single-connection rates, but as the
> number of connections climb, they should more or less even out (I remember
> Dave Taht showing the oddities that happen with say 4 streams and 2 cores,
> where it's common to end up with 3 streams on the same core).  But assuming
> that the hashing function results in even sharing of streams, it should be
> fairly balanced (after plotting some binomial distributions with higher "n"
> values).  Still not perfect, especially since streams aren't likely to all
> be elephants.
>
> One reason why we are seeing "tcp rack" pushed so hard is due to cable
> modems having multiple channels, and thus ooo packets are probable
> when you try to push a stream across those channels.
>

I don't know anything about the channels and how they're bonded.  separate
packets on each, or symbols that are spread across all the channels that
are used to construct a packet in less time..?  I'd expect to see more out
of order packets than I do, if they were using them all separately.  But
then none of the tests really do single-stream gigabit.


> Me, I'm reasonably confident we've hit the age of "peak bandwidth" for
> most things at up/dl rates above 40Mbit.
>

Very little of what I do gets to high (>50Mbps) rates.  But those that do,
I'm glad it's there.


> And in the real world at home, a couple hash collissions and unequal
> distribution really don't matter for real traffic.
>

[-- Attachment #2: Type: text/html, Size: 3960 bytes --]