[Bloat] Need Guidance on reducing bufferbloat on a Fedora-based router

[Bloat] Need Guidance on reducing bufferbloat on a Fedora-based router

[dag dg]

[-- Attachment #1: Type: text/plain, Size: 6162 bytes --]

Hey folks. I'm new to this list but I've been following bufferbloat.net's
initiatives for awhile. Some time ago I built a Fedora-based router using
desktop hardware:

AMD FX-8120 3.1GHz
16 GB RAM
Intel i350-t2v2 dual port NIC

and I've pretty much been fighting bufferbloat since I built it. About a
year ago I bumped into the sqm-scripts initiative and was able to get it
set up on Fedora and began to get much better bufferbloat results.

Recently with the Meltdown/Spectre incidents I've been doing some
diagnostics on said box and noticed that under the sqm-scripts config the
number of available queues on my uplink interface is reduced due to my
using the "simple" QoS script they provide:

[root@router ~]# tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc htb 1: dev enp2s0f0 root refcnt 9 r2q 10 default 12
direct_packets_stat 3 direct_qlen 1000
qdisc fq_codel 120: dev enp2s0f0 parent 1:12 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 130: dev enp2s0f0 parent 1:13 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 110: dev enp2s0f0 parent 1:11 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc ingress ffff: dev enp2s0f0 parent ffff:fff1 ----------------
qdisc mq 0: dev enp2s0f1 root
qdisc fq_codel 0: dev enp2s0f1 parent :8 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :7 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :6 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :5 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :4 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :3 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :1 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum
1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc htb 1: dev ifb4enp2s0f0 root refcnt 2 r2q 10 default 10
direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 110: dev ifb4enp2s0f0 parent 1:10 limit 1001p flows 1024
quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn

When I turn the sqm-scripts off:

[root@router ~]# tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev enp2s0f0 root
qdisc fq_codel 0: dev enp2s0f0 parent :8 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :7 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :6 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :5 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :4 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :3 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f0 parent :1 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc mq 0: dev enp2s0f1 root
qdisc fq_codel 0: dev enp2s0f1 parent :8 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :7 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :6 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :5 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :4 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :3 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev enp2s0f1 parent :1 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum
1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn

The i350 series NIC I'm using supports up to 8 Tx and Rx queues depending
on the number of cores the CPU has.

I've read up on the developments with cake however just as fq_codel took
awhile to move over to fedora so is cake taking awhile to become available.
I could compile cake from source but I'm a little nervous in gutting the
distribution's iproute2 in order to add cake support.

This hardware is super overkill for a home connection but I like to run a
lot of network diagnostic tools to monitor the health of the network that
just cripple pretty much any standard home routing hardware I use. As a
side note I also realize that being a bulldozer architecture the 8120 is
technically a 4 module chip; this was just the hardware I had available at
the time and I'm planning on moving over to a 10 core xeon chip(no
hyperthreading) in the future so there'd be 2 cores for the OS and 8 for
the NIC.

At this point I'm just looking for some guidance on steps to move forward,
any suggestions would be appreciated.

~dag

[-- Attachment #2: Type: text/html, Size: 8700 bytes --]

Re: [Bloat] Need Guidance on reducing bufferbloat on a Fedora-based router

[Toke Høiland-Jørgensen]

dag dg <dagofthedofg@gmail.com> writes:

> At this point I'm just looking for some guidance on steps to move
> forward, any suggestions would be appreciated.

From your email I'm not quite sure what problem it is you are trying to
solve? You say that sqm-scripts with fq_codel has gotten your
bufferbloat under control, so what is the problem, exactly? :)

-Toke

Re: [Bloat] Need Guidance on reducing bufferbloat on a Fedora-based router

[Toke Høiland-Jørgensen]

(Adding the bloat list back as CC)

> Well there are two things that are bothering me. The first is that I've
> gone from 8 queues down to just 3 excluding the ingress queue and I just
> can't shake this feeling that there's got to be a better way to go about
> things.

Well, software shaping interacts badly with multiple hardware queues. If
you need the software shaper (whether that is Cake or HTB), you can't
also have multiple independent hardware queues. The shaper needs global
state for the shaped link, so you would have to synchronise across
queues anyway, even if it was supported by the shaper.

Besides, on the box you're running, you probably don't need the large
number of hardware queues to push 1 Gbps... So unless you are seeing
a specific performance problem I wouldn't worry too much about it :)

-Toke