[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Thomas Rosenstein]

On 6 Nov 2020, at 12:45, Toke Høiland-Jørgensen wrote:

> "Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> writes:
>
>> On 6 Nov 2020, at 12:18, Jesper Dangaard Brouer wrote:
>>
>>> On Fri, 06 Nov 2020 10:18:10 +0100
>>> "Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:
>>>
>>>>>> I just tested 5.9.4 seems to also fix it partly, I have long
>>>>>> stretches where it looks good, and then some increases again. 
>>>>>> (3.10
>>>>>> Stock has them too, but not so high, rather 1-3 ms)
>>>>>>
>>>
>>> That you have long stretches where latency looks good is interesting
>>> information.   My theory is that your system have a periodic 
>>> userspace
>>> process that does a kernel syscall that takes too long, blocking
>>> network card from processing packets. (Note it can also be a kernel
>>> thread).
>>
>> The weird part is, I first only updated router-02 and pinged to
>> router-04 (out of traffic flow), there I noticed these long stretches 
>> of
>> ok ping.
>>
>> When I updated also router-03 and router-04, the old behaviour kind 
>> of
>> was back, this confused me.
>>
>> Could this be related to netlink? I have gobgpd running on these
>> routers, which injects routes via netlink.
>> But the churn rate during the tests is very minimal, maybe 30 - 40
>> routes every second.
>>
>> Otherwise we got: salt-minion, collectd, node_exporter, sshd
>
> collectd may be polling the interface stats; try turning that off?

I can, but shouldn't that also influence iperf3 performance then?

>
>>>
>>> Another theory is the NIC HW does strange things, but it is not very
>>> likely.  E.g. delaying the packets before generating the IRQ
>>> interrupt,
>>> which hide it from my IRQ-to-softirq latency tool.
>>>
>>> A question: What traffic control qdisc are you using on your system?
>>
>> kernel 4+ uses pfifo, but there's no dropped packets
>> I have also tested with fq_codel, same behaviour and also no 
>> weirdness
>> in the packets queue itself
>>
>> kernel 3.10 uses mq, and for the vlan interfaces noqueue
>
> Do you mean that you only have a single pfifo qdisc on kernel 4+? Why 
> is
> it not using mq?

oh, actually, I just noticed that's a remnant of the previous tests, I 
had

net.core.default_qdisc = fq_codel

in the sysctl.conf... so disregard my previous wrong info



so all kernel by default look like that, mq + pfifo_fast:

qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth1 root
qdisc pfifo_fast 0: dev eth1 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth2 root
qdisc pfifo_fast 0: dev eth2 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth3 root
qdisc pfifo_fast 0: dev eth3 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth4 root
qdisc pfifo_fast 0: dev eth4 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :9 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :a bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :b bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :c bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :d bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :e bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :f bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :10 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :11 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :12 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :13 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :14 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :15 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :16 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :17 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :18 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
qdisc noqueue 0: dev eth4.2300 root refcnt 2
qdisc noqueue 0: dev eth4.2503 root refcnt 2


>
> Was there anything in the ethtool stats?

no, just rx / tx increase, not a single err

>
> -Toke