From: "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com>
To: "Toke Høiland-Jørgensen" <toke@toke.dk>
Cc: "Jesper Dangaard Brouer" <brouer@redhat.com>,
Bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60
Date: Fri, 06 Nov 2020 13:01:38 +0100 [thread overview]
Message-ID: <56413531-311F-40CC-B78F-25A39D9F48BE@creamfinance.com> (raw)
In-Reply-To: <87blgaso84.fsf@toke.dk>
On 6 Nov 2020, at 12:45, Toke Høiland-Jørgensen wrote:
> "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com> writes:
>
>> On 6 Nov 2020, at 12:18, Jesper Dangaard Brouer wrote:
>>
>>> On Fri, 06 Nov 2020 10:18:10 +0100
>>> "Thomas Rosenstein" <thomas.rosenstein@creamfinance.com> wrote:
>>>
>>>>>> I just tested 5.9.4 seems to also fix it partly, I have long
>>>>>> stretches where it looks good, and then some increases again.
>>>>>> (3.10
>>>>>> Stock has them too, but not so high, rather 1-3 ms)
>>>>>>
>>>
>>> That you have long stretches where latency looks good is interesting
>>> information. My theory is that your system have a periodic
>>> userspace
>>> process that does a kernel syscall that takes too long, blocking
>>> network card from processing packets. (Note it can also be a kernel
>>> thread).
>>
>> The weird part is, I first only updated router-02 and pinged to
>> router-04 (out of traffic flow), there I noticed these long stretches
>> of
>> ok ping.
>>
>> When I updated also router-03 and router-04, the old behaviour kind
>> of
>> was back, this confused me.
>>
>> Could this be related to netlink? I have gobgpd running on these
>> routers, which injects routes via netlink.
>> But the churn rate during the tests is very minimal, maybe 30 - 40
>> routes every second.
>>
>> Otherwise we got: salt-minion, collectd, node_exporter, sshd
>
> collectd may be polling the interface stats; try turning that off?
I can, but shouldn't that also influence iperf3 performance then?
>
>>>
>>> Another theory is the NIC HW does strange things, but it is not very
>>> likely. E.g. delaying the packets before generating the IRQ
>>> interrupt,
>>> which hide it from my IRQ-to-softirq latency tool.
>>>
>>> A question: What traffic control qdisc are you using on your system?
>>
>> kernel 4+ uses pfifo, but there's no dropped packets
>> I have also tested with fq_codel, same behaviour and also no
>> weirdness
>> in the packets queue itself
>>
>> kernel 3.10 uses mq, and for the vlan interfaces noqueue
>
> Do you mean that you only have a single pfifo qdisc on kernel 4+? Why
> is
> it not using mq?
oh, actually, I just noticed that's a remnant of the previous tests, I
had
net.core.default_qdisc = fq_codel
in the sysctl.conf... so disregard my previous wrong info
so all kernel by default look like that, mq + pfifo_fast:
qdisc noqueue 0: dev lo root refcnt 2
qdisc mq 0: dev eth0 root
qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth1 root
qdisc pfifo_fast 0: dev eth1 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth2 root
qdisc pfifo_fast 0: dev eth2 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth3 root
qdisc pfifo_fast 0: dev eth3 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth3 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc mq 0: dev eth4 root
qdisc pfifo_fast 0: dev eth4 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :9 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :a bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :b bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :c bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :d bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :e bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :f bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :10 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :11 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :12 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :13 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :14 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :15 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :16 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :17 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth4 parent :18 bands 3 priomap 1 2 2 2 1 2 0 0
1 1 1 1 1 1 1 1
qdisc noqueue 0: dev eth4.2300 root refcnt 2
qdisc noqueue 0: dev eth4.2503 root refcnt 2
>
> Was there anything in the ethtool stats?
no, just rx / tx increase, not a single err
>
> -Toke
next prev parent reply other threads:[~2020-11-06 12:01 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-04 15:23 Thomas Rosenstein
2020-11-04 16:10 ` Toke Høiland-Jørgensen
2020-11-04 16:24 ` Thomas Rosenstein
2020-11-05 0:10 ` Toke Høiland-Jørgensen
2020-11-05 8:48 ` Thomas Rosenstein
2020-11-05 11:21 ` Toke Høiland-Jørgensen
2020-11-05 12:22 ` Thomas Rosenstein
2020-11-05 12:38 ` Toke Høiland-Jørgensen
2020-11-05 12:41 ` Thomas Rosenstein
2020-11-05 12:47 ` Toke Høiland-Jørgensen
2020-11-05 13:33 ` Jesper Dangaard Brouer
2020-11-06 8:48 ` Thomas Rosenstein
2020-11-06 10:53 ` Jesper Dangaard Brouer
2020-11-06 9:18 ` Thomas Rosenstein
2020-11-06 11:18 ` Jesper Dangaard Brouer
2020-11-06 11:37 ` Thomas Rosenstein
2020-11-06 11:45 ` Toke Høiland-Jørgensen
2020-11-06 12:01 ` Thomas Rosenstein [this message]
2020-11-06 12:53 ` Jesper Dangaard Brouer
2020-11-06 14:13 ` Jesper Dangaard Brouer
2020-11-06 17:04 ` Thomas Rosenstein
2020-11-06 20:19 ` Jesper Dangaard Brouer
2020-11-07 12:37 ` Thomas Rosenstein
2020-11-07 12:40 ` Jan Ceuleers
2020-11-07 12:43 ` Thomas Rosenstein
2020-11-07 13:00 ` Thomas Rosenstein
2020-11-09 8:24 ` Jesper Dangaard Brouer
2020-11-09 10:09 ` Thomas Rosenstein
2020-11-09 11:40 ` Jesper Dangaard Brouer
2020-11-09 11:51 ` Toke Høiland-Jørgensen
2020-11-09 12:25 ` Thomas Rosenstein
2020-11-09 14:33 ` Thomas Rosenstein
2020-11-12 10:05 ` Jesper Dangaard Brouer
2020-11-12 11:26 ` Thomas Rosenstein
2020-11-12 13:31 ` Jesper Dangaard Brouer
2020-11-12 13:42 ` Thomas Rosenstein
2020-11-12 15:42 ` Jesper Dangaard Brouer
2020-11-13 6:31 ` Thomas Rosenstein
2020-11-16 11:56 ` Jesper Dangaard Brouer
2020-11-16 12:05 ` Thomas Rosenstein
2020-11-09 16:39 ` Thomas Rosenstein
2020-11-07 13:33 ` Thomas Rosenstein
2020-11-07 16:46 ` Jesper Dangaard Brouer
2020-11-07 17:01 ` Thomas Rosenstein
2020-11-07 17:26 ` Sebastian Moeller
2020-11-16 12:34 ` Jesper Dangaard Brouer
2020-11-16 12:49 ` Thomas Rosenstein
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56413531-311F-40CC-B78F-25A39D9F48BE@creamfinance.com \
--to=thomas.rosenstein@creamfinance.com \
--cc=bloat@lists.bufferbloat.net \
--cc=brouer@redhat.com \
--cc=toke@toke.dk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox