[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Jesper Dangaard Brouer]

On Fri, 06 Nov 2020 18:04:49 +0100
"Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:

> On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote:
> 
> > On Fri, 6 Nov 2020 13:53:58 +0100
> > Jesper Dangaard Brouer <brouer at redhat.com> wrote:
> >  
> >> [...]  
> >>>>
> >>>> Could this be related to netlink? I have gobgpd running on these
> >>>> routers, which injects routes via netlink.
> >>>> But the churn rate during the tests is very minimal, maybe 30 - 40
> >>>> routes every second.  
> >>
> >> Yes, this could be related.  The internal data-structure for FIB
> >> lookups is a fibtrie which is a compressed patricia tree, related to
> >> radix tree idea.  Thus, I can imagine that the kernel have to
> >> rebuild/rebalance the tree with all these updates.  
> >
> > Reading the kernel code. The IPv4 fib_trie code is very well tuned,
> > fully RCU-ified, meaning read-side is lock-free.  The resize() 
> > function code in net//ipv4/fib_trie.c have max_work limiter to avoid it uses 
> > too much time.  And the update looks lockfree.
> >
> > The IPv6 update looks more scary, as it seems to take a "bh" spinlock
> > that can block softirq from running code in net/ipv6/ip6_fib.c
> > (spin_lock_bh(&f6i->fib6_table->tb6_lock).  
> 
> I'm using ping on IPv4, but I'll try to see if IPv6 makes any 
> difference!

I think you misunderstand me.  I'm not asking you to use ping6. The
gobgpd daemon updates will both update IPv4 and IPv6 routes, right.
Updating IPv6 routes are more problematic than IPv4 routes.  The IPv6
route tables update can potentially stall softirq from running, which
was the latency tool was measuring... and it did show some outliers.


> > Have you tried to use 'perf record' to observe that is happening on 
> > the system while these latency incidents happen?  (let me know if you 
> > want some cmdline hints)  
> 
> Haven't tried this yet. If you have some hints what events to monitor 
> I'll take them!

Okay to record everything (-a) on the system and save call-graph (-g),
and run for 5 seconds (via profiling the sleep function).

 # perf record -g -a  sleep 5

To view the result the simply use the 'perf report', but likely you
want to use option --no-children as you are profiling the kernel (and
not a userspace program you want to have grouped 'children' by).  I
also include the CPU column via '--sort cpu,comm,dso,symbol' and you
can select/zoom-in-on a specific CPU via '-C zero-indexed-cpu-num'.

 # perf report --sort cpu,comm,dso,symbol --no-children

When we ask you to provide the output, you can use the --stdio option,
and provide txt-info via a pastebin link as it is very long.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer