[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Jesper Dangaard Brouer]

On Sat, 07 Nov 2020 13:37:01 +0100
"Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:

> On 6 Nov 2020, at 21:19, Jesper Dangaard Brouer wrote:
> 
> > On Fri, 06 Nov 2020 18:04:49 +0100
> > "Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:
> >  
> >> On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote:
> >>
> >>
[...]
> >  
> >>> Have you tried to use 'perf record' to observe that is happening on
> >>> the system while these latency incidents happen?  (let me know if 
> >>> you want some cmdline hints)  
> >>
> >> Haven't tried this yet. If you have some hints what events to monitor
> >> I'll take them!  
> >
> > Okay to record everything (-a) on the system and save call-graph (-g),
> > and run for 5 seconds (via profiling the sleep function).
> >
> >  # perf record -g -a  sleep 5
> >
> > To view the result the simply use the 'perf report', but likely you
> > want to use option --no-children as you are profiling the kernel (and
> > not a userspace program you want to have grouped 'children' by).  I
> > also include the CPU column via '--sort cpu,comm,dso,symbol' and you
> > can select/zoom-in-on a specific CPU via '-C zero-indexed-cpu-num'.
> >
> >  # perf report --sort cpu,comm,dso,symbol --no-children
> >
> > When we ask you to provide the output, you can use the --stdio option,
> > and provide txt-info via a pastebin link as it is very long.  
> 
> Here is the output from kernel 3.10_1127 (I updated to the really newest 
> in that branch):  https://pastebin.com/5mxirXPw
> Here is the output from kernel 5.9.4: https://pastebin.com/KDZ2Ei2F

Did the latency issue happen during this this perf recording?
(it is obviously important that you record during the issue)

Finding a single latency event/issue in perf.data is like looking in
for needle in a haystack.  It does give hints and e.g. tell me details
like what part of the kernel are used/activated.

I can see that you have likely configured XPS (Transmit Packet
Steering). Is this on purpose?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer