[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Jesper Dangaard Brouer]

On Thu, 12 Nov 2020 12:26:20 +0100
"Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:

> > Long-shot theory(2): The NIC driver IRQ interrupt coalesce
> > adaptive-IRQ (adj via ethtool -C) is wrong. The reason it is wrong is
> > because of TSO frames, due packet-to-byte ratio isn't taken into
> > account.  
> 
> here are the coalese parameters: (are the same for 3.10 and 5.4.60)
> 
> ethtool -c eth4
> Coalesce parameters for eth4:
> Adaptive RX: on  TX: on

Notice "Adaptive" setting is on.  My long-shot theory(2) is that this
adaptive algorithm in the driver code can guess wrong (due to not
taking TSO into account) and cause issues for 

Try to turn this adaptive algorithm off:

  ethtool -C eth4 adaptive-rx off adaptive-tx off

> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> 
> rx-usecs: 32

When you run off "adaptive-rx" you will get 31250 interrupts/sec
(calc 1/(32/10^6) = 31250).

> rx-frames: 64

You likely want to reduce this.

> rx-usecs-irq: 0
> rx-frames-irq: 0
> 
> tx-usecs: 8
> tx-frames: 128

These 128 frames also seems too large.

> tx-usecs-irq: 0
> tx-frames-irq: 0
> 
[...]
> >
> > You can test theory(2) in two ways, via disabling TSO and via 
> > adjusting
> > ethtool --show-coalesce|--coalesce (-c|-C).  The --coalesce change can
> > change part of theory(1), the NIC wake-up/IRQ interval.  

What is the NIC ring size settings?

 ethtool -g eth4

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer