[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Thomas Rosenstein]

On 12 Nov 2020, at 14:31, Jesper Dangaard Brouer wrote:

> On Thu, 12 Nov 2020 12:26:20 +0100
> "Thomas Rosenstein" <thomas.rosenstein at creamfinance.com> wrote:
>
>>> Long-shot theory(2): The NIC driver IRQ interrupt coalesce
>>> adaptive-IRQ (adj via ethtool -C) is wrong. The reason it is wrong is
>>> because of TSO frames, due packet-to-byte ratio isn't taken into
>>> account.
>>
>> here are the coalese parameters: (are the same for 3.10 and 5.4.60)
>>
>> ethtool -c eth4
>> Coalesce parameters for eth4:
>> Adaptive RX: on  TX: on
>
> Notice "Adaptive" setting is on.  My long-shot theory(2) is that this
> adaptive algorithm in the driver code can guess wrong (due to not
> taking TSO into account) and cause issues for
>
> Try to turn this adaptive algorithm off:
>
>   ethtool -C eth4 adaptive-rx off adaptive-tx off
>
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>>
>> rx-usecs: 32
>
> When you run off "adaptive-rx" you will get 31250 interrupts/sec
> (calc 1/(32/10^6) = 31250).
>
>> rx-frames: 64
>
> You likely want to reduce this.
>
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 128
>
> These 128 frames also seems too large.
>
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
> [...]

I have now updated the settings to:

ethtool -c eth4
Coalesce parameters for eth4:
Adaptive RX: off  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 0
rx-frames: 32
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 16
tx-frames: 32
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

but it makes absolutely no difference, also discard keep coming as before.

>>>
>>> You can test theory(2) in two ways, via disabling TSO and via
>>> adjusting
>>> ethtool --show-coalesce|--coalesce (-c|-C).  The --coalesce change can
>>> change part of theory(1), the NIC wake-up/IRQ interval.
here's the current nic settings:

ethtool -k eth4
Features for eth4:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off
	tx-tcp6-segmentation: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

>
> What is the NIC ring size settings?
>
>  ethtool -g eth4
>

ethtool -g eth4
Ring parameters for eth4:
Pre-set maximums:
RX:		8192
RX Mini:	0
RX Jumbo:	0
TX:		8192
Current hardware settings:
RX:		1024
RX Mini:	0
RX Jumbo:	0
TX:		1024

> -- 
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer