[Bloat] Router congestion, slow ping/ack times with kernel 5.4.60

[Jesper Dangaard Brouer]

On Sat, 07 Nov 2020 14:00:04 +0100
Thomas Rosenstein via Bloat <bloat at lists.bufferbloat.net> wrote:
 
> Here's an extract from the ethtool https://pastebin.com/cabpWGFz just in 
> case there's something hidden.

Yes, there is something hiding in the data from ethtool_stats.pl[1]:
(10G Mellanox Connect-X cards via 10G SPF+ DAC)

 stat:            1 (          1) <= outbound_pci_stalled_wr_events /sec
 stat:    339731557 (339,731,557) <= rx_buffer_passed_thres_phy /sec

I've not seen this counter 'rx_buffer_passed_thres_phy' before, looking
in the kernel driver code it is related to "rx_buffer_almost_full".
The numbers per second is excessive (but it be related to a driver bug
as it ends up reading "high" -> rx_buffer_almost_full_high in the
extended counters).

 stat:     29583661 ( 29,583,661) <= rx_bytes /sec
 stat:     30343677 ( 30,343,677) <= rx_bytes_phy /sec

You are receiving with 236 Mbit/s in 10Gbit/s link.  There is a
difference between what the OS sees (rx_bytes) and what the NIC
hardware sees (rx_bytes_phy) (diff approx 6Mbit/s).

 stat:        19552 (     19,552) <= rx_packets /sec
 stat:        19950 (     19,950) <= rx_packets_phy /sec

Above RX packet counters also indicated HW is seeing more packets that
OS is receiving.

Next counters is likely your problem:

 stat:          718 (        718) <= tx_global_pause /sec
 stat:       954035 (    954,035) <= tx_global_pause_duration /sec
 stat:          714 (        714) <= tx_pause_ctrl_phy /sec

It looks like you have enabled Ethernet Flow-Control, and something is
causing pause frames to be generated.  It seem strange that this happen
on a 10Gbit/s link with only 236 Mbit/s.

The TX byte counters are also very strange:

 stat:        26063 (     26,063) <= tx_bytes /sec
 stat:        71950 (     71,950) <= tx_bytes_phy /sec

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

[1] https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl

Strange size distribution:
 stat:     19922 (     19,922) <= rx_1519_to_2047_bytes_phy /sec
 stat:        14 (         14) <= rx_65_to_127_bytes_phy /sec