From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Thomas Rosenstein via Bloat <bloat@lists.bufferbloat.net>
Cc: brouer@redhat.com,
Thomas Rosenstein <thomas.rosenstein@creamfinance.com>,
"Jan Ceuleers" <jan.ceuleers@gmail.com>,
Saeed Mahameed <saeed@kernel.org>,
Tariq Toukan <ttoukan.linux@gmail.com>,
kheib@redhat.com
Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60
Date: Mon, 9 Nov 2020 09:24:28 +0100 [thread overview]
Message-ID: <20201109092428.293104ea@carbon> (raw)
In-Reply-To: <12D28386-7C00-4A31-91E4-37083C1674F9@creamfinance.com>
On Sat, 07 Nov 2020 14:00:04 +0100
Thomas Rosenstein via Bloat <bloat@lists.bufferbloat.net> wrote:
> Here's an extract from the ethtool https://pastebin.com/cabpWGFz just in
> case there's something hidden.
Yes, there is something hiding in the data from ethtool_stats.pl[1]:
(10G Mellanox Connect-X cards via 10G SPF+ DAC)
stat: 1 ( 1) <= outbound_pci_stalled_wr_events /sec
stat: 339731557 (339,731,557) <= rx_buffer_passed_thres_phy /sec
I've not seen this counter 'rx_buffer_passed_thres_phy' before, looking
in the kernel driver code it is related to "rx_buffer_almost_full".
The numbers per second is excessive (but it be related to a driver bug
as it ends up reading "high" -> rx_buffer_almost_full_high in the
extended counters).
stat: 29583661 ( 29,583,661) <= rx_bytes /sec
stat: 30343677 ( 30,343,677) <= rx_bytes_phy /sec
You are receiving with 236 Mbit/s in 10Gbit/s link. There is a
difference between what the OS sees (rx_bytes) and what the NIC
hardware sees (rx_bytes_phy) (diff approx 6Mbit/s).
stat: 19552 ( 19,552) <= rx_packets /sec
stat: 19950 ( 19,950) <= rx_packets_phy /sec
Above RX packet counters also indicated HW is seeing more packets that
OS is receiving.
Next counters is likely your problem:
stat: 718 ( 718) <= tx_global_pause /sec
stat: 954035 ( 954,035) <= tx_global_pause_duration /sec
stat: 714 ( 714) <= tx_pause_ctrl_phy /sec
It looks like you have enabled Ethernet Flow-Control, and something is
causing pause frames to be generated. It seem strange that this happen
on a 10Gbit/s link with only 236 Mbit/s.
The TX byte counters are also very strange:
stat: 26063 ( 26,063) <= tx_bytes /sec
stat: 71950 ( 71,950) <= tx_bytes_phy /sec
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
[1] https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
Strange size distribution:
stat: 19922 ( 19,922) <= rx_1519_to_2047_bytes_phy /sec
stat: 14 ( 14) <= rx_65_to_127_bytes_phy /sec
next prev parent reply other threads:[~2020-11-09 8:24 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-04 15:23 Thomas Rosenstein
2020-11-04 16:10 ` Toke Høiland-Jørgensen
2020-11-04 16:24 ` Thomas Rosenstein
2020-11-05 0:10 ` Toke Høiland-Jørgensen
2020-11-05 8:48 ` Thomas Rosenstein
2020-11-05 11:21 ` Toke Høiland-Jørgensen
2020-11-05 12:22 ` Thomas Rosenstein
2020-11-05 12:38 ` Toke Høiland-Jørgensen
2020-11-05 12:41 ` Thomas Rosenstein
2020-11-05 12:47 ` Toke Høiland-Jørgensen
2020-11-05 13:33 ` Jesper Dangaard Brouer
2020-11-06 8:48 ` Thomas Rosenstein
2020-11-06 10:53 ` Jesper Dangaard Brouer
2020-11-06 9:18 ` Thomas Rosenstein
2020-11-06 11:18 ` Jesper Dangaard Brouer
2020-11-06 11:37 ` Thomas Rosenstein
2020-11-06 11:45 ` Toke Høiland-Jørgensen
2020-11-06 12:01 ` Thomas Rosenstein
2020-11-06 12:53 ` Jesper Dangaard Brouer
2020-11-06 14:13 ` Jesper Dangaard Brouer
2020-11-06 17:04 ` Thomas Rosenstein
2020-11-06 20:19 ` Jesper Dangaard Brouer
2020-11-07 12:37 ` Thomas Rosenstein
2020-11-07 12:40 ` Jan Ceuleers
2020-11-07 12:43 ` Thomas Rosenstein
2020-11-07 13:00 ` Thomas Rosenstein
2020-11-09 8:24 ` Jesper Dangaard Brouer [this message]
2020-11-09 10:09 ` Thomas Rosenstein
2020-11-09 11:40 ` Jesper Dangaard Brouer
2020-11-09 11:51 ` Toke Høiland-Jørgensen
2020-11-09 12:25 ` Thomas Rosenstein
2020-11-09 14:33 ` Thomas Rosenstein
2020-11-12 10:05 ` Jesper Dangaard Brouer
2020-11-12 11:26 ` Thomas Rosenstein
2020-11-12 13:31 ` Jesper Dangaard Brouer
2020-11-12 13:42 ` Thomas Rosenstein
2020-11-12 15:42 ` Jesper Dangaard Brouer
2020-11-13 6:31 ` Thomas Rosenstein
2020-11-16 11:56 ` Jesper Dangaard Brouer
2020-11-16 12:05 ` Thomas Rosenstein
2020-11-09 16:39 ` Thomas Rosenstein
2020-11-07 13:33 ` Thomas Rosenstein
2020-11-07 16:46 ` Jesper Dangaard Brouer
2020-11-07 17:01 ` Thomas Rosenstein
2020-11-07 17:26 ` Sebastian Moeller
2020-11-16 12:34 ` Jesper Dangaard Brouer
2020-11-16 12:49 ` Thomas Rosenstein
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201109092428.293104ea@carbon \
--to=brouer@redhat.com \
--cc=bloat@lists.bufferbloat.net \
--cc=jan.ceuleers@gmail.com \
--cc=kheib@redhat.com \
--cc=saeed@kernel.org \
--cc=thomas.rosenstein@creamfinance.com \
--cc=ttoukan.linux@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox