<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal">
<p dir="auto">On 9 Nov 2020, at 12:40, Jesper Dangaard Brouer wrote:</p>
<blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px">
<p dir="auto">On Mon, 09 Nov 2020 11:09:33 +0100<br>
"Thomas Rosenstein" <a href="mailto:thomas.rosenstein@creamfinance.com" style="color:#777">thomas.rosenstein@creamfinance.com</a> wrote:</p>
<p dir="auto">Could you also provide ethtool_stats for the TX interface?</p>
<p dir="auto">Notice that the tool[1] ethtool_stats.pl support monitoring several<br>
interfaces at the same time, e.g. run:</p>
<p dir="auto">ethtool_stats.pl --sec 3 --dev eth4 --dev ethTX</p>
<p dir="auto">And provide output as pastebin.</p>
</blockquote>
<p dir="auto">I have now also repeated the same test with 3.10, here are the ethtool outputs:</p>
<p dir="auto"><a href="https://drive.google.com/file/d/1c98MVV0JYl6Su6xZTpqwS7m-6OlbmAFp/view?usp=sharing" style="color:#3983C4">https://drive.google.com/file/d/1c98MVV0JYl6Su6xZTpqwS7m-6OlbmAFp/view?usp=sharing</a></p>
<p dir="auto">and the ping times:</p>
<p dir="auto"><a href="https://drive.google.com/file/d/1xhbGJHb5jUbPsee4frbx-c-uqh-7orXY/view?usp=sharing" style="color:#3983C4">https://drive.google.com/file/d/1xhbGJHb5jUbPsee4frbx-c-uqh-7orXY/view?usp=sharing</a></p>
<p dir="auto">Sadly the parameters we were looking at are not supported below 4.14.</p>
<p dir="auto">but I immediatly saw 1 thing very different:</p>
<p dir="auto">ethtool --statistics eth4 | grep discards<br>
rx_discards_phy: 0<br>
tx_discards_phy: 0</p>
<p dir="auto">if we check the ethtool output from 5.9.4 were have:</p>
<pre style="background-color:#F7F7F7; border-radius:5px 5px 5px 5px; margin-left:15px; margin-right:15px; max-width:90vw; overflow-x:auto; padding:5px" bgcolor="#F7F7F7"><code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0" bgcolor="#F7F7F7"> rx_discards_phy: 151793
</code></pre>
<p dir="auto">And also the outbound_pci_stalled_wr_events get more frequent the lower the total bandwidth / the higher the ping is.<br>
Logically there must be something blocking the the buffers, either they are not getting freed, or not rotated correctly, or processing is too slow.<br>
I would exclude the processing, simply based on 0% CPU load, and also that it doesn't happen in 3.10.<br>
Suspicious is also, that the issue only appears after a certain time of activity (maybe total traffic?!)</p>
</div>
</div>
</body>
</html>