<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">As an update on this, I now suspect a problem with either the Ethernet hardware or (more likely) sky2 driver on ‘mbp’, my 2007 MBP that acts as Flent server and where I’m often using a qdisc. I should have looked at dmesg earlier, as there are log entries like this:</div><div class=""><br class=""></div><div class="">-----</div><div class="">[ 221.478753] eth0: hw csum failure<br class="">[ 221.478756] CPU: 1 PID: 1890 Comm: netserver Tainted: G W 4.8.0-37-generic #39-Ubuntu<br class="">[ 221.478757] Hardware name: Apple Inc. MacBookPro4,1/Mac-F42C89C8, BIOS MBP41.88Z.00C1.B03.0802271651 02/27/08<br class="">[ 221.478762] 0000000000000286 000000003844a735 ffff9c293fd03ba8 ffffffffb5c30e12<br class="">[ 221.478765] ffff9c293a505000 ffffffffb66fb5c0 ffff9c293fd03bc0 ffffffffb5f7c028<br class="">[ 221.478769] ffff9c29399ea800 ffff9c293fd03be0 ffffffffb5f71f26 af75267500000000<br class="">[ 221.478770] Call Trace:<br class="">[ 221.478775] <IRQ> [<ffffffffb5c30e12>] dump_stack+0x63/0x81<br class="">[ 221.478778] [<ffffffffb5f7c028>] netdev_rx_csum_fault+0x38/0x40<br class="">[ 221.478781] [<ffffffffb5f71f26>] __skb_checksum_complete+0xb6/0xc0</div><div class="">…</div><div class="">[ 226.478373] net_ratelimit: 386 callbacks suppressed<br class="">[ 226.478378] eth0: hw csum failure<br class="">[ 226.479523] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 4.8.0-37-generic #39-Ubuntu<br class="">[ 226.479527] Hardware name: Apple Inc. MacBookPro4,1/Mac-F42C89C8, BIOS MBP41.88Z.00C1.B03.0802271651 02/27/08<br class="">[ 226.479533] 0000000000000286 f78e43dca42a09d0 ffff9c293fd03b88 ffffffffb5c30e12<br class="">[ 226.479542] ffff9c293a505000 ffffffffb66fb5c0 ffff9c293fd03ba0 ffffffffb5f7c028<br class="">[ 226.479549] ffff9c2932093b00 ffff9c293fd03bc0 ffffffffb5f71f26 46898f6100000000<br class="">[ 226.479557] Call Trace:<br class="">[ 226.479560] <IRQ> [<ffffffffb5c30e12>] dump_stack+0x63/0x81<br class="">[ 226.479581] [<ffffffffb5f7c028>] netdev_rx_csum_fault+0x38/0x40</div><div class="">-----<br class=""><br class=""></div><div class="">What’s interesting is that they only occur during testing, and when QoS with rate limiting is applied (Cake or HTB+X also). It’s also interesting that they occur on exactly 5 second intervals, not every 5 seconds, but sometimes after 10, or 15 seconds, but on 5 second intervals. I went back and looked at my results, and realized that a very large number of the latency and throughput shifts I saw are also quantized to 5 second intervals. I don’t think that’s a coincidence.</div><div class=""><br class=""></div><div class="">I saw Dave posted something that he saw a similar 'hw csum failure' on raspi earlier in 2016:</div><div class=""><br class=""></div><div class=""><a href="https://github.com/raspberrypi/linux/issues/1371" class="">https://github.com/raspberrypi/linux/issues/1371</a></div><div class=""><br class=""></div><div class="">but since I’ve also seen more reports of this over the years with no clear solution.</div><div class=""><br class=""></div><div class="">Why I saw it more with Cake than other qdiscs I don’t know, but I think it’s safe to say there’s no point in you trying to reproduce this until I can get past this with my hardware, and also I’m likely going to have to do a re-run of all of my tests after this is sorted out.</div><div class=""><br class=""></div><div class="">Pete</div><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 10, 2017, at 1:21 PM, Pete Heist <<a href="mailto:peteheist@gmail.com" class="">peteheist@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Feb 10, 2017, at 12:35 PM, Sebastian Moeller <<a href="mailto:moeller0@gmx.de" class="">moeller0@gmx.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hi Pete,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">On Feb 10, 2017, at 12:08, Pete Heist <<a href="mailto:peteheist@gmail.com" class="">peteheist@gmail.com</a>> wrote:<br class=""><br class="">Not a problem. I’ll run a spread of Cake and fq_codel over Ethernet at various bandwidths. It will be through their Apple USB Ethernet adapters (used now for management), which are also connected through a switch, but I think that setup should be fine for this purpose. Should be done in a hour or so and we’ll see…<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span class="Apple-tab-span" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> </span><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I believe the Apple USB dongles are fastEthernet only, at least the USB2 types I have available here, which for your tested bandwidth would work, but it will not allow you test at what shaper rate things go pear shaped… Also it wifi creates a bit more CPU load than wired ethernet, it _might_ make sense to concurrently excercise the WIFI cards just to re-create the SIRQ load (but probably not as the first experiment ;) ).</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Best Regards</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span class="Apple-tab-span" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: pre; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> </span><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Sebastian<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote></div><br class=""><div class="">Hi Sebastian, yes, they’re only 100 Mbit, but that’s enough to cover the rates where I was seeing the problem with Wi-Fi. Also in my test setup there are four nodes connected as described under Configuration #1:</div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html" class="">http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html</a></div><div class=""><br class=""></div><div class="">I’m running Cake on ‘mini’ and ‘mbp’, and the Wi-Fi radios are only on ‘om1’ and ‘om2’, so the CPU load shouldn’t be different for mini and mbp when connected directly via Ethernet, instead of via Ethernet and a Wi-Fi link, I suppose.</div><div class=""><br class=""></div><div class="">I think we just wanted to see if the throughput shifting would reproduce over Ethernet at the same rates, but so far it didn’t for me, although there are other anomalies that don’t look like the throughput shifts I sent before (there’s a throughput anomaly for Cake 20Mbit and latency anomalies for fq_codel 60Mbit and 90Mbit):</div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_10mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_10mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_20mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_20mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_30mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_30mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_40mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_40mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_50mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_50mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_60mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_60mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_70mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_70mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_75mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_75mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_80mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_80mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_85mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_85mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_90mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_90mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/cake_hd-eth_100mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/cake_hd-eth_100mbit/index.html</a></div><div class=""><br class=""></div><div class="">fq_codel:</div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_10mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_10mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_20mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_20mbit/index.html</a><br class=""></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_30mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_30mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_40mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_40mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_50mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_50mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_60mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_60mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_70mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_70mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_75mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_75mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_80mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_80mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_85mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_85mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_90mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_90mbit/index.html</a></div><div class=""><br class=""></div><div class=""><a href="http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_100mbit/index.html" class="">http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth_100mbit/index.html</a></div><div class=""><br class=""></div><div class="">So that suggests that the throughput shifting problem may also be somehow related to Wi-Fi. I’m still going to be testing Chaos Calmer, as well as two Ubiquiti NanoStation M5’s, though this will take some more time. We might learn some more from this, or if you can reproduce it with ath9k hardware that would be good too...</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pete</div><div class=""><br class=""></div></div></div></blockquote></div><br class=""></body></html>