<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I have a simpler setup now to remove some variables, both hosts are APU2 on Debian 9.6, kernel 4.9.0-8:</div><div class=""><br class=""></div><div class=""><div class="">apu2a (iperf3 client) <— default VLAN —> apu2b (iperf3 server)</div><div class=""><br class=""></div><div class="">Both have cake at 100mbit only on egress, with dual-srchost on client and dual-dsthost on server. With this setup (and probably previous ones, I just didn’t test it this way), bi-directional fairness with these flow counts works:</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP1 8-flow TCP up: 46.4<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP2 1-flow TCP up: 47.3<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP1 8-flow TCP down: 46.8<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP2 1-flow TCP down: 46.7</div></div><div class=""><br class=""></div><div class="">but with the original flow counts reported it’s still similarly imbalanced as before:</div><div class=""><br class=""></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP1 8-flow TCP up: 82.9<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP2 1-flow TCP up: 10.9<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP1 1-flow TCP down: 10.8<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>IP2 8-flow TCP down: 83.3</div></div><div class=""><br class=""></div><div class="">and now with ack-filter on both ends (not much change):</div><div class=""><br class=""></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow TCP up: 82.8<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP up: 10.9<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 1-flow TCP down: 10.5<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 8-flow TCP down: 83.2</div></div><div class=""><br class=""></div><div class=""><div class="">Before I go further, what I’m seeing with this rig is that when “interplanetary” is used and the number of iperf3 TCP flows goes above the number of CPUs minus one (in my case, 4 cores), the UDP send rate starts dropping. This only happens with interplanetary for some reason, but such as it is, I’m changed my tests to pit 8 UDP flows against 1 TCP flow instead, giving the UDP senders more CPU, as this seems to work much better. All tests except the last are with “interplanetary”.</div><div class=""><br class=""></div><div class="">UDP upload competition (looks good):</div></div><div class=""><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 1-flow TCP up: 48.6</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 8-flow UDP 48-mbit up: 48.2 (0% loss)</div><div class=""><br class=""></div></div><div class=""><div class="">UDP download competition (some imbalance, maybe a difference in how iperf3 reverse mode works?):</div><div class=""><br class=""></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow UDP 48-mbit down: 43.1 (0% loss)</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP down: 53.4 (0% loss)</div></div><div class=""><br class=""></div><div class="">All four at once (looks similar to previous two tests not impacting one another, which is good):</div></div><div class=""><br class=""></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 1-flow TCP up: 47.7</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 8-flow UDP 48-mbit up: 48.2 (0% loss)</div></div><div class=""><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow UDP 48-mbit down: 43.3 (0% loss)</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP down: 52.3</div></div></div><div class=""><br class=""></div><div class="">All four at once, up IPs flipped (less fair):</div><div class=""><br class=""></div><div class=""><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow UDP 48-mbit up: 37.7 (0% loss)</div><div class=""></div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP up: 57.9</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow UDP 48-mbit down: 38.9 (0% loss)</div></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP down: 56.3</div></div></div><div class=""><br class=""></div><div class="">All four at once, interplanetary off again, to double check it, and yes, UDP gets punished in this case:</div><div class=""><br class=""></div><div class=""><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 1-flow TCP up: 60.6</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 8-flow UDP 48-mbit up: 6.7 (86% loss)</div></div><div class=""><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP1 8-flow UDP 48-mbit down: 2.9 (94% loss)</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>IP2 1-flow TCP down: 63.1</div></div></div><div class=""><br class=""></div><div class="">So have we learned something from this? Yes, fairness is improved when using UDP instead of TCP for the 8-flow clients, but by turning AQM off we’re also testing a very different scenario, one that’s not too realistic. Does this prove the cause of the problem is TCP ack traffic?</div><div class=""><br class=""></div><div class="">Thanks again for the help on this. After a whole day on it, I’ll have to shift gears tomorrow to FreeNet router changes. I’ll show them the progress on Monday so of course I’d like to have a great host fairness story for Cake, as this is one of the main reasons to use it instead of fq_codel, but perhaps this will get sorted out before then. :)</div><div class=""><br class=""></div><div class="">I agree with George that we’ve been through this before, and also with how he explained it in his latest email, but there have been many changes to Cake since we tested in 2017, so this could be a regression. I’m almost sure I tested this exact scenario, and would not have put 8 up / 8 down on one IP and 1 up / 1 down on the other, which works with fairness for some reason.</div><div class=""><br class=""></div><div class="">FWIW, I also reproduced it in flent between the same APU2s used above, to be sure iperf3 wasn’t somehow causing it:</div><div class=""><br class=""></div><div class=""><a href="https://www.heistp.net/downloads/fairness_8_1/" class="">https://www.heistp.net/downloads/fairness_8_1/</a></div><div class=""><br class=""></div></body></html>