<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">As another clue, the un-fairness goes away when ECN is enabled on the client (both results with ingress keyword in use):</div><div class=""><br class=""></div><div class=""><div class="">sysctl -w net.ipv4.tcp_ecn=2</div></div><div class=""><br class=""></div><div class="">IP1, 1 up: 46.7 Mbit<br class="">IP2, 16 up: 46.7 Mbit<br class="">IP1, 16 down: 41.0 Mbit<br class="">IP2, 1 down: 49.3 Mbit<br class=""></div><div class=""><br class=""></div><div class="">sysctl -w net.ipv4.tcp_ecn=1</div><div class=""><br class=""></div><div class="">IP1, 1 up: 47.0 Mbit<br class="">IP2, 16 up: 47.0 Mbit<br class="">IP1, 16 down: 46.1 Mbit<br class="">IP2, 1 down: 46.1 Mbit<br class=""><br class=""></div><div class="">Also, this is probably just academic, but here are pcaps of "IP1, 16 down", with and without “ingress” and ECN enabled, along with some throughput and RTT graphs of one of the 16 flows for each: <a href="https://www.heistp.net/downloads/ingress_fairness/" class="">https://www.heistp.net/downloads/ingress_fairness/</a></div><div class=""><div class=""><br class=""></div><div class="">I find the different TCP RTT “slots” interesting (see the rtt graphs, all times in ms):</div><div class=""><br class=""></div><div class="">no ingress: 4, 8.5, 13, 17</div><div class="">no ingress+ecn: 8, 12, 16 (a few outliers at 20)</div><div class="">ingress: 4, 8, 12, 16, 20</div><div class="">ingress+ecn: 8, 12, 16 (outliers at 4 and 20 later in test)</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Mar 3, 2019, at 8:19 AM, Pete Heist <<a href="mailto:pete@heistp.net" class="">pete@heistp.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Oops, sorry I didn’t notice this before but it’s the ingress keyword that makes the difference:<div class=""><br class=""></div><div class="">qdisc cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 </div><div class=""><br class=""></div><div class="">IP1, 1 up: 46.8 Mbit</div><div class="">IP2, 16 up: 46.8 Mbit<br class=""><div class=""><div class="">IP1, 16 down: 41.9 Mbit</div><div class=""><div class="">IP2, 1 down: 51.1 Mbit</div><div class=""><span style="text-align: right;" class=""><br class=""></span></div><div class=""><span style="text-align: right;" class="">qdisc cake 803a: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 </span></div></div><div class=""><span style="text-align: right;" class=""><br class=""></span></div><div class=""><div class="">IP1, 1 up: 46.6 Mbit</div><div class="">IP2, 16 up: 46.5 Mbit<br class=""><div class=""><div class="">IP1, 16 down: 46.4 Mbit</div><div class="">IP2, 1 down: 46.5 Mbit</div><div class=""><br class=""></div><div class="">Also, my setup is different in that I use cake as a leaf to either hfsc or htb. But, it’s adding the ingress keyword that causes the imbalance for me.</div></div></div></div><div class=""><br class=""></div><blockquote type="cite" class=""><div class="">On Mar 2, 2019, at 11:20 AM, Pete Heist <<a href="mailto:pete@heistp.net" class="">pete@heistp.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Great, thanks for trying it. That strongly suggests a problem with the kernel (or driver) I’m using. Feels better to know it works as it should in recent kernels though...<br class=""><br class=""><blockquote type="cite" class="">On Mar 2, 2019, at 5:47 AM, George Amanakis <<a href="mailto:gamanakis@gmail.com" class="">gamanakis@gmail.com</a>> wrote:<br class=""><br class="">On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:<br class=""><blockquote type="cite" class=""><br class="">I will setup a vlan and try again.<br class=""><br class=""></blockquote><br class="">I replicated Pete's VLAN setup, and I am getting fairness:<br class=""><br class="">IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server<br class=""><br class="">IP1, 1 up: 46.73 mbit/s<br class="">IP2, 32 up: 46.91<br class="">IP1, 32 down: 46.70<br class="">IP2, 1 down: 46.89<br class=""><br class="">qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit<br class="">diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark<br class="">rtt 100.0ms noatm overhead 4<br class=""><br class="">qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3<br class="">dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms<br class="">noatm overhead 4<br class=""><br class=""><br class=""><br class=""></blockquote><br class=""></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></body></html>