[Cake] dual-src/dsthost unfairness, only with bi-directional traffic

[Georgios Amanakis]

I can reproduce this one to my surprise, too. 
I tested on my Comcast connection, with a WRT1900ACS, running openwrt
(r8082-95b3f8ec8d, 4.14.70), with two interfaces br-lan and eth0(wan).

IP1=1 up / 8 down    IP2=4 up / 4 down
	src/dst, bidir: IP1=0.88 /  8.44, IP2=0.66 / 7.75 (ok)
dualsrc/dualdst, bidir: IP1=0.27 / 10.56, IP2=1.41 / 6.42 (unfair)

No VLANs, no other schedulers on eth0 and br-lan apart from cake.



On Wed, 2019-01-02 at 00:04 +0100, Pete Heist wrote:
> In my one-armed router setup I’m seeing host fairness work perfectly
> with srchost or dsthost, but with dual-srchost or dual-dsthost, host
> fairness deviates from the ideal, _only_ when there's bi-directional
> traffic. The deviation is then dependent on the number of flows. Is
> this expected?
> 
> I had thought that dual-src/dsthost worked the same as src/dsthost
> (fairness between hosts) with the exception that there is also
> fairness of flows within each host.
> 
> Here are some results (all rates aggregate throughput in Mbit):
> 
> IP1=8 up / 1 down   IP2=1 up / 8 down (post-test tc stats attached):
> 	srchost/dsthost, upload only: IP1=48.1, IP2=47.9  (OK)
> 	srchost/dsthost, download only: IP1=47.8, IP2=47.8  (OK)
> 	srchost/dsthost, bi-directional: IP1=47.5 up / 43.9 down,
> IP2=44.7 up / 46.7 down  (OK)
> 
> 	dual-srchost/dual-dsthost, upload only: IP1=48.1,
> IP2=48.0  (OK)
> 	dual-srchost/dual-dsthost, download only: IP1=47.9,
> IP2=47.9  (OK)
> 	dual-srchost/dual-dsthost, bi-directional: IP1=83.0 up / 10.7
> down, IP2=10.6 up / 83.0 down (*** asymmetric ***)
> 
> Dual-srchost/dual-dsthost, bi-directional tests with different flow
> counts:
> 
> IP1=4 up / 1 down   IP2=1 up / 4 down:
> 	IP1=74.8 up / 18.8 down, IP2=18.8 up / 74.8 down
> 
> IP1=2 up / 1 down   IP2=1 up / 2 down:
> 	IP1=62.4 up / 31.3 down, IP2=31.3 up / 62.4 down
> 
> IP1=4 up / 1 down   IP2=1 up / 8 down:
> 	IP1=81.8 up / 11.5 down, IP2=17.4 up / 76.3 down
> 
> IP1=2 up / 1 down   IP2=1 up / 8 down:
> 	IP1=79.9 up / 13.5 down, IP2=25.7 up / 68.1 down
> 
> The setup:
> 
> 	apu2a (kernel 4.9)  <— default VLAN —>  apu1a (kernel
> 3.16.7)  <— VLAN 3300 —>  apu2b (kernel 4.9)
> 
> - apu1a is the router, and has cake only on egress of both eth0 and
> eth0.3300, rate limited to 100mbit for both
> - it has no trouble shaping at 100mbit up and down simultaneously, so
> that should not be a problem
> - the same problem occurs at 25mbit or 50mbit)
> - since apu2a is the client [dual-]dsthost is used on eth0 and [dual-
> ]srchost is used on eth0.3300
> - the fairness test setup seems correct, based on the results of most
> of the tests, at least.
> - note in the qdisc stats attached there is a prio qdisc on eth0 for
> filtering out VLAN traffic so it isn’t shaped twice
> - I also get the exact same results with an htb or hfsc hierarchy on
> eth0 instead of adding a qdisc to eth0.3300
> - printk’s in sch_cake.c shows values of flow_mode, srchost_hash and
> dsthost_hash as expected
> - I also see it going into allocate_src and allocate_dst as expected,
> and later ending up in found_src and found_dst
> 
> I’m stumped. I know I’ve tested fairness of dual-src/dsthost before,
> but that was from the egress of client and server, and it was on a
> recent kernel. Time to sleep on it...
> 
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake