[Cake] dual-src/dsthost unfairness, only with bi-directional traffic

[Pete Heist]

In my one-armed router setup I’m seeing host fairness work perfectly with srchost or dsthost, but with dual-srchost or dual-dsthost, host fairness deviates from the ideal, _only_ when there's bi-directional traffic. The deviation is then dependent on the number of flows. Is this expected?

I had thought that dual-src/dsthost worked the same as src/dsthost (fairness between hosts) with the exception that there is also fairness of flows within each host.

Here are some results (all rates aggregate throughput in Mbit):

IP1=8 up / 1 down   IP2=1 up / 8 down (post-test tc stats attached):
	srchost/dsthost, upload only: IP1=48.1, IP2=47.9  (OK)
	srchost/dsthost, download only: IP1=47.8, IP2=47.8  (OK)
	srchost/dsthost, bi-directional: IP1=47.5 up / 43.9 down, IP2=44.7 up / 46.7 down  (OK)

	dual-srchost/dual-dsthost, upload only: IP1=48.1, IP2=48.0  (OK)
	dual-srchost/dual-dsthost, download only: IP1=47.9, IP2=47.9  (OK)
	dual-srchost/dual-dsthost, bi-directional: IP1=83.0 up / 10.7 down, IP2=10.6 up / 83.0 down (*** asymmetric ***)

Dual-srchost/dual-dsthost, bi-directional tests with different flow counts:

IP1=4 up / 1 down   IP2=1 up / 4 down:
	IP1=74.8 up / 18.8 down, IP2=18.8 up / 74.8 down

IP1=2 up / 1 down   IP2=1 up / 2 down:
	IP1=62.4 up / 31.3 down, IP2=31.3 up / 62.4 down

IP1=4 up / 1 down   IP2=1 up / 8 down:
	IP1=81.8 up / 11.5 down, IP2=17.4 up / 76.3 down

IP1=2 up / 1 down   IP2=1 up / 8 down:
	IP1=79.9 up / 13.5 down, IP2=25.7 up / 68.1 down

The setup:

	apu2a (kernel 4.9)  <— default VLAN —>  apu1a (kernel 3.16.7)  <— VLAN 3300 —>  apu2b (kernel 4.9)

- apu1a is the router, and has cake only on egress of both eth0 and eth0.3300, rate limited to 100mbit for both
- it has no trouble shaping at 100mbit up and down simultaneously, so that should not be a problem
- the same problem occurs at 25mbit or 50mbit)
- since apu2a is the client [dual-]dsthost is used on eth0 and [dual-]srchost is used on eth0.3300
- the fairness test setup seems correct, based on the results of most of the tests, at least.
- note in the qdisc stats attached there is a prio qdisc on eth0 for filtering out VLAN traffic so it isn’t shaped twice
- I also get the exact same results with an htb or hfsc hierarchy on eth0 instead of adding a qdisc to eth0.3300
- printk’s in sch_cake.c shows values of flow_mode, srchost_hash and dsthost_hash as expected
- I also see it going into allocate_src and allocate_dst as expected, and later ending up in found_src and found_dst

I’m stumped. I know I’ve tested fairness of dual-src/dsthost before, but that was from the egress of client and server, and it was on a recent kernel. Time to sleep on it...

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: qdisc_stats.txt
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20190102/17923b45/attachment.txt>