[Cake] dual-src/dsthost unfairness, only with bi-directional traffic

[Pete Heist <pete@heistp.net>]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

In my one-armed router setup I’m seeing host fairness work perfectly with srchost or dsthost, but with dual-srchost or dual-dsthost, host fairness deviates from the ideal, _only_ when there's bi-directional traffic. The deviation is then dependent on the number of flows. Is this expected?

I had thought that dual-src/dsthost worked the same as src/dsthost (fairness between hosts) with the exception that there is also fairness of flows within each host.

Here are some results (all rates aggregate throughput in Mbit):

IP1=8 up / 1 down   IP2=1 up / 8 down (post-test tc stats attached):
	srchost/dsthost, upload only: IP1=48.1, IP2=47.9  (OK)
	srchost/dsthost, download only: IP1=47.8, IP2=47.8  (OK)
	srchost/dsthost, bi-directional: IP1=47.5 up / 43.9 down, IP2=44.7 up / 46.7 down  (OK)

	dual-srchost/dual-dsthost, upload only: IP1=48.1, IP2=48.0  (OK)
	dual-srchost/dual-dsthost, download only: IP1=47.9, IP2=47.9  (OK)
	dual-srchost/dual-dsthost, bi-directional: IP1=83.0 up / 10.7 down, IP2=10.6 up / 83.0 down (*** asymmetric ***)

Dual-srchost/dual-dsthost, bi-directional tests with different flow counts:

IP1=4 up / 1 down   IP2=1 up / 4 down:
	IP1=74.8 up / 18.8 down, IP2=18.8 up / 74.8 down

IP1=2 up / 1 down   IP2=1 up / 2 down:
	IP1=62.4 up / 31.3 down, IP2=31.3 up / 62.4 down

IP1=4 up / 1 down   IP2=1 up / 8 down:
	IP1=81.8 up / 11.5 down, IP2=17.4 up / 76.3 down

IP1=2 up / 1 down   IP2=1 up / 8 down:
	IP1=79.9 up / 13.5 down, IP2=25.7 up / 68.1 down

The setup:

	apu2a (kernel 4.9)  <— default VLAN —>  apu1a (kernel 3.16.7)  <— VLAN 3300 —>  apu2b (kernel 4.9)

- apu1a is the router, and has cake only on egress of both eth0 and eth0.3300, rate limited to 100mbit for both
- it has no trouble shaping at 100mbit up and down simultaneously, so that should not be a problem
- the same problem occurs at 25mbit or 50mbit)
- since apu2a is the client [dual-]dsthost is used on eth0 and [dual-]srchost is used on eth0.3300
- the fairness test setup seems correct, based on the results of most of the tests, at least.
- note in the qdisc stats attached there is a prio qdisc on eth0 for filtering out VLAN traffic so it isn’t shaped twice
- I also get the exact same results with an htb or hfsc hierarchy on eth0 instead of adding a qdisc to eth0.3300
- printk’s in sch_cake.c shows values of flow_mode, srchost_hash and dsthost_hash as expected
- I also see it going into allocate_src and allocate_dst as expected, and later ending up in found_src and found_dst

I’m stumped. I know I’ve tested fairness of dual-src/dsthost before, but that was from the egress of client and server, and it was on a recent kernel. Time to sleep on it...


[-- Attachment #2: qdisc_stats.txt --]
[-- Type: text/plain, Size: 2195 bytes --]

qdisc prio 1: dev eth0 root refcnt 2 bands 2 priomap  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 Sent 1502062182 bytes 1525226 pkt (dropped 204373, overlimits 0 requeues 0) 
 backlog 0b 4294550547p requeues 0
qdisc cake 10: dev eth0 parent 1:1 bandwidth 100Mbit besteffort dual-dsthost nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 
 Sent 751856301 bytes 762993 pkt (dropped 8512, overlimits 2414269 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 361600b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                  Tin 0
  thresh        100Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        3.9ms
  av_delay        2.4ms
  sp_delay         60us
  backlog            0b
  pkts           771505
  bytes       764743469
  way_inds            6
  way_miss           26
  way_cols            0
  drops            8512
  marks               0
  ack_drop            0
  sp_flows            9
  bk_flows            4
  un_flows            0
  max_len         18168
  quantum          1514

qdisc cake 8060: dev eth0.3300 root refcnt 2 bandwidth 100Mbit besteffort dual-srchost nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0 
 Sent 750205881 bytes 762233 pkt (dropped 8542, overlimits 904221 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 279744b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                  Tin 0
  thresh        100Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        157us
  av_delay         83us
  sp_delay          1us
  backlog            0b
  pkts           770775
  bytes       763138469
  way_inds        31166
  way_miss           23
  way_cols            0
  drops            8542
  marks               0
  ack_drop            0
  sp_flows           17
  bk_flows            1
  un_flows            0
  max_len         30280
  quantum          1514