[Cake] dual-src/dsthost unfairness, only with bi-directional traffic

[Pete Heist]

> On Jan 3, 2019, at 6:18 AM, Jonathan Morton <chromatix99 at gmail.com> wrote:
> 
>> On 3 Jan, 2019, at 6:15 am, Georgios Amanakis <gamanakis at gmail.com> wrote:
>> 
>> It seems if both clients are having bidirectional traffic, dual-
>> {dst,src}host has the same effect as triple-isolate (on both lan and
>> wan interfaces) on their bandwidth.

Exactly what I’m seeing- thanks for testing George...

> I'm left wondering whether the sense of src and dst has got accidentally reversed at some point, or if the dual modes are being misinterpreted as triple-isolate.  To figure that out, I'd need to look carefully at several related parts of the code.  Can anyone reproduce it from the latest kernels' upstream code, or is it only in the module?  And precisely which version of iproute2 is everyone using?

It will be a while before I can try this on 4.19+, but: iproute2/oldstable,now 3.16.0-2 i386. I compile tc-adv from HEAD.

Here are more bi-directional tests with 8 up / 1 down on IP1 and 1 up / 8 down on IP2:

dual-srchost/dual-dsthost:
	IP1: 83.1 / 10.9, IP2: 10.7 / 83.0
dual-dsthost/dual-srchost (sense flipped):
	IP1: 83.0 / 10.5, IP2: 10.7 / 82.9
triple-isolate:
	IP1: 83.1 / 10.5, IP2: 10.7 / 82.9
srchost/dsthost (sanity check):
	IP1: 47.6 / 43.8, IP2: 44.2 / 47.4
dsthost/srchost (sanity check, sense flipped):
	IP1: 81.3 / 9.79, IP2: 11.0 / 80.7
flows:
	IP1: 83.0 / 10.4, IP2: 10.5 / 82.9

I also tried testing shaping on eth0.3300 and ingress of eth0.3300 instead of egress of both eth0 and eth0.3300, because that’s more like what I tested before. There was no significant change from the above results.

I managed to compile versions all the way back to July 15, 2018 (1e2473f702cf253f8f5ade4d622c6e4ba661a09d) and still see the same result. I’ll try to go earlier.

As far as the code goes, the easy stuff:
- flow_mode values in cake_hash are 5 for dual-srchost, 6 for dual-dsthost and 7 for triple-isolate
- the values from cake_dsrc(flow_mode) and cake_ddst(flow_mode) are as expected in all three cases
- flow_override and host_override are both 0
- looks correct: !(flow_mode & CAKE_FLOW_FLOWS) == 0
- this looks normal to me (shows reply packets on eth0):
   IP1 ping: dsthost_idx = 450, reduced_hash = 129
   IP1 irtt: dsthost_idx = 450, reduced_hash = 158
   IP2 ping: dsthost_idx = 301, reduced_hash = 78
   IP2 irtt: dsthost_idx = 301, reduced_hash = 399

Jon, is there anything I can check by instrumenting the code somewhere specific?