[Cake] dual-src/dsthost unfairness, only with bi-directional traffic

[Georgios Amanakis]

It seems if both clients are having bidirectional traffic, dual-
{dst,src}host has the same effect as triple-isolate (on both lan and
wan interfaces) on their bandwidth.
This shouldn't happen though, or am I wrong?


On Wed, 2019-01-02 at 22:57 -0500, Georgios Amanakis wrote:
> I can reproduce this one to my surprise, too. 
> I tested on my Comcast connection, with a WRT1900ACS, running openwrt
> (r8082-95b3f8ec8d, 4.14.70), with two interfaces br-lan and
> eth0(wan).
> 
> IP1=1 up / 8 down    IP2=4 up / 4 down
> 	src/dst, bidir: IP1=0.88 /  8.44, IP2=0.66 / 7.75 (ok)
> dualsrc/dualdst, bidir: IP1=0.27 / 10.56, IP2=1.41 / 6.42 (unfair)
> 
> No VLANs, no other schedulers on eth0 and br-lan apart from cake.
> 
> 
> 
> On Wed, 2019-01-02 at 00:04 +0100, Pete Heist wrote:
> > In my one-armed router setup I’m seeing host fairness work
> > perfectly
> > with srchost or dsthost, but with dual-srchost or dual-dsthost,
> > host
> > fairness deviates from the ideal, _only_ when there's bi-
> > directional
> > traffic. The deviation is then dependent on the number of flows. Is
> > this expected?
> > 
> > I had thought that dual-src/dsthost worked the same as src/dsthost
> > (fairness between hosts) with the exception that there is also
> > fairness of flows within each host.
> > 
> > Here are some results (all rates aggregate throughput in Mbit):
> > 
> > IP1=8 up / 1 down   IP2=1 up / 8 down (post-test tc stats
> > attached):
> > 	srchost/dsthost, upload only: IP1=48.1, IP2=47.9  (OK)
> > 	srchost/dsthost, download only: IP1=47.8, IP2=47.8  (OK)
> > 	srchost/dsthost, bi-directional: IP1=47.5 up / 43.9 down,
> > IP2=44.7 up / 46.7 down  (OK)
> > 
> > 	dual-srchost/dual-dsthost, upload only: IP1=48.1,
> > IP2=48.0  (OK)
> > 	dual-srchost/dual-dsthost, download only: IP1=47.9,
> > IP2=47.9  (OK)
> > 	dual-srchost/dual-dsthost, bi-directional: IP1=83.0 up / 10.7
> > down, IP2=10.6 up / 83.0 down (*** asymmetric ***)
> > 
> > Dual-srchost/dual-dsthost, bi-directional tests with different flow
> > counts:
> > 
> > IP1=4 up / 1 down   IP2=1 up / 4 down:
> > 	IP1=74.8 up / 18.8 down, IP2=18.8 up / 74.8 down
> > 
> > IP1=2 up / 1 down   IP2=1 up / 2 down:
> > 	IP1=62.4 up / 31.3 down, IP2=31.3 up / 62.4 down
> > 
> > IP1=4 up / 1 down   IP2=1 up / 8 down:
> > 	IP1=81.8 up / 11.5 down, IP2=17.4 up / 76.3 down
> > 
> > IP1=2 up / 1 down   IP2=1 up / 8 down:
> > 	IP1=79.9 up / 13.5 down, IP2=25.7 up / 68.1 down
> > 
> > The setup:
> > 
> > 	apu2a (kernel 4.9)  <— default VLAN —>  apu1a (kernel
> > 3.16.7)  <— VLAN 3300 —>  apu2b (kernel 4.9)
> > 
> > - apu1a is the router, and has cake only on egress of both eth0 and
> > eth0.3300, rate limited to 100mbit for both
> > - it has no trouble shaping at 100mbit up and down simultaneously,
> > so
> > that should not be a problem
> > - the same problem occurs at 25mbit or 50mbit)
> > - since apu2a is the client [dual-]dsthost is used on eth0 and
> > [dual-
> > ]srchost is used on eth0.3300
> > - the fairness test setup seems correct, based on the results of
> > most
> > of the tests, at least.
> > - note in the qdisc stats attached there is a prio qdisc on eth0
> > for
> > filtering out VLAN traffic so it isn’t shaped twice
> > - I also get the exact same results with an htb or hfsc hierarchy
> > on
> > eth0 instead of adding a qdisc to eth0.3300
> > - printk’s in sch_cake.c shows values of flow_mode, srchost_hash
> > and
> > dsthost_hash as expected
> > - I also see it going into allocate_src and allocate_dst as
> > expected,
> > and later ending up in found_src and found_dst
> > 
> > I’m stumped. I know I’ve tested fairness of dual-src/dsthost
> > before,
> > but that was from the egress of client and server, and it was on a
> > recent kernel. Time to sleep on it...
> > 
> > _______________________________________________
> > Cake mailing list
> > Cake at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake