[Bloat] cake + ipv6

[Daniel Sterling]

On Wed, Sep 23, 2020 at 2:13 PM Jonathan Morton <chromatix99 at gmail.com> wrote:
> It fits my mental model, yes, though obviously the ideal would be to recognise that the xbox is a singular machine.  Are you seeing a larger disparity than that?  If so, is it even larger than four connections would justify without host-fairness?

Thanks Jonathan,

Unfortunately I fear I've been running an improperly configured cake
all this time-- leave it to me to take a piece of cake and mess it up
:) I should say, even with it being possibly misconfigured, it has
still worked amazingly well for me! I have no end of gratitude for
y'alls work --

So taking your advice, with my cake properly configured, the answer is
no -- it looks like host-fairness is working correctly -- with a
notable exception -- read on for details...

I was using these probably wrong settings:

WAN: besteffort bandwidth $UPBANDWIDTH internet nat egress ethernet
LAN: besteffort bandwidth $BANDWIDTH internet nat ingress ethernet

So, I was using triple-isolate instead of the correct src/dest
isolation options per NIC, and I was using "nat" on the LAN side,
which may have been further breaking host isolation as per the openwrt
wiki:

"don't add the nat option on LAN interfaces (this option should only
be used in the WAN interface) or Per-Host Isolation stops working" -
https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details

I changed my configuration to:

$WAN handle 1: root cake besteffort bandwidth $UPBANDWIDTH internet
nat egress dual-srchost ethernet
$LAN handle 1: root cake besteffort bandwidth $BANDWIDTH internet
ingress dual-dsthost ethernet

So:

root at OpenWrt:~# tc -s qdisc | grep cake
qdisc cake 1: dev eth1 root refcnt 2 bandwidth 20Mbit besteffort
dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm
overhead 38 mpu 84
qdisc cake 1: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort
dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100.0ms
noatm overhead 38 mpu 84

WAN:
* nat option enabled
* egress option (default) enabled
* dual-srchost

LAN:
* nat option disabled (default)
* ingress option enabled
* dual-dsthost

This worked well, but I ran into a bizarre issue:

My xbox FPS gaming UDP stream would start at a low latency (even while
other systems were doing bulk downloads), but after some time, the
latency went from a steady 30ms (+/- <10ms for jitter) to over 100ms !
100ms is of course notably the default cake rtt parameter. Resetting
cake (tc del and then tc add again) fixed this; the same stream's
latency went back down to about 30ms.

This gaming stream uses well under 512kbps down and less than half
that up, normally.

This is worrisome! My use-case may be an edge case, as I have AT&T
gigabit fiber but I limit it to 40mbit with cake, since empirically
that's the highest throughput I've found that I can use while also
maintaining the low jitter / latency over wifi. So certainly the WAN
NIC is getting a lot of traffic that cake on the LAN NIC is throwing
away. That is, there is basically always a large number of bytes in
LAN cake's backlog.

For now I've solved this by using the "prio" tc qdisc such that xbox
traffic bypasses cake, and all other traffic goes thru cake.

Is this a known issue at all? I was quite surprised to see this
behaviour. Let me know if I can help debug this. It likely only
happens when multiple other hosts are doing bulk downloads / video
streaming.

Thanks,
Dan