[Bloat] cake + ipv6

[Toke Høiland-Jørgensen]

Daniel Sterling <sterling.daniel at gmail.com> writes:

> I guess the reason I'm surprised is I'm confused about the following:
>
> For UDP streams that use <1mbit down , should I expect cake in ingress
> mode to keep those at low latency even in the face of a constantly
> full queue / backlog, using "besteffort" but also using host
> isolation?

It depends. A 'sparse' flow should get consistent priority (and hence
low latency), but what exactly constitutes a sparse flow varies with the
link bandwidth and usage. I wrote a longish analysis of this for
FQ-CoDel[0] which should more or less carry over to CAKE; basically it
boils down to any flow using less than it's "fair share" of the link
should get priority, assuming its packets are spaced evenly. The main
difference between CAKE and FQ-CoDel is that the host fairness changes a
flows 'fair share', and that CAKE has a 'cooldown' period for sparse
flows after they disappear, which may change dynamics slightly. But the
basic mechanism is the same.

[0] https://ieeexplore.ieee.org/document/8469111/

> By backlog I mean what I see with tc -s qdisc. I assume that's the
> total of all individual flow backlogs, right?

Yup.

> I'm guessing no, but I'm wondering why not. Let's say we have some
> hosts doing downloads as fast as they can with as many connections as
> they can, and another separate host doing "light" UDP (xbox FPS
> traffic).
>
> So that's, say, 4 hosts constantly filling their queue's backlog -- it
> will always have as many bytes as the rtt setting allows -- so by
> default up to 100ms of bytes at my bandwidth setting, per flow, right?
> Or is that per host?

That's per flow. But you're misunderstanding the 100ms value. That's the
'interval', which is (simplifying a bit) the amount of time CAKE will
wait until it reacts to a flow building a queue. The actual amount of
queueing CAKE is *aiming for* is the 'target', which is interval/20 so
5ms by default. So in the 'steady state' a flow's backlog should
oscillate around this (ignoring for the moment that "steady state" is an
idealisation that rarely, if ever, exists in the real world).

BTW, you can see per-flow statistics by using 'tc -s class show
$DEVICE'.

> And then another host (the xbox) that will have a constant flow that
> doesn't really respond to shaping hints -- it's going to have a steady
> state of packets that it wants to receive and send no matter what. It
> might go from "high" to "low" update resolution (e.g. 256kbit to
> 128bkit), but that's about it. It will always want about 256kbit down
> and 128kbit up with v4 UDP.
>
> Normally that stream will have an rtt of < 50ms. Sometimes, e.g.
> in-between rounds of the same game (thus the same UDP flow), the
> server might let the rtt spike to 100+ ms since nothing much needs to
> be sent between rounds.

I'm not quite sure what you mean by "the server will let the RTT spike",
actually. Above you seemed to be saying that the gaming flow would just
carry one sending at whatever rate it wants?

> But once the new round starts, we'll want low latency again.
>
> Is it at all possible that cake, seeing the UDP stream is no longer
> demanding low latency (in-between rounds), it figures it can let its
> rtt stay at 100+ms per its rtt settings, even after the new round
> starts and the xbox wants low latency again?

You're attributing a bit more intent to CAKE here than it really
possesses ;)

CAKE doesn't have a notion of "what a flow wants". It just schedules
flows in a certain way, and if a flow happens to use less than its fair
share of bandwidth (the analysis linked above), it'll get temporary
priority whenever a packet from that flow arrives.

> That is, since every host wants some traffic, and most if not all the
> queues / backlogs will always be filled, is it possible that once a
> flow allows its rtt to rise, cake won't let it back down again until
> there's a lull?

Yes, this part is right, sorta. If a flow exceeds its fair share *rate*,
it'll no longer (from CAKEs) PoV be a 'sparse flow', and it'll get the
same treatment as all other flows (round-robin scheduling), and if it
keeps sending at this higher rate, it'll keep being scheduled in this
way. If the flow is non-elastic (i.e., doesn't slows down in response to
packet drops), it'll self-congest and you'll see that as increased
latency. So if what you meant by "the server will let the RTT spike", is
that the flow is bursty at times, and in some periods it'll increase its
rate above what CAKE's notion of fair share is, then yeah, that can lead
to the behaviour you're seeing.

> As I said, I solved this by giving xbox traffic absolute first
> priority with the "prio" qdisc. Obviously this means the xbox traffic
> can starve everything else given a malicious flow, but that's not
> likely to happen and if it does, I will notice.

If it helps to always prioritise the flow, then that seems to be an
indication that what I'm describing above is what you're seeing.

The various diffserv modes of CAKE are meant to be a way to deal with
this, BTW. It's similar to the prio qdisc in that it will use diffserv
markings to prioritise some traffic, but it will also impose a max total
bandwidth that each higher-priority tier can use before being
automatically demoted. If those limits fit your requirements, you may
be able to just use that instead of the prio qdisc (you don't have to
use real diffserv marks either, CAKE can be configured to get the
priority band from fwmark or skb->priority instead).

-Toke