From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2a0c:4d80:42:2001::664]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EE4A73B2A4 for ; Mon, 28 Sep 2020 11:14:26 -0400 (EDT) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1601306064; bh=zO0GopcbCDI23bBNv/agM57pWT9yK/M02KF8t93ScpU=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=BVm4OK5nGv/O7CkIzxi48dwiOmAsoDz+eB1OUb/Pq6uBXH5tfLbfJUaIOdiE0Nm2P QNHZdSbV+SG7bDbP7jQbSTTGYlUkTi3gFaXl/JrYzULqMGMWzrcwKllEwA1Oke7Tsl YPzKtiqgspoXu5h+sTSqBP11YapVr11/qqBuIGYkFprfhwqT4hsJ8ieSGwIihGcGHq GTbj/6yCORze0TX+5oZMU/ryfdHpGHPDYAdZi9E2L/TUTuk3P7xYUZSJzxsK0L8DnM sUrM6FumzogtpaHRguGgMnx3qGc70mmKx7vnz2L2zI6AnM2s2ghL1qe6+MkYphp8hf 7RobHZzV/bOjQ== To: Daniel Sterling , Jonathan Morton Cc: Michael Richardson , bloat In-Reply-To: References: <32080.1597787724@localhost> <3A782CD0-01F1-40FA-9DFD-B969BD11A566@gmail.com> Date: Mon, 28 Sep 2020 17:14:24 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87blhq3pdb.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Bloat] cake + ipv6 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Sep 2020 15:14:27 -0000 Daniel Sterling writes: > I guess the reason I'm surprised is I'm confused about the following: > > For UDP streams that use <1mbit down , should I expect cake in ingress > mode to keep those at low latency even in the face of a constantly > full queue / backlog, using "besteffort" but also using host > isolation? It depends. A 'sparse' flow should get consistent priority (and hence low latency), but what exactly constitutes a sparse flow varies with the link bandwidth and usage. I wrote a longish analysis of this for FQ-CoDel[0] which should more or less carry over to CAKE; basically it boils down to any flow using less than it's "fair share" of the link should get priority, assuming its packets are spaced evenly. The main difference between CAKE and FQ-CoDel is that the host fairness changes a flows 'fair share', and that CAKE has a 'cooldown' period for sparse flows after they disappear, which may change dynamics slightly. But the basic mechanism is the same. [0] https://ieeexplore.ieee.org/document/8469111/ > By backlog I mean what I see with tc -s qdisc. I assume that's the > total of all individual flow backlogs, right? Yup. > I'm guessing no, but I'm wondering why not. Let's say we have some > hosts doing downloads as fast as they can with as many connections as > they can, and another separate host doing "light" UDP (xbox FPS > traffic). > > So that's, say, 4 hosts constantly filling their queue's backlog -- it > will always have as many bytes as the rtt setting allows -- so by > default up to 100ms of bytes at my bandwidth setting, per flow, right? > Or is that per host? That's per flow. But you're misunderstanding the 100ms value. That's the 'interval', which is (simplifying a bit) the amount of time CAKE will wait until it reacts to a flow building a queue. The actual amount of queueing CAKE is *aiming for* is the 'target', which is interval/20 so 5ms by default. So in the 'steady state' a flow's backlog should oscillate around this (ignoring for the moment that "steady state" is an idealisation that rarely, if ever, exists in the real world). BTW, you can see per-flow statistics by using 'tc -s class show $DEVICE'. > And then another host (the xbox) that will have a constant flow that > doesn't really respond to shaping hints -- it's going to have a steady > state of packets that it wants to receive and send no matter what. It > might go from "high" to "low" update resolution (e.g. 256kbit to > 128bkit), but that's about it. It will always want about 256kbit down > and 128kbit up with v4 UDP. > > Normally that stream will have an rtt of < 50ms. Sometimes, e.g. > in-between rounds of the same game (thus the same UDP flow), the > server might let the rtt spike to 100+ ms since nothing much needs to > be sent between rounds. I'm not quite sure what you mean by "the server will let the RTT spike", actually. Above you seemed to be saying that the gaming flow would just carry one sending at whatever rate it wants? > But once the new round starts, we'll want low latency again. > > Is it at all possible that cake, seeing the UDP stream is no longer > demanding low latency (in-between rounds), it figures it can let its > rtt stay at 100+ms per its rtt settings, even after the new round > starts and the xbox wants low latency again? You're attributing a bit more intent to CAKE here than it really possesses ;) CAKE doesn't have a notion of "what a flow wants". It just schedules flows in a certain way, and if a flow happens to use less than its fair share of bandwidth (the analysis linked above), it'll get temporary priority whenever a packet from that flow arrives. > That is, since every host wants some traffic, and most if not all the > queues / backlogs will always be filled, is it possible that once a > flow allows its rtt to rise, cake won't let it back down again until > there's a lull? Yes, this part is right, sorta. If a flow exceeds its fair share *rate*, it'll no longer (from CAKEs) PoV be a 'sparse flow', and it'll get the same treatment as all other flows (round-robin scheduling), and if it keeps sending at this higher rate, it'll keep being scheduled in this way. If the flow is non-elastic (i.e., doesn't slows down in response to packet drops), it'll self-congest and you'll see that as increased latency. So if what you meant by "the server will let the RTT spike", is that the flow is bursty at times, and in some periods it'll increase its rate above what CAKE's notion of fair share is, then yeah, that can lead to the behaviour you're seeing. > As I said, I solved this by giving xbox traffic absolute first > priority with the "prio" qdisc. Obviously this means the xbox traffic > can starve everything else given a malicious flow, but that's not > likely to happen and if it does, I will notice. If it helps to always prioritise the flow, then that seems to be an indication that what I'm describing above is what you're seeing. The various diffserv modes of CAKE are meant to be a way to deal with this, BTW. It's similar to the prio qdisc in that it will use diffserv markings to prioritise some traffic, but it will also impose a max total bandwidth that each higher-priority tier can use before being automatically demoted. If those limits fit your requirements, you may be able to just use that instead of the prio qdisc (you don't have to use real diffserv marks either, CAKE can be configured to get the priority band from fwmark or skb->priority instead). -Toke