[Cake] Re: help request for cake on a large network

[David Lang <david@lang.hm>]

I'll read this in more detail later (it's almost 6am after I've been driving all 
night), but large number of simultanious flows in not unlikely.

Several classes that start at the same time, each with 100-300 people in it.

one of the first slides that go up tell the students to fetch something (over 
https, and for example with nix, with certificate pinning, so no ability to have 
a local cache for it), or more commonly, fetch a script tht then downloads a 
bunch of somethings.

all of the sudden, 500 simulanious flows doesn't look that unlikely :-)

David Lang

Toke Høiland-Jørgensen wrote:

> David Lang <david@lang.hm> writes:
>
>> Sebastian Moeller wrote:
>>
>>> Hi David,
>>>
>>> while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.
>>
>> I will do that, however the network is only up under load for 4 days a year, so
>> it's a slow feedback loop :-)
>>
>> I would welcome any other suggestions for data to gather.
>
> Having queue statistics at a scale as granular as you can manage would
> be cool. It's around ~400 bytes of raw data per sample Capturing that
> every 100ms for four days is only around 1.4 GB of data; should
> theoretically be manageable? :)
>
> Note that the 400 bytes is the in-kernel binary representation; the
> output of `tc -s` is somewhat more; using JSON output (`tc -j -s`) and
> compressing the output may get within something that server-grade
> hardware should handle just fine.
>
>>>> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
>>>>
>>>> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
>>>>
>>>> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)
>>>
>>
>>> So one issue might be that with several 100 users the default compile-time
>>> size of queues (1024, IIRC) that cake will entertain might be too little, even
>>> in light of the 8 way assoziative hashing design. I believe this can be
>>> changed (within limits) only by modifying at source and recompilation of the
>>> kernel, if that should be needed at all.
>>
>> custom compiling a kernel is very much an option (and this sort of tweaking is
>> the sort of thing I'm expecting to need to do)
>>
>> The conference is in March, so we have some time to think about this and
>> customize things, just no chance to test before the show.
>>
>>> I wonder whether multi-queue cake would not solve this to some degree, as I
>>> assume each queue's instance would bring its own independent set of 1024 bins?
>>
>> good thought
>
> While I certainly wouldn't mind having a large-scale test of the
> multi-queue variant of cake, I don't really think it's necessary at 1G.
> Assuming you're using server/desktop-grade hardware for the gateways,
> CAKE should scale just fine to 1Gbit.
>
> Sebastian is right that the MQ variant will install independent CAKE
> instances on each hardware queue, which will give you more flow queues.
> However, the round-robin dequeueing among those queues will also be
> completely independent, so you won't get fairness among them either
> (only between the flows that share a HWQ).
>
> As for collision probability, we actually have a calculation of this in
> the original CAKE paper[0], in figure 1. With set-associative hashing,
> collision probability only start to rise around 500 simultaneous flows.
> And bear in mind that these need to be active flows *from the PoV of the
> router*. I.e., they need to all be actively transmitting data at the
> same time; even with lots of users with active connections as seen from
> the endpoint, the number of active flows in the router should be way
> smaller (there's a paper discussing this that I can't find right now).
> Having some data about this would be interesting, of course (and should
> be part of the tc statistics).
>
> -Toke
>
> [0] https://arxiv.org/pdf/1804.07617
>