From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail.toke.dk; dkim=pass header.d=toke.dk; arc=none (Message is not ARC signed); dmarc=pass (Used From Domain Record) header.from=toke.dk policy.dmarc=reject From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1759224197; bh=sgPyA27LdphefuhDD8iMIzFjwbyr69ZQO5Gt89Fxl1A=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=ZN++dTzowfv8DBbuxeVsupMN+uFUsPdaRAucTQMrIZarVl0zQaTWdmXIt8khSsZkj tiJrUxDKD8s6PqqH8gdGL2vPgeUvEZmPALSTIuU4yH9g/O2eYmlpZPupNm/RWceqo1 GZExrIQ2jwrPwIRvwVZyi4leZG0ulKlxiwJL4k3bxDxJpy+SmMCqoRFWf+dAxA7YBj IZal6gvytxKLpTn+6qby1re66Jge3zPgcv9Zkaewl9oqOdlte0QdGqZvaRy+4GR27y 8u6in8WJSbcnMRJxNLegzyTJ3nWVppTI2SViElJBeePyCTBPTQlLhbob+hitRtNhkw 0ONistyu03uNg== To: Sebastian Moeller Cc: cake@lists.bufferbloat.net In-Reply-To: <837EA4ED-26D3-4D83-84D9-5C0C75CFB80D@gmx.de> References: <06DAA0CB-70EF-4FDD-BD3D-16A4FC28AD12@gmx.de> <87zfacmhuc.fsf@toke.dk> <837EA4ED-26D3-4D83-84D9-5C0C75CFB80D@gmx.de> Date: Tue, 30 Sep 2025 11:23:16 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87v7l0mgzf.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: MBGNY2UAJ6UK4HEEMOS5CURCQEIASBJ2 X-Message-ID-Hash: MBGNY2UAJ6UK4HEEMOS5CURCQEIASBJ2 X-MailFrom: toke@toke.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Subject: [Cake] Re: help request for cake on a large network List-Id: Cake - FQ_codel the next generation Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Sebastian Moeller writes: > Hi Toke, > > >> On 30. Sep 2025, at 11:04, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >>=20 >> David Lang writes: >>=20 >>> Sebastian Moeller wrote: >>>=20 >>>> Hi David, >>>>=20 >>>> while I have no real answer for your questions (due to never having ha= d that kind of load in my home network ;) ) I would like to ask you to make= take scripted captures of tc -s qdisc for the wan interface is reasonable = short intervals (say every 10 minutes?) as that might be just what we need = to actually answer your question. >>>=20 >>> I will do that, however the network is only up under load for 4 days a = year, so=20 >>> it's a slow feedback loop :-) >>>=20 >>> I would welcome any other suggestions for data to gather. >>=20 >> Having queue statistics at a scale as granular as you can manage would >> be cool. It's around ~400 bytes of raw data per sample Capturing that >> every 100ms for four days is only around 1.4 GB of data; should >> theoretically be manageable? :) >>=20 >> Note that the 400 bytes is the in-kernel binary representation; the >> output of `tc -s` is somewhat more; using JSON output (`tc -j -s`) and >> compressing the output may get within something that server-grade >> hardware should handle just fine. >>=20 >>>>> On 28. Sep 2025, at 13:06, David Lang wrote: >>>>>=20 >>>>> I'm starting to prepare for the next Scale conference and we are swit= ching from Juniper routers to Linux routers. This gives me the ability to i= mplement cake. >>>>>=20 >>>>> One problem we have is classes that tell everyone 'go download this' = that trigger hundreds of people to hammer the network at the same time (thi= s is both a wifi and a network bandwidth issue, wifi is being worked on) >>>>=20 >>>=20 >>>> So one issue might be that with several 100 users the default compile-= time=20 >>>> size of queues (1024, IIRC) that cake will entertain might be too litt= le, even=20 >>>> in light of the 8 way assoziative hashing design. I believe this can b= e=20 >>>> changed (within limits) only by modifying at source and recompilation = of the=20 >>>> kernel, if that should be needed at all. >>>=20 >>> custom compiling a kernel is very much an option (and this sort of twea= king is=20 >>> the sort of thing I'm expecting to need to do) >>>=20 >>> The conference is in March, so we have some time to think about this an= d=20 >>> customize things, just no chance to test before the show. >>>=20 >>>> I wonder whether multi-queue cake would not solve this to some degree,= as I=20 >>>> assume each queue's instance would bring its own independent set of 10= 24 bins? >>>=20 >>> good thought >>=20 >> While I certainly wouldn't mind having a large-scale test of the >> multi-queue variant of cake, I don't really think it's necessary at 1G. >> Assuming you're using server/desktop-grade hardware for the gateways, >> CAKE should scale just fine to 1Gbit. >>=20 >> Sebastian is right that the MQ variant will install independent CAKE >> instances on each hardware queue, which will give you more flow queues. >> However, the round-robin dequeueing among those queues will also be >> completely independent, so you won't get fairness among them either >> (only between the flows that share a HWQ). >>=20 >> As for collision probability, we actually have a calculation of this in >> the original CAKE paper[0], in figure 1. With set-associative hashing, >> collision probability only start to rise around 500 simultaneous flows. >> And bear in mind that these need to be active flows *from the PoV of the >> router*. I.e., they need to all be actively transmitting data at the >> same time; even with lots of users with active connections as seen from >> the endpoint, the number of active flows in the router should be way >> smaller (there's a paper discussing this that I can't find right now). > > Maybe Luca's "Evaluating the Number of Active Flows in a Scheduler > Realizing Fair Statistical Bandwidth Sharing" from 2013? > https://team.inria.fr/rap/files/2013/12/KMOR05b.pdf Yup, that was exactly the one I was thinking about - thank you for digging up the link! (Adding back the list to Cc so others can see it too). -Toke