From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail.toke.dk; spf=pass smtp.mailfrom=; dkim=pass header.d=gmail.com; arc=none (Message is not ARC signed); dmarc=pass (Used From Domain Record) header.from=gmail.com policy.dmarc=quarantine Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by mail.toke.dk (Postfix) with ESMTPS id 578F1727D52 for ; Wed, 01 Oct 2025 22:58:22 +0200 (CEST) Received: by mail-ej1-x634.google.com with SMTP id a640c23a62f3a-b3e25a4bfd5so71424566b.2 for ; Wed, 01 Oct 2025 13:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759352301; x=1759957101; darn=lists.bufferbloat.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=LljycIaqggayAAOwpeeT/rqti2XIR5e0dmg8U6/xG28=; b=R1MkuNUbS3SEcl3tfpaUyFjxG+ppe8yLHZI1Z3WWh6u0ooJF3BFq19oi6snhPxncFM LuiqTle2VcFE46A+vqEk4sdt7P5/VeMQBc4I+XFuGHHTwQYfU4utRT4T3a0Z0aPNgsVZ OfzFndPFdOXIGkRJtrOBF9WtiqOgBBMhpqdrSEd1lVGtZtZ1USSFmjA8xHe/zxr00lNe HsdfO+l+hReRc0jJqeK7Jg/q+vLtscCcBgm6IB9Fgg1QNVc5SnqGdF+nW6MZPbIEfTS1 8qwit4UuclFuhEq4lSr8UgWlLnbwMJHxXMb5p17DTanai0e6PGGJygzX0m++roHhBfdE DS2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759352301; x=1759957101; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LljycIaqggayAAOwpeeT/rqti2XIR5e0dmg8U6/xG28=; b=bAw7luwu8IJBrmUhNKb6YcN1ib1POOc78a+w2LdA2eW2YHniYeFsKQ60QVAMf1YvG9 z9XnWiuG0VkSteLNxk5mwilzTtG3xC8KigqvhwRDgICrWi+i5wGRQChDly04ffgl4vId SHtj0SGjgWsuXhiGlUTR+jk2zUV0fmwrT9+7JcRN4fwHJ29+FjhqnfhzTMGMoyyVcdj3 sASXtfqY7IbEhJx2vfN5rUFTLXhzHsZGsAoSoyALBByIKycghXDtWXP+EgKyfN4j1C4S MOULuLk7BpVfX+qEcmMqnMIWRMNTlyVzIDgmUtSC2GS67E+OH+0wXfiMLfrWm74xdV36 FRHA== X-Forwarded-Encrypted: i=1; AJvYcCXVmOkR+vfnfiL65XMcUvDYzt0RTVZy18MKZrhLG3SiLKa30XYZB95wxovHO0ImE1Cd79sd@lists.bufferbloat.net X-Gm-Message-State: AOJu0YwTpypOL61JJuTkOG8Z2xLwm26dlW62ft1jC/jSfl0SPH1PDlfP 4r1/mBbq8VWWmT7hItJQoRV36m1q7Pgc1fdTjmu22u04ANTaLsTsExd8GnlNiB0dytb3vQqrqIq SC0eGUJk58nCfWNECIwJ9qxoLcZYe+Lg= X-Gm-Gg: ASbGncum1kvidYKayd5dZx+NjWy/7D7bMLJP9ov4tVd9KQaAG1pvwgMbuwLjLx+0Uaj D7IKoyr98ukQS9hXk8r9O0sB4i/oPHXiPqJjRZkSrWi4XXxk+ivboXijqFWnQ9aJhDGYvIlguQv yutYdz/3fYCqI+MKtAwjLrmZ/Mfx+ktT6Hf2qUNFZZprg+7KIaL99AdW+aUSkpgMuwGzlb7uWXc AcZwivOIM04rQacVsLfvdcd6Na+Y/dno9Oamf2zTdqpynVPuxO3T76wOUrbcqir9CZxS3c8z34b X-Google-Smtp-Source: AGHT+IF1HowEGMebrWwZKKCUftPp0tOe2IDm8TKnRKYluukLm8tcHNOnHkpoFENBFvo3M1lVvezE5A/ZS/Wledtr694= X-Received: by 2002:a17:907:8694:b0:b3b:b839:577b with SMTP id a640c23a62f3a-b46e42b6948mr699692266b.12.1759352301230; Wed, 01 Oct 2025 13:58:21 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: dave seddon Date: Wed, 1 Oct 2025 13:58:10 -0700 X-Gm-Features: AS18NWA_53SaB71YNYj1pskZVRat7ZXgyc8fQ1WRjfrQwTevdmPFRtIjvf3reNc Message-ID: To: Jonathan Morton Cc: Sebastian Moeller , David Lang , cake@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: EU455RZ7HRR2SWRVHIPJ2UN7Q3SYG3VD X-Message-ID-Hash: EU455RZ7HRR2SWRVHIPJ2UN7Q3SYG3VD X-MailFrom: dave.seddon.ca@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Subject: [Cake] Re: help request for cake on a large network List-Id: Cake - FQ_codel the next generation Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Regarding patching net/sched/sch_cake.c "#define CAKE_QUEUES (1024)". We should be able to get Nix to do this for us, but I honestly haven't tried doing this before. I've actually been meaning to write some nix so that NixOS can build the L4S and Prague kernel modules, so NixOS can have these nice things. On Tue, Sep 30, 2025 at 11:46=E2=80=AFAM Jonathan Morton wrote: > > > On 30 Sep, 2025, at 3:56 pm, Sebastian Moeller wrote: > > > >> what would need to be done to increase resources to allow for full iso= lation of more flows? > > > > I believe editing net/sched/sch_cake.c and increase the number of CAKE_= QUEUES: > > #define CAKE_QUEUES (1024) > > > > to something larger... > > > > however I do not know what a realistic maximum is... and IIRC cake will= search through this repeatedly > > There's nothing that should repeatedly scan the entire list of queues, so= there should only be a minor hit to performance from increasing the number= of queues. > > The set-associative hash searches only the set of 8 queues that the flow = hashes to, and only when there's enough pressure to cause a hash collision,= so this cost would be expected to go down if the number of queues is incre= ased in anticipation of a heavy workload. > > The list of active queues is scanned O(1) elements per delivered packet. = This doesn't mean that each queue is visited only once per delivery cycle,= nor that a complete scan of the list can never happen, but amortised over = many packets, the scanning cost is independent of the number of active flow= s. (At least in theory. It's grown into rather complex logic that I plan = to simplify in the next-gen qdisc. Running in "besteffort" mode already si= mplifies the control flow significantly.) > > Queues that recently went inactive are kept in a separate list to "decay"= their AQM state to quiescent. This is scanned at a rate of one element pe= r delivered packet. > > In the corner case of the configured memory limit being reached, Cake nee= ds to find the longest queue(s) to drop from. This is done using a heap st= ructure, *specifically* to avoid the possibility of a load-triggered DoS fa= ult. A linear scan is done only to initialise the heap (when the need for = overload dropping is first identified) which is a one-time operation. So e= ven in this case, the cost per dropped packet is nearly independent - O(log= N) - of the number of queues. > > So the main limit on the practical number of queues is really the number = of hash bits that are allocated to selecting a queue. Without specifically = checking, I would be reasonably confident that you could run as many as 64K= queues without difficulty. I would still test this in the lab before depl= oying to production. > > - Jonathan Morton > _______________________________________________ > Cake mailing list -- cake@lists.bufferbloat.net > To unsubscribe send an email to cake-leave@lists.bufferbloat.net --=20 Regards, Dave Seddon +1 415 857 5102