From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail.toke.dk; spf=pass smtp.mailfrom=; dkim=pass header.d=gmail.com; arc=none (Message is not ARC signed); dmarc=pass (Used From Domain Record) header.from=gmail.com policy.dmarc=quarantine Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by mail.toke.dk (Postfix) with ESMTPS id B98047031B8 for ; Sun, 28 Sep 2025 19:08:53 +0200 (CEST) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-afcb78ead12so650439166b.1 for ; Sun, 28 Sep 2025 10:08:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759079272; x=1759684072; darn=lists.bufferbloat.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=rKt2tKXGPcOlZ3t9G1kLib2YRO9ujc8loEo3wuCbyWg=; b=JYMtBpumSzDA65DDFlMwp9e9xCnbq/UQrJsfYf6Z3FpxaXaUsMDvHPSwcPuZIxJQjG qZzF1IVgULm1FWlTo9TkcJPUhAw9u9K4eYyJrdC57xb5fO9IvzUTtMa1WBIr1Z90VrVy Gh6CMFDx3RgZ4bpIa+B64FzEgGzk6XxXd8jLyb37lOvbh+LxAFEsk8UzzD/uZY14fHkL +GYCzE5JPt4S8gDJ1GuD6HpZxZ24z8liA+qNkfZJxgUy5RJXI89DQYt7IbaWkCQIfw+p zPGMdDhZOPxw0VS/q8UnNMUn1RtuVzzNwxA32awNzCmFMpWPd+92RLc6wjXai0ER7YXs uMEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759079272; x=1759684072; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rKt2tKXGPcOlZ3t9G1kLib2YRO9ujc8loEo3wuCbyWg=; b=fn+ZRsZwMKdMfthRGFgUX8gvg7jABcLyRu1nAxdNu5SlgnDcpy58HQpditfbb4ws3d KHpKYdi3OzCylRwTdj6oxEgXQL2hk3GkppDY629nmm5XTZlf9UB+EkC25DNZCW4+3s6H +cNrMBcZH6kSEGMg8zwtDhtau+SMIv6HQ9uxn2QXEJTxxc/eiUapozpkimd5Tey19I1+ NME/ICj6x3OkEP4MMoyPgTG39Z683MjqQQFceS/2f/v8HQCetoaDxyU0J/FFo/uHuNCR 04jp8UEWcEXcwSS0AoWtpKTMnDsK5gX/zb4p3XqqQSFIVGBbMn5bAYNfjcln0PUHmeQk XWxQ== X-Forwarded-Encrypted: i=1; AJvYcCX6lJ8zA0h9SNocuMArb+bOhHQ47LnPkxohIJb8J8/nlp41nkUyKMydyc4V2nsqjjHhuXNY@lists.bufferbloat.net X-Gm-Message-State: AOJu0YxtyjV7HRtwl+pJtW7OK3fBrOjPC89anLBoeS1D9I/YVsAxEkwy 2WCmK7daBWwFysETY53njWd9TC1cxqIkZVsNjA01Y50PyiLjJapzthWe3njcpQDC+tMF2g/L8DB Du/6tgx/ir57hqMAYLX99q+csiSEEcqE= X-Gm-Gg: ASbGnctARMYAIMa6mg9fI8GpYvAJGxMrSckBV+Q20MvWv2uUNK5jVLtbeZGMW6QctIP AKYgi8Ug910luUfXNLhCxQ0VxhwD6LaYQkVzL6ecn74CmQQeS/2UokrgO82QZkHZXZfV2jMuQg+ 9/6wC6E99KgPn/pH+eBKnZFuwe9P5ZhuW+HD//Ib79rJgnRD+EaLJ7lBXF30PyYgwPpHtS7FFnR 6tou+oemPprOublm35aDFwUCO7uUCIPp4r31XocPcOs/2ExzYUfPzxpkzJT3Yw= X-Google-Smtp-Source: AGHT+IHINGc0Aa1CoOfkphXB1bypx5KFGGghvjcRxGUnyd6+KYbXTGsg1/CNgKQnuenPgbA5JkgYg3zLI9h5N5LKw0o= X-Received: by 2002:a17:907:3f87:b0:b09:2331:f150 with SMTP id a640c23a62f3a-b34b84aba85mr1672150066b.16.1759079271831; Sun, 28 Sep 2025 10:07:51 -0700 (PDT) MIME-Version: 1.0 References: <2064666241.42029.1759061544846@app.mailbox.org> In-Reply-To: From: dave seddon Date: Sun, 28 Sep 2025 10:07:40 -0700 X-Gm-Features: AS18NWBpY_FyB49rh8IKS9cqy_l-rMBgn1sttTR301rNtJk0arxjGbDRSoyTCyo Message-ID: To: David Lang Cc: m@jaap.pro, "cake@lists.bufferbloat.net" , Frantisek Borsik Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: ZGZJXLUNR4T6WK4WG7G6EYEEAL26Q42B X-Message-ID-Hash: ZGZJXLUNR4T6WK4WG7G6EYEEAL26Q42B X-MailFrom: dave.seddon.ca@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Subject: [Cake] Re: help request for cake on a large network List-Id: Cake - FQ_codel the next generation Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: G'day David, This sounds great and I'd be very happy to discuss this in real time to give you a hand. The SCALE team does a lot of Nix, right? I think all the OpenWrt One routers were configured with nix, and so I assume you're going to go with Nix for this? I think the TLDR is that cake will mostly do what you want. The idea being that "it is a piece of cake" to use cake. ;) I was recently configuring an active/standby firewall, including configuring cake, with Nix, which I really need to 1) put config on github 2) document 3) and make a video about. One nice thing I worked out is that the systemd networkd makes it super easy to apply the qdisc of choice. This isn't the whole config, but hopefully gives you the vague idea cakeConfig =3D { Bandwidth =3D "990M"; # We currently have 1Gb/s, so this is our limit OverheadBytes =3D 38; # Ethernet overhead (preamble + inter-frame gap = + FCS) MPUBytes =3D 84; # Minimum packet unit for Ethernet NAT =3D true; FlowIsolationMode =3D "triple"; PriorityQueueingPreset =3D "besteffort"; }; systemd.network.networks =3D { # Bond interface configuration "bond0" =3D { matchConfig.Name =3D bondConfig.Name; networkConfig =3D { Address =3D [ (constructIP bondConfig.subnet4 (getNodeAddress bondConfig)) #bondConfig.vrrp_ip # secondary IP #fd00::1/64" #FIXME!! ]; LinkLocalAddressing =3D "no"; # Note: Gateway is configured in routes section with high metric # to make it less preferred than DHCP route (metric 1024) }; routes =3D [ { Gateway =3D bondConfig.gateway_ip; Destination =3D "0.0.0.0/0"; # Default route (all destinations) Metric =3D 2000; # Higher than DHCP metric (1024) to make it less preferred } ]; # IMPORTANT: systemd-networkd Route syntax requirements: # - Use "0.0.0.0/0" for IPv4 default routes (NOT "default") # - Use "::/0" for IPv6 default routes (NOT "default") # - "default" is NOT a valid Destination value in systemd-networkd # - Reference: https://www.freedesktop.org/software/systemd/man/latest/systemd.network.htm= l linkConfig =3D { #RequiredForOnline =3D "carrier"; MTUBytes =3D MTUBytes; }; vlan =3D vlanNames; inherit cakeConfig; <------------------------ CAKE }; # Bond device configuration systemd.network.netdevs =3D { "bond0" =3D { netdevConfig =3D { Name =3D bondConfig.Name; Kind =3D "bond"; MTUBytes =3D MTUBytes; }; bondConfig =3D { Mode =3D "802.3ad"; # MIIMonitorSec =3D "100ms"; LACPTransmitRate =3D "fast"; # fast is only 1 second, so it's not really very fast :) TransmitHashPolicy =3D "layer3+4"; }; }; # VLAN devices - generated from vlanConfigs } // lib.foldl' (acc: name: acc // createVlanNetdev name vlanConfigs.${name}) {} (lib.attrNames vlanConfigs); } Regards, Dave Seddon On Sun, Sep 28, 2025 at 5:55=E2=80=AFAM Frantisek Borsik wrote: > > Jaap - thank you very much for recommending LibreQoS. > > Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in > various other setups, like to help with event Wi-Fi - here is one of the > deployments https://libreqos.io/2023/11/13/success-story-raceway/ > > To brag a little, other deployment scenarios we see are data centers, > cruise ships, university campuses, even mines :) > > All the best, > > Frank > > Frantisek (Frank) Borsik > > > *In loving memory of Dave T=C3=A4ht: *1965-2025 > > https://libreqos.io/2025/04/01/in-loving-memory-of-dave/ > > > https://www.linkedin.com/in/frantisekborsik > > Signal, Telegram, WhatsApp: +421919416714 > > iMessage, mobile: +420775230885 > > Skype: casioa5302ca > > frantisek.borsik@gmail.com > > > On Sun, Sep 28, 2025 at 2:38=E2=80=AFPM David Lang wrote: > > > I guess I didn't give enough info on our network architecture. > > > > The upstream ISP is between 500Mb and 1G. > > > > We disconnect the conference center firewall and connect our firewall i= n > > it's > > place (Linux system) > > > > This border router connect to the conference center firewall (so their > > offices > > still get connectivity), and to two routers (Linux, one per building) t= hat > > are > > also direct connected (a triangle between the three routers. The border > > router > > is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through = a > > tunnel > > (google doesn't do PMTU discovery on IPv6) > > > > (we are currently debating between deploying OSPF between the three > > routers, or > > just giving them static routes, high priority direct and low priority f= or > > the > > indirect path) > > > > Each building router then connects to one or a few switches in differen= t > > closets > > that then go to a switch in each room (trunked ports). > > We hve a handful of wired users, 120APs for the users, and AV equipment > > for each > > of the 16 tracks that are recorded and streamed. > > > > a quick look at libreqos and it's pushing for a more complex network > > layout that > > will be a hard sell at least this year (last year our juniper routers f= ell > > over > > under the MSS squashing load for IPv6, so we are going to Linux systems > > this > > year, getting people to consider cake is hard enough, let alone another > > middleware box) > > > > David Lang > > > > On Sun, 28 Sep 2025, Jaap de Vos wrote: > > > > > Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST) > > > From: Jaap de Vos > > > Reply-To: "cake@lists.bufferbloat.net" > > > To: David Lang , cake@lists.bufferbloat.net > > > Subject: Re: [Cake] help request for cake on a large network > > > > > > Hi David, > > > > > > I haven't tried it myself yet, but this sounds like a use case for wh= ich > > LibreQoS is suitable. However, LibreQoS would fit best if you can build > > something in line with the design assumptions: > > https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html > > > Specifically the part about putting it in between other routers and > > running a routing protocol with those. > > > > > > There are some good recommendations around suitable hardware and > > sizing/scaling too: > > https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html > > > > > > What kind of upstream bandwidth are you working with at the conferenc= e? > > I run the Mikrotik implementation of CAKE on a campsite. It really help= s to > > get the most out of our asymmetrical gigabit coax line over there, no f= iber > > available yet. So far, we got close to 500 devices on our outdoor Wi-Fi > > network with mostly good experiences, amazing. Before that, FQ_CoDel on > > PfSense worked well too, but the network wasn't as busy back then. I'd = like > > to see CAKE on BSD some day. > > > > > > Kind regards, > > > Jaap de Vos > > > > > > > > >> On 09/28/2025 1:06 PM CEST David Lang wrote: > > >> > > >> > > >> I'm starting to prepare for the next Scale conference and we are > > switching from > > >> Juniper routers to Linux routers. This gives me the ability to > > implement cake. > > >> > > >> One problem we have is classes that tell everyone 'go download this' > > that > > >> trigger hundreds of people to hammer the network at the same time (t= his > > is both > > >> a wifi and a network bandwidth issue, wifi is being worked on) > > >> > > >> The network is pretty flat, a couple of subnets each on ipv4 and ipv= 6. > > >> > > >> Any suggestions on how to configure cake for this sort of environmen= t > > where > > >> there are so many devices? > > >> > > >> David Lang > > >> _______________________________________________ > > >> Cake mailing list -- cake@lists.bufferbloat.net > > >> To unsubscribe send an email to cake-leave@lists.bufferbloat.net > > > > > _______________________________________________ > > Cake mailing list -- cake@lists.bufferbloat.net > > To unsubscribe send an email to cake-leave@lists.bufferbloat.net > > > _______________________________________________ > Cake mailing list -- cake@lists.bufferbloat.net > To unsubscribe send an email to cake-leave@lists.bufferbloat.net --=20 Regards, Dave Seddon +1 415 857 5102