[Cake] Re: help request for cake on a large network

[dave seddon <dave.seddon.ca@gmail.com>]

G'day David,

This sounds great and I'd be very happy to discuss this in real time
to give you a hand.  The SCALE team does a lot of Nix, right?  I think
all the OpenWrt One routers were configured with nix, and so I assume
you're going to go with Nix for this?

I think the TLDR is that cake will mostly do what you want.  The idea
being that "it is a piece of cake" to use cake. ;)


I was recently configuring an active/standby firewall, including
configuring cake, with Nix, which I really need to 1) put config on
github 2) document 3) and make a video about.


One nice thing I worked out is that the systemd networkd makes it
super easy to apply the qdisc of choice.

This isn't the whole config, but hopefully gives you the vague idea

  cakeConfig = {
    Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
    OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
    MPUBytes = 84;       # Minimum packet unit for Ethernet
    NAT = true;
    FlowIsolationMode = "triple";
    PriorityQueueingPreset = "besteffort";
  };

systemd.network.networks = {

    # Bond interface configuration
    "bond0" = {
      matchConfig.Name = bondConfig.Name;
      networkConfig = {
        Address = [
          (constructIP bondConfig.subnet4 (getNodeAddress bondConfig))
          #bondConfig.vrrp_ip # secondary IP
          #fd00::1/64" #FIXME!!
        ];
        LinkLocalAddressing = "no";
        # Note: Gateway is configured in routes section with high metric
        # to make it less preferred than DHCP route (metric 1024)
      };
      routes = [
        {
          Gateway = bondConfig.gateway_ip;
          Destination = "0.0.0.0/0"; # Default route (all destinations)
          Metric = 2000; # Higher than DHCP metric (1024) to make it
less preferred
        }
      ];
      # IMPORTANT: systemd-networkd Route syntax requirements:
      # - Use "0.0.0.0/0" for IPv4 default routes (NOT "default")
      # - Use "::/0" for IPv6 default routes (NOT "default")
      # - "default" is NOT a valid Destination value in systemd-networkd
      # - Reference:
https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html
      linkConfig = {
        #RequiredForOnline = "carrier";
        MTUBytes = MTUBytes;
      };
      vlan = vlanNames;
      inherit cakeConfig;
<------------------------ CAKE
    };

  # Bond device configuration
  systemd.network.netdevs = {
    "bond0" = {
      netdevConfig = {
        Name = bondConfig.Name;
        Kind = "bond";
        MTUBytes = MTUBytes;
      };
      bondConfig = {
        Mode = "802.3ad";
        # MIIMonitorSec = "100ms";
        LACPTransmitRate = "fast"; # fast is only 1 second, so it's
not really very fast :)
        TransmitHashPolicy = "layer3+4";
      };
    };

    # VLAN devices - generated from vlanConfigs
  } // lib.foldl' (acc: name: acc // createVlanNetdev name
vlanConfigs.${name}) {} (lib.attrNames vlanConfigs);

}

Regards,
Dave Seddon

On Sun, Sep 28, 2025 at 5:55 AM Frantisek Borsik
<frantisek.borsik@gmail.com> wrote:
>
> Jaap - thank you very much for recommending LibreQoS.
>
> Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in
> various other setups, like to help with event Wi-Fi - here is one of the
> deployments https://libreqos.io/2023/11/13/success-story-raceway/
>
> To brag a little, other deployment scenarios we see are data centers,
> cruise ships, university campuses, even mines :)
>
> All the best,
>
> Frank
>
> Frantisek (Frank) Borsik
>
>
> *In loving memory of Dave Täht: *1965-2025
>
> https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
>
>
> https://www.linkedin.com/in/frantisekborsik
>
> Signal, Telegram, WhatsApp: +421919416714
>
> iMessage, mobile: +420775230885
>
> Skype: casioa5302ca
>
> frantisek.borsik@gmail.com
>
>
> On Sun, Sep 28, 2025 at 2:38 PM David Lang <david@lang.hm> wrote:
>
> > I guess I didn't give enough info on our network architecture.
> >
> > The upstream ISP is between 500Mb and 1G.
> >
> > We disconnect the conference center firewall and connect our firewall in
> > it's
> > place (Linux system)
> >
> > This border router connect to the conference center firewall (so their
> > offices
> > still get connectivity), and to two routers (Linux, one per building) that
> > are
> > also direct connected (a triangle between the three routers. The border
> > router
> > is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a
> > tunnel
> > (google doesn't do PMTU discovery on IPv6)
> >
> > (we are currently debating between deploying OSPF between the three
> > routers, or
> > just giving them static routes, high priority direct and low priority for
> > the
> > indirect path)
> >
> > Each building router then connects to one or a few switches in different
> > closets
> > that then go to a switch in each room (trunked ports).
> > We hve a handful of wired users, 120APs for the users, and AV equipment
> > for each
> > of the 16 tracks that are recorded and streamed.
> >
> > a quick look at libreqos and it's pushing for a more complex network
> > layout that
> > will be a hard sell at least this year (last year our juniper routers fell
> > over
> > under the MSS squashing load for IPv6, so we are going to Linux systems
> > this
> > year, getting people to consider cake is hard enough, let alone another
> > middleware box)
> >
> > David Lang
> >
> > On Sun, 28 Sep 2025, Jaap de Vos wrote:
> >
> > > Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
> > > From: Jaap de Vos <m@jaap.pro>
> > > Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
> > > To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
> > > Subject: Re: [Cake] help request for cake on a large network
> > >
> > > Hi David,
> > >
> > > I haven't tried it myself yet, but this sounds like a use case for which
> > LibreQoS is suitable. However, LibreQoS would fit best if you can build
> > something in line with the design assumptions:
> > https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
> > > Specifically the part about putting it in between other routers and
> > running a routing protocol with those.
> > >
> > > There are some good recommendations around suitable hardware and
> > sizing/scaling too:
> > https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
> > >
> > > What kind of upstream bandwidth are you working with at the conference?
> > I run the Mikrotik implementation of CAKE on a campsite. It really helps to
> > get the most out of our asymmetrical gigabit coax line over there, no fiber
> > available yet. So far, we got close to 500 devices on our outdoor Wi-Fi
> > network with mostly good experiences, amazing. Before that, FQ_CoDel on
> > PfSense worked well too, but the network wasn't as busy back then. I'd like
> > to see CAKE on BSD some day.
> > >
> > > Kind regards,
> > > Jaap de Vos
> > >
> > >
> > >> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
> > >>
> > >>
> > >> I'm starting to prepare for the next Scale conference and we are
> > switching from
> > >> Juniper routers to Linux routers. This gives me the ability to
> > implement cake.
> > >>
> > >> One problem we have is classes that tell everyone 'go download this'
> > that
> > >> trigger hundreds of people to hammer the network at the same time (this
> > is both
> > >> a wifi and a network bandwidth issue, wifi is being worked on)
> > >>
> > >> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> > >>
> > >> Any suggestions on how to configure cake for this sort of environment
> > where
> > >> there are so many devices?
> > >>
> > >> David Lang
> > >> _______________________________________________
> > >> Cake mailing list -- cake@lists.bufferbloat.net
> > >> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
> > >
> > _______________________________________________
> > Cake mailing list -- cake@lists.bufferbloat.net
> > To unsubscribe send an email to cake-leave@lists.bufferbloat.net
> >
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net



-- 
Regards,
Dave Seddon
+1 415 857 5102