[Cake] Re: help request for cake on a large network

[David Lang <david@lang.hm>]

dave seddon wrote:

> G'day David,
>
> This sounds great and I'd be very happy to discuss this in real time
> to give you a hand.  The SCALE team does a lot of Nix, right?

Yes

> I think all the OpenWrt One routers were configured with nix, and so I assume 
> you're going to go with Nix for this?

they openwrt one routers were configured with a custom build of openwrt (we do 
have someone working on bringing nix up on them)

these routers will probably be running nix

> I think the TLDR is that cake will mostly do what you want.  The idea
> being that "it is a piece of cake" to use cake. ;)

that's what I expect, I just expect to need tuning of things to handle the large 
number of endpoints, what is appropriate for a dozen users will probably need 
larger tables to handle 3000 users :-)

> I was recently configuring an active/standby firewall, including
> configuring cake, with Nix, which I really need to 1) put config on
> github 2) document 3) and make a video about.

looking forward to it.

David Lang

>
> One nice thing I worked out is that the systemd networkd makes it
> super easy to apply the qdisc of choice.
>
> This isn't the whole config, but hopefully gives you the vague idea
>
>  cakeConfig = {
>    Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
>    OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
>    MPUBytes = 84;       # Minimum packet unit for Ethernet
>    NAT = true;
>    FlowIsolationMode = "triple";
>    PriorityQueueingPreset = "besteffort";
>  };
>
> systemd.network.networks = {
>
>    # Bond interface configuration
>    "bond0" = {
>      matchConfig.Name = bondConfig.Name;
>      networkConfig = {
>        Address = [
>          (constructIP bondConfig.subnet4 (getNodeAddress bondConfig))
>          #bondConfig.vrrp_ip # secondary IP
>          #fd00::1/64" #FIXME!!
>        ];
>        LinkLocalAddressing = "no";
>        # Note: Gateway is configured in routes section with high metric
>        # to make it less preferred than DHCP route (metric 1024)
>      };
>      routes = [
>        {
>          Gateway = bondConfig.gateway_ip;
>          Destination = "0.0.0.0/0"; # Default route (all destinations)
>          Metric = 2000; # Higher than DHCP metric (1024) to make it
> less preferred
>        }
>      ];
>      # IMPORTANT: systemd-networkd Route syntax requirements:
>      # - Use "0.0.0.0/0" for IPv4 default routes (NOT "default")
>      # - Use "::/0" for IPv6 default routes (NOT "default")
>      # - "default" is NOT a valid Destination value in systemd-networkd
>      # - Reference:
> https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html
>      linkConfig = {
>        #RequiredForOnline = "carrier";
>        MTUBytes = MTUBytes;
>      };
>      vlan = vlanNames;
>      inherit cakeConfig;
> <------------------------ CAKE
>    };
>
>  # Bond device configuration
>  systemd.network.netdevs = {
>    "bond0" = {
>      netdevConfig = {
>        Name = bondConfig.Name;
>        Kind = "bond";
>        MTUBytes = MTUBytes;
>      };
>      bondConfig = {
>        Mode = "802.3ad";
>        # MIIMonitorSec = "100ms";
>        LACPTransmitRate = "fast"; # fast is only 1 second, so it's
> not really very fast :)
>        TransmitHashPolicy = "layer3+4";
>      };
>    };
>
>    # VLAN devices - generated from vlanConfigs
>  } // lib.foldl' (acc: name: acc // createVlanNetdev name
> vlanConfigs.${name}) {} (lib.attrNames vlanConfigs);
>
> }
>
> Regards,
> Dave Seddon
>
> On Sun, Sep 28, 2025 at 5:55 AM Frantisek Borsik
> <frantisek.borsik@gmail.com> wrote:
>>
>> Jaap - thank you very much for recommending LibreQoS.
>>
>> Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in
>> various other setups, like to help with event Wi-Fi - here is one of the
>> deployments https://libreqos.io/2023/11/13/success-story-raceway/
>>
>> To brag a little, other deployment scenarios we see are data centers,
>> cruise ships, university campuses, even mines :)
>>
>> All the best,
>>
>> Frank
>>
>> Frantisek (Frank) Borsik
>>
>>
>> *In loving memory of Dave Täht: *1965-2025
>>
>> https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
>>
>>
>> https://www.linkedin.com/in/frantisekborsik
>>
>> Signal, Telegram, WhatsApp: +421919416714
>>
>> iMessage, mobile: +420775230885
>>
>> Skype: casioa5302ca
>>
>> frantisek.borsik@gmail.com
>>
>>
>> On Sun, Sep 28, 2025 at 2:38 PM David Lang <david@lang.hm> wrote:
>>
>>> I guess I didn't give enough info on our network architecture.
>>>
>>> The upstream ISP is between 500Mb and 1G.
>>>
>>> We disconnect the conference center firewall and connect our firewall in
>>> it's
>>> place (Linux system)
>>>
>>> This border router connect to the conference center firewall (so their
>>> offices
>>> still get connectivity), and to two routers (Linux, one per building) that
>>> are
>>> also direct connected (a triangle between the three routers. The border
>>> router
>>> is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a
>>> tunnel
>>> (google doesn't do PMTU discovery on IPv6)
>>>
>>> (we are currently debating between deploying OSPF between the three
>>> routers, or
>>> just giving them static routes, high priority direct and low priority for
>>> the
>>> indirect path)
>>>
>>> Each building router then connects to one or a few switches in different
>>> closets
>>> that then go to a switch in each room (trunked ports).
>>> We hve a handful of wired users, 120APs for the users, and AV equipment
>>> for each
>>> of the 16 tracks that are recorded and streamed.
>>>
>>> a quick look at libreqos and it's pushing for a more complex network
>>> layout that
>>> will be a hard sell at least this year (last year our juniper routers fell
>>> over
>>> under the MSS squashing load for IPv6, so we are going to Linux systems
>>> this
>>> year, getting people to consider cake is hard enough, let alone another
>>> middleware box)
>>>
>>> David Lang
>>>
>>> On Sun, 28 Sep 2025, Jaap de Vos wrote:
>>>
>>>> Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
>>>> From: Jaap de Vos <m@jaap.pro>
>>>> Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
>>>> To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
>>>> Subject: Re: [Cake] help request for cake on a large network
>>>>
>>>> Hi David,
>>>>
>>>> I haven't tried it myself yet, but this sounds like a use case for which
>>> LibreQoS is suitable. However, LibreQoS would fit best if you can build
>>> something in line with the design assumptions:
>>> https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
>>>> Specifically the part about putting it in between other routers and
>>> running a routing protocol with those.
>>>>
>>>> There are some good recommendations around suitable hardware and
>>> sizing/scaling too:
>>> https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
>>>>
>>>> What kind of upstream bandwidth are you working with at the conference?
>>> I run the Mikrotik implementation of CAKE on a campsite. It really helps to
>>> get the most out of our asymmetrical gigabit coax line over there, no fiber
>>> available yet. So far, we got close to 500 devices on our outdoor Wi-Fi
>>> network with mostly good experiences, amazing. Before that, FQ_CoDel on
>>> PfSense worked well too, but the network wasn't as busy back then. I'd like
>>> to see CAKE on BSD some day.
>>>>
>>>> Kind regards,
>>>> Jaap de Vos
>>>>
>>>>
>>>>> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
>>>>>
>>>>>
>>>>> I'm starting to prepare for the next Scale conference and we are
>>> switching from
>>>>> Juniper routers to Linux routers. This gives me the ability to
>>> implement cake.
>>>>>
>>>>> One problem we have is classes that tell everyone 'go download this'
>>> that
>>>>> trigger hundreds of people to hammer the network at the same time (this
>>> is both
>>>>> a wifi and a network bandwidth issue, wifi is being worked on)
>>>>>
>>>>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>>>>
>>>>> Any suggestions on how to configure cake for this sort of environment
>>> where
>>>>> there are so many devices?
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> Cake mailing list -- cake@lists.bufferbloat.net
>>>>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>>>>
>>> _______________________________________________
>>> Cake mailing list -- cake@lists.bufferbloat.net
>>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>>>
>> _______________________________________________
>> Cake mailing list -- cake@lists.bufferbloat.net
>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>
>
>
>