[Cake] help request for cake on a large network

[Cake] help request for cake on a large network

[David Lang]

I'm starting to prepare for the next Scale conference and we are switching from 
Juniper routers to Linux routers. This gives me the ability to implement cake.

One problem we have is classes that tell everyone 'go download this' that 
trigger hundreds of people to hammer the network at the same time (this is both 
a wifi and a network bandwidth issue, wifi is being worked on)

The network is pretty flat, a couple of subnets each on ipv4 and ipv6.

Any suggestions on how to configure cake for this sort of environment where 
there are so many devices?

David Lang

[Cake] Re: help request for cake on a large network

[Sebastian Moeller]

Hi David,

while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.



> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
> 
> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
> 
> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)

So one issue might be that with several 100 users the default compile-time size of queues (1024, IIRC) that cake will entertain might be too little, even in light of the 8 way assoziative hashing design. I believe this can be changed (within limits) only by modifying at source and recompilation of the kernel, if that should be needed at all.

I wonder whether multi-queue cake would not solve this to some degree, as I assume each queue's instance would bring its own independent set of 1024 bins?


> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> 
> Any suggestions on how to configure cake for this sort of environment where there are so many devices?

Maybe switch to a simpler pure per-flow isolation mode than the default triple-isolate?

BTW what kind of uplink capacity will you use?

> 
> David Lang
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net

[Cake] Re: help request for cake on a large network

[Jaap de Vos]

Hi David,

I haven't tried it myself yet, but this sounds like a use case for which LibreQoS is suitable. However, LibreQoS would fit best if you can build something in line with the design assumptions: https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
Specifically the part about putting it in between other routers and running a routing protocol with those.

There are some good recommendations around suitable hardware and sizing/scaling too: https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html

What kind of upstream bandwidth are you working with at the conference? I run the Mikrotik implementation of CAKE on a campsite. It really helps to get the most out of our asymmetrical gigabit coax line over there, no fiber available yet. So far, we got close to 500 devices on our outdoor Wi-Fi network with mostly good experiences, amazing. Before that, FQ_CoDel on PfSense worked well too, but the network wasn't as busy back then. I'd like to see CAKE on BSD some day.

Kind regards,
Jaap de Vos


> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
> 
>  
> I'm starting to prepare for the next Scale conference and we are switching from 
> Juniper routers to Linux routers. This gives me the ability to implement cake.
> 
> One problem we have is classes that tell everyone 'go download this' that 
> trigger hundreds of people to hammer the network at the same time (this is both 
> a wifi and a network bandwidth issue, wifi is being worked on)
> 
> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> 
> Any suggestions on how to configure cake for this sort of environment where 
> there are so many devices?
> 
> David Lang
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net

[Cake] Re: help request for cake on a large network

[David Lang]

Sebastian Moeller wrote:

> Hi David,
>
> while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.

I will do that, however the network is only up under load for 4 days a year, so 
it's a slow feedback loop :-)

I would welcome any other suggestions for data to gather.

>> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
>>
>> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
>>
>> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)
>

> So one issue might be that with several 100 users the default compile-time 
> size of queues (1024, IIRC) that cake will entertain might be too little, even 
> in light of the 8 way assoziative hashing design. I believe this can be 
> changed (within limits) only by modifying at source and recompilation of the 
> kernel, if that should be needed at all.

custom compiling a kernel is very much an option (and this sort of tweaking is 
the sort of thing I'm expecting to need to do)

The conference is in March, so we have some time to think about this and 
customize things, just no chance to test before the show.

> I wonder whether multi-queue cake would not solve this to some degree, as I 
> assume each queue's instance would bring its own independent set of 1024 bins?

good thought


>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>
>> Any suggestions on how to configure cake for this sort of environment where there are so many devices?
>
> Maybe switch to a simpler pure per-flow isolation mode than the default triple-isolate?
>
> BTW what kind of uplink capacity will you use?

last year we had something around 500Mb, they are in the process of changing 
ISPs and we may get a bit more, but probably still under 1G

hotel and conference center Internet connections are far smaller than most 
people think.

David Lang

[Cake] Re: help request for cake on a large network

[David Lang]

I guess I didn't give enough info on our network architecture.

The upstream ISP is between 500Mb and 1G.

We disconnect the conference center firewall and connect our firewall in it's 
place (Linux system)

This border router connect to the conference center firewall (so their offices 
still get connectivity), and to two routers (Linux, one per building) that are 
also direct connected (a triangle between the three routers. The border router 
is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a tunnel 
(google doesn't do PMTU discovery on IPv6)

(we are currently debating between deploying OSPF between the three routers, or 
just giving them static routes, high priority direct and low priority for the 
indirect path)

Each building router then connects to one or a few switches in different closets 
that then go to a switch in each room (trunked ports).
We hve a handful of wired users, 120APs for the users, and AV equipment for each 
of the 16 tracks that are recorded and streamed.

a quick look at libreqos and it's pushing for a more complex network layout that 
will be a hard sell at least this year (last year our juniper routers fell over 
under the MSS squashing load for IPv6, so we are going to Linux systems this 
year, getting people to consider cake is hard enough, let alone another 
middleware box)

David Lang

On Sun, 28 Sep 2025, Jaap de Vos wrote:

> Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
> From: Jaap de Vos <m@jaap.pro>
> Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
> To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
> Subject: Re: [Cake] help request for cake on a large network
> 
> Hi David,
>
> I haven't tried it myself yet, but this sounds like a use case for which LibreQoS is suitable. However, LibreQoS would fit best if you can build something in line with the design assumptions: https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
> Specifically the part about putting it in between other routers and running a routing protocol with those.
>
> There are some good recommendations around suitable hardware and sizing/scaling too: https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
>
> What kind of upstream bandwidth are you working with at the conference? I run the Mikrotik implementation of CAKE on a campsite. It really helps to get the most out of our asymmetrical gigabit coax line over there, no fiber available yet. So far, we got close to 500 devices on our outdoor Wi-Fi network with mostly good experiences, amazing. Before that, FQ_CoDel on PfSense worked well too, but the network wasn't as busy back then. I'd like to see CAKE on BSD some day.
>
> Kind regards,
> Jaap de Vos
>
>
>> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
>>
>>
>> I'm starting to prepare for the next Scale conference and we are switching from
>> Juniper routers to Linux routers. This gives me the ability to implement cake.
>>
>> One problem we have is classes that tell everyone 'go download this' that
>> trigger hundreds of people to hammer the network at the same time (this is both
>> a wifi and a network bandwidth issue, wifi is being worked on)
>>
>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>
>> Any suggestions on how to configure cake for this sort of environment where
>> there are so many devices?
>>
>> David Lang
>> _______________________________________________
>> Cake mailing list -- cake@lists.bufferbloat.net
>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>

[Cake] Re: help request for cake on a large network

[Frantisek Borsik]

Jaap - thank you very much for recommending LibreQoS.

Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in
various other setups, like to help with event Wi-Fi - here is one of the
deployments https://libreqos.io/2023/11/13/success-story-raceway/

To brag a little, other deployment scenarios we see are data centers,
cruise ships, university campuses, even mines :)

All the best,

Frank

Frantisek (Frank) Borsik


*In loving memory of Dave Täht: *1965-2025

https://libreqos.io/2025/04/01/in-loving-memory-of-dave/


https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Sun, Sep 28, 2025 at 2:38 PM David Lang <david@lang.hm> wrote:

> I guess I didn't give enough info on our network architecture.
>
> The upstream ISP is between 500Mb and 1G.
>
> We disconnect the conference center firewall and connect our firewall in
> it's
> place (Linux system)
>
> This border router connect to the conference center firewall (so their
> offices
> still get connectivity), and to two routers (Linux, one per building) that
> are
> also direct connected (a triangle between the three routers. The border
> router
> is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a
> tunnel
> (google doesn't do PMTU discovery on IPv6)
>
> (we are currently debating between deploying OSPF between the three
> routers, or
> just giving them static routes, high priority direct and low priority for
> the
> indirect path)
>
> Each building router then connects to one or a few switches in different
> closets
> that then go to a switch in each room (trunked ports).
> We hve a handful of wired users, 120APs for the users, and AV equipment
> for each
> of the 16 tracks that are recorded and streamed.
>
> a quick look at libreqos and it's pushing for a more complex network
> layout that
> will be a hard sell at least this year (last year our juniper routers fell
> over
> under the MSS squashing load for IPv6, so we are going to Linux systems
> this
> year, getting people to consider cake is hard enough, let alone another
> middleware box)
>
> David Lang
>
> On Sun, 28 Sep 2025, Jaap de Vos wrote:
>
> > Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
> > From: Jaap de Vos <m@jaap.pro>
> > Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
> > To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
> > Subject: Re: [Cake] help request for cake on a large network
> >
> > Hi David,
> >
> > I haven't tried it myself yet, but this sounds like a use case for which
> LibreQoS is suitable. However, LibreQoS would fit best if you can build
> something in line with the design assumptions:
> https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
> > Specifically the part about putting it in between other routers and
> running a routing protocol with those.
> >
> > There are some good recommendations around suitable hardware and
> sizing/scaling too:
> https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
> >
> > What kind of upstream bandwidth are you working with at the conference?
> I run the Mikrotik implementation of CAKE on a campsite. It really helps to
> get the most out of our asymmetrical gigabit coax line over there, no fiber
> available yet. So far, we got close to 500 devices on our outdoor Wi-Fi
> network with mostly good experiences, amazing. Before that, FQ_CoDel on
> PfSense worked well too, but the network wasn't as busy back then. I'd like
> to see CAKE on BSD some day.
> >
> > Kind regards,
> > Jaap de Vos
> >
> >
> >> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
> >>
> >>
> >> I'm starting to prepare for the next Scale conference and we are
> switching from
> >> Juniper routers to Linux routers. This gives me the ability to
> implement cake.
> >>
> >> One problem we have is classes that tell everyone 'go download this'
> that
> >> trigger hundreds of people to hammer the network at the same time (this
> is both
> >> a wifi and a network bandwidth issue, wifi is being worked on)
> >>
> >> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> >>
> >> Any suggestions on how to configure cake for this sort of environment
> where
> >> there are so many devices?
> >>
> >> David Lang
> >> _______________________________________________
> >> Cake mailing list -- cake@lists.bufferbloat.net
> >> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
> >
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>

[Cake] Re: help request for cake on a large network

[dave seddon]

G'day David,

This sounds great and I'd be very happy to discuss this in real time
to give you a hand.  The SCALE team does a lot of Nix, right?  I think
all the OpenWrt One routers were configured with nix, and so I assume
you're going to go with Nix for this?

I think the TLDR is that cake will mostly do what you want.  The idea
being that "it is a piece of cake" to use cake. ;)


I was recently configuring an active/standby firewall, including
configuring cake, with Nix, which I really need to 1) put config on
github 2) document 3) and make a video about.


One nice thing I worked out is that the systemd networkd makes it
super easy to apply the qdisc of choice.

This isn't the whole config, but hopefully gives you the vague idea

  cakeConfig = {
    Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
    OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
    MPUBytes = 84;       # Minimum packet unit for Ethernet
    NAT = true;
    FlowIsolationMode = "triple";
    PriorityQueueingPreset = "besteffort";
  };

systemd.network.networks = {

    # Bond interface configuration
    "bond0" = {
      matchConfig.Name = bondConfig.Name;
      networkConfig = {
        Address = [
          (constructIP bondConfig.subnet4 (getNodeAddress bondConfig))
          #bondConfig.vrrp_ip # secondary IP
          #fd00::1/64" #FIXME!!
        ];
        LinkLocalAddressing = "no";
        # Note: Gateway is configured in routes section with high metric
        # to make it less preferred than DHCP route (metric 1024)
      };
      routes = [
        {
          Gateway = bondConfig.gateway_ip;
          Destination = "0.0.0.0/0"; # Default route (all destinations)
          Metric = 2000; # Higher than DHCP metric (1024) to make it
less preferred
        }
      ];
      # IMPORTANT: systemd-networkd Route syntax requirements:
      # - Use "0.0.0.0/0" for IPv4 default routes (NOT "default")
      # - Use "::/0" for IPv6 default routes (NOT "default")
      # - "default" is NOT a valid Destination value in systemd-networkd
      # - Reference:
https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html
      linkConfig = {
        #RequiredForOnline = "carrier";
        MTUBytes = MTUBytes;
      };
      vlan = vlanNames;
      inherit cakeConfig;
<------------------------ CAKE
    };

  # Bond device configuration
  systemd.network.netdevs = {
    "bond0" = {
      netdevConfig = {
        Name = bondConfig.Name;
        Kind = "bond";
        MTUBytes = MTUBytes;
      };
      bondConfig = {
        Mode = "802.3ad";
        # MIIMonitorSec = "100ms";
        LACPTransmitRate = "fast"; # fast is only 1 second, so it's
not really very fast :)
        TransmitHashPolicy = "layer3+4";
      };
    };

    # VLAN devices - generated from vlanConfigs
  } // lib.foldl' (acc: name: acc // createVlanNetdev name
vlanConfigs.${name}) {} (lib.attrNames vlanConfigs);

}

Regards,
Dave Seddon

On Sun, Sep 28, 2025 at 5:55 AM Frantisek Borsik
<frantisek.borsik@gmail.com> wrote:
>
> Jaap - thank you very much for recommending LibreQoS.
>
> Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in
> various other setups, like to help with event Wi-Fi - here is one of the
> deployments https://libreqos.io/2023/11/13/success-story-raceway/
>
> To brag a little, other deployment scenarios we see are data centers,
> cruise ships, university campuses, even mines :)
>
> All the best,
>
> Frank
>
> Frantisek (Frank) Borsik
>
>
> *In loving memory of Dave Täht: *1965-2025
>
> https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
>
>
> https://www.linkedin.com/in/frantisekborsik
>
> Signal, Telegram, WhatsApp: +421919416714
>
> iMessage, mobile: +420775230885
>
> Skype: casioa5302ca
>
> frantisek.borsik@gmail.com
>
>
> On Sun, Sep 28, 2025 at 2:38 PM David Lang <david@lang.hm> wrote:
>
> > I guess I didn't give enough info on our network architecture.
> >
> > The upstream ISP is between 500Mb and 1G.
> >
> > We disconnect the conference center firewall and connect our firewall in
> > it's
> > place (Linux system)
> >
> > This border router connect to the conference center firewall (so their
> > offices
> > still get connectivity), and to two routers (Linux, one per building) that
> > are
> > also direct connected (a triangle between the three routers. The border
> > router
> > is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a
> > tunnel
> > (google doesn't do PMTU discovery on IPv6)
> >
> > (we are currently debating between deploying OSPF between the three
> > routers, or
> > just giving them static routes, high priority direct and low priority for
> > the
> > indirect path)
> >
> > Each building router then connects to one or a few switches in different
> > closets
> > that then go to a switch in each room (trunked ports).
> > We hve a handful of wired users, 120APs for the users, and AV equipment
> > for each
> > of the 16 tracks that are recorded and streamed.
> >
> > a quick look at libreqos and it's pushing for a more complex network
> > layout that
> > will be a hard sell at least this year (last year our juniper routers fell
> > over
> > under the MSS squashing load for IPv6, so we are going to Linux systems
> > this
> > year, getting people to consider cake is hard enough, let alone another
> > middleware box)
> >
> > David Lang
> >
> > On Sun, 28 Sep 2025, Jaap de Vos wrote:
> >
> > > Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
> > > From: Jaap de Vos <m@jaap.pro>
> > > Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
> > > To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
> > > Subject: Re: [Cake] help request for cake on a large network
> > >
> > > Hi David,
> > >
> > > I haven't tried it myself yet, but this sounds like a use case for which
> > LibreQoS is suitable. However, LibreQoS would fit best if you can build
> > something in line with the design assumptions:
> > https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
> > > Specifically the part about putting it in between other routers and
> > running a routing protocol with those.
> > >
> > > There are some good recommendations around suitable hardware and
> > sizing/scaling too:
> > https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
> > >
> > > What kind of upstream bandwidth are you working with at the conference?
> > I run the Mikrotik implementation of CAKE on a campsite. It really helps to
> > get the most out of our asymmetrical gigabit coax line over there, no fiber
> > available yet. So far, we got close to 500 devices on our outdoor Wi-Fi
> > network with mostly good experiences, amazing. Before that, FQ_CoDel on
> > PfSense worked well too, but the network wasn't as busy back then. I'd like
> > to see CAKE on BSD some day.
> > >
> > > Kind regards,
> > > Jaap de Vos
> > >
> > >
> > >> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
> > >>
> > >>
> > >> I'm starting to prepare for the next Scale conference and we are
> > switching from
> > >> Juniper routers to Linux routers. This gives me the ability to
> > implement cake.
> > >>
> > >> One problem we have is classes that tell everyone 'go download this'
> > that
> > >> trigger hundreds of people to hammer the network at the same time (this
> > is both
> > >> a wifi and a network bandwidth issue, wifi is being worked on)
> > >>
> > >> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> > >>
> > >> Any suggestions on how to configure cake for this sort of environment
> > where
> > >> there are so many devices?
> > >>
> > >> David Lang
> > >> _______________________________________________
> > >> Cake mailing list -- cake@lists.bufferbloat.net
> > >> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
> > >
> > _______________________________________________
> > Cake mailing list -- cake@lists.bufferbloat.net
> > To unsubscribe send an email to cake-leave@lists.bufferbloat.net
> >
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net



-- 
Regards,
Dave Seddon
+1 415 857 5102

[Cake] Re: help request for cake on a large network

[David Lang]

dave seddon wrote:

> G'day David,
>
> This sounds great and I'd be very happy to discuss this in real time
> to give you a hand.  The SCALE team does a lot of Nix, right?

Yes

> I think all the OpenWrt One routers were configured with nix, and so I assume 
> you're going to go with Nix for this?

they openwrt one routers were configured with a custom build of openwrt (we do 
have someone working on bringing nix up on them)

these routers will probably be running nix

> I think the TLDR is that cake will mostly do what you want.  The idea
> being that "it is a piece of cake" to use cake. ;)

that's what I expect, I just expect to need tuning of things to handle the large 
number of endpoints, what is appropriate for a dozen users will probably need 
larger tables to handle 3000 users :-)

> I was recently configuring an active/standby firewall, including
> configuring cake, with Nix, which I really need to 1) put config on
> github 2) document 3) and make a video about.

looking forward to it.

David Lang

>
> One nice thing I worked out is that the systemd networkd makes it
> super easy to apply the qdisc of choice.
>
> This isn't the whole config, but hopefully gives you the vague idea
>
>  cakeConfig = {
>    Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
>    OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
>    MPUBytes = 84;       # Minimum packet unit for Ethernet
>    NAT = true;
>    FlowIsolationMode = "triple";
>    PriorityQueueingPreset = "besteffort";
>  };
>
> systemd.network.networks = {
>
>    # Bond interface configuration
>    "bond0" = {
>      matchConfig.Name = bondConfig.Name;
>      networkConfig = {
>        Address = [
>          (constructIP bondConfig.subnet4 (getNodeAddress bondConfig))
>          #bondConfig.vrrp_ip # secondary IP
>          #fd00::1/64" #FIXME!!
>        ];
>        LinkLocalAddressing = "no";
>        # Note: Gateway is configured in routes section with high metric
>        # to make it less preferred than DHCP route (metric 1024)
>      };
>      routes = [
>        {
>          Gateway = bondConfig.gateway_ip;
>          Destination = "0.0.0.0/0"; # Default route (all destinations)
>          Metric = 2000; # Higher than DHCP metric (1024) to make it
> less preferred
>        }
>      ];
>      # IMPORTANT: systemd-networkd Route syntax requirements:
>      # - Use "0.0.0.0/0" for IPv4 default routes (NOT "default")
>      # - Use "::/0" for IPv6 default routes (NOT "default")
>      # - "default" is NOT a valid Destination value in systemd-networkd
>      # - Reference:
> https://www.freedesktop.org/software/systemd/man/latest/systemd.network.html
>      linkConfig = {
>        #RequiredForOnline = "carrier";
>        MTUBytes = MTUBytes;
>      };
>      vlan = vlanNames;
>      inherit cakeConfig;
> <------------------------ CAKE
>    };
>
>  # Bond device configuration
>  systemd.network.netdevs = {
>    "bond0" = {
>      netdevConfig = {
>        Name = bondConfig.Name;
>        Kind = "bond";
>        MTUBytes = MTUBytes;
>      };
>      bondConfig = {
>        Mode = "802.3ad";
>        # MIIMonitorSec = "100ms";
>        LACPTransmitRate = "fast"; # fast is only 1 second, so it's
> not really very fast :)
>        TransmitHashPolicy = "layer3+4";
>      };
>    };
>
>    # VLAN devices - generated from vlanConfigs
>  } // lib.foldl' (acc: name: acc // createVlanNetdev name
> vlanConfigs.${name}) {} (lib.attrNames vlanConfigs);
>
> }
>
> Regards,
> Dave Seddon
>
> On Sun, Sep 28, 2025 at 5:55 AM Frantisek Borsik
> <frantisek.borsik@gmail.com> wrote:
>>
>> Jaap - thank you very much for recommending LibreQoS.
>>
>> Yes, besides a typical last-mile ISP scenario, LibreQoS is being used in
>> various other setups, like to help with event Wi-Fi - here is one of the
>> deployments https://libreqos.io/2023/11/13/success-story-raceway/
>>
>> To brag a little, other deployment scenarios we see are data centers,
>> cruise ships, university campuses, even mines :)
>>
>> All the best,
>>
>> Frank
>>
>> Frantisek (Frank) Borsik
>>
>>
>> *In loving memory of Dave Täht: *1965-2025
>>
>> https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
>>
>>
>> https://www.linkedin.com/in/frantisekborsik
>>
>> Signal, Telegram, WhatsApp: +421919416714
>>
>> iMessage, mobile: +420775230885
>>
>> Skype: casioa5302ca
>>
>> frantisek.borsik@gmail.com
>>
>>
>> On Sun, Sep 28, 2025 at 2:38 PM David Lang <david@lang.hm> wrote:
>>
>>> I guess I didn't give enough info on our network architecture.
>>>
>>> The upstream ISP is between 500Mb and 1G.
>>>
>>> We disconnect the conference center firewall and connect our firewall in
>>> it's
>>> place (Linux system)
>>>
>>> This border router connect to the conference center firewall (so their
>>> offices
>>> still get connectivity), and to two routers (Linux, one per building) that
>>> are
>>> also direct connected (a triangle between the three routers. The border
>>> router
>>> is doing NAT for IPv4 and MSS squashing for IPV6 that goes out through a
>>> tunnel
>>> (google doesn't do PMTU discovery on IPv6)
>>>
>>> (we are currently debating between deploying OSPF between the three
>>> routers, or
>>> just giving them static routes, high priority direct and low priority for
>>> the
>>> indirect path)
>>>
>>> Each building router then connects to one or a few switches in different
>>> closets
>>> that then go to a switch in each room (trunked ports).
>>> We hve a handful of wired users, 120APs for the users, and AV equipment
>>> for each
>>> of the 16 tracks that are recorded and streamed.
>>>
>>> a quick look at libreqos and it's pushing for a more complex network
>>> layout that
>>> will be a hard sell at least this year (last year our juniper routers fell
>>> over
>>> under the MSS squashing load for IPv6, so we are going to Linux systems
>>> this
>>> year, getting people to consider cake is hard enough, let alone another
>>> middleware box)
>>>
>>> David Lang
>>>
>>> On Sun, 28 Sep 2025, Jaap de Vos wrote:
>>>
>>>> Date: Sun, 28 Sep 2025 14:12:24 +0200 (CEST)
>>>> From: Jaap de Vos <m@jaap.pro>
>>>> Reply-To: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
>>>> To: David Lang <david@lang.hm>, cake@lists.bufferbloat.net
>>>> Subject: Re: [Cake] help request for cake on a large network
>>>>
>>>> Hi David,
>>>>
>>>> I haven't tried it myself yet, but this sounds like a use case for which
>>> LibreQoS is suitable. However, LibreQoS would fit best if you can build
>>> something in line with the design assumptions:
>>> https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html
>>>> Specifically the part about putting it in between other routers and
>>> running a routing protocol with those.
>>>>
>>>> There are some good recommendations around suitable hardware and
>>> sizing/scaling too:
>>> https://libreqos.readthedocs.io/en/latest/docs/v2.0/requirements.html
>>>>
>>>> What kind of upstream bandwidth are you working with at the conference?
>>> I run the Mikrotik implementation of CAKE on a campsite. It really helps to
>>> get the most out of our asymmetrical gigabit coax line over there, no fiber
>>> available yet. So far, we got close to 500 devices on our outdoor Wi-Fi
>>> network with mostly good experiences, amazing. Before that, FQ_CoDel on
>>> PfSense worked well too, but the network wasn't as busy back then. I'd like
>>> to see CAKE on BSD some day.
>>>>
>>>> Kind regards,
>>>> Jaap de Vos
>>>>
>>>>
>>>>> On 09/28/2025 1:06 PM CEST David Lang <david@lang.hm> wrote:
>>>>>
>>>>>
>>>>> I'm starting to prepare for the next Scale conference and we are
>>> switching from
>>>>> Juniper routers to Linux routers. This gives me the ability to
>>> implement cake.
>>>>>
>>>>> One problem we have is classes that tell everyone 'go download this'
>>> that
>>>>> trigger hundreds of people to hammer the network at the same time (this
>>> is both
>>>>> a wifi and a network bandwidth issue, wifi is being worked on)
>>>>>
>>>>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>>>>
>>>>> Any suggestions on how to configure cake for this sort of environment
>>> where
>>>>> there are so many devices?
>>>>>
>>>>> David Lang
>>>>> _______________________________________________
>>>>> Cake mailing list -- cake@lists.bufferbloat.net
>>>>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>>>>
>>> _______________________________________________
>>> Cake mailing list -- cake@lists.bufferbloat.net
>>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>>>
>> _______________________________________________
>> Cake mailing list -- cake@lists.bufferbloat.net
>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>
>
>
>

[Cake] Re: help request for cake on a large network

[Jonathan Morton]

On Sunday, 28 September 2025, David Lang wrote:
> I'm starting to prepare for the next Scale conference and we are switching from 
> Juniper routers to Linux routers. This gives me the ability to implement cake.
> 
> One problem we have is classes that tell everyone 'go download this' that 
> trigger hundreds of people to hammer the network at the same time (this is both 
> a wifi and a network bandwidth issue, wifi is being worked on)
> 
> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> 
> Any suggestions on how to configure cake for this sort of environment where 
> there are so many devices?

So far as Cake is concerned, the normal setup should work fine even under stress conditions.  If there are too many simultaneous flows to achieve full flow isolation, it degrades gracefully to statistical multiplexing, and you still have the AQM working to keep everything responsive.  Or rather, a thousand AQMs working in parallel...

Of course, this only matters when Cake is set up to be the bottleneck.  If wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel, which I believe still applies to only some hardware since it needs to manage the MAC queue which some devices don't expose.  This has the extra smarts needed to adapt gracefully to wifi's foibles, and might already be enough to convert an effectively nonfunctional wifi into one that feels, if not fast, then at least reliable.

 - Jonathan Morton

[Cake] Re: help request for cake on a large network

[dave seddon]

Agreed.

I nearly replied to suggest getting Crown Castle to just provide dark fiber
to One Wilshire and fire it up at 10GE. ( I assume any of the major
sponsors could provide a 10ge internet link for a few days without any
concern.)

This way the bottle neck moves to the APs, which have a significant smaller
number of flows each.


The other option, if upgrading the Internets isn't an option, could be to
use 100mbs links to the APs. This sounds counter intuitive, is essentially
moving the bottleneck to the APs.

I assume you have traffic graphs from the APs from last year? I guess none
really exceeded 100mbs?

Regards,
Dave Seddon

On Mon, Sep 29, 2025, 20:56 Jonathan Morton <chromatix99@gmail.com> wrote:

>
>
> On Sunday, 28 September 2025, David Lang wrote:
> > I'm starting to prepare for the next Scale conference and we are
> switching from
> > Juniper routers to Linux routers. This gives me the ability to implement
> cake.
> >
> > One problem we have is classes that tell everyone 'go download this'
> that
> > trigger hundreds of people to hammer the network at the same time (this
> is both
> > a wifi and a network bandwidth issue, wifi is being worked on)
> >
> > The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
> >
> > Any suggestions on how to configure cake for this sort of environment
> where
> > there are so many devices?
>
> So far as Cake is concerned, the normal setup should work fine even under
> stress conditions.  If there are too many simultaneous flows to achieve
> full flow isolation, it degrades gracefully to statistical multiplexing,
> and you still have the AQM working to keep everything responsive.  Or
> rather, a thousand AQMs working in parallel...
>
> Of course, this only matters when Cake is set up to be the bottleneck.  If
> wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel,
> which I believe still applies to only some hardware since it needs to
> manage the MAC queue which some devices don't expose.  This has the extra
> smarts needed to adapt gracefully to wifi's foibles, and might already be
> enough to convert an effectively nonfunctional wifi into one that feels, if
> not fast, then at least reliable.
>
>  - Jonathan Morton
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>

[Cake] Re: help request for cake on a large network

[Jonathan Morton]

> On 28 Sep, 2025, at 8:07 pm, dave seddon <dave.seddon.ca@gmail.com> wrote:
> 
>  cakeConfig = {
>    Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
>    OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
>    MPUBytes = 84;       # Minimum packet unit for Ethernet
>    NAT = true;
>    FlowIsolationMode = "triple";
>    PriorityQueueingPreset = "besteffort";
>  };

It's perhaps worth talking about these parameters a bit.

You probably realise that the overhead compensation should be set up for the physical bottleneck link.  Not everything is, or behaves exactly like, an Ethernet cable, even if the connection between your router and the transceiver for that link physically is one.  Or it might really be Ethernet, but with extra headers patched into each Ethernet frame (in a domestic context, that's what PPPoE partially is, but VLANs add an extra couple of words to a frame too).  If in doubt, err on the side of assuming there's more overhead than you know about.  These parameters are, however, completely inadequate to describe wireless links, especially WiFi.

I'd also like to remind everyone what "triple" flow isolation is.  It's a way to get something like host-and-flow isolation when you're not certain which side of the link the hosts that want isolating from each other are; there's a heuristic which effectively decides, dynamically for each flow, whether to treat it as being src-host or dst-host isolated.  I put it in mainly to serve as a sensible default, since blindly choosing one or the other would get it wrong exactly half the time.  I don't really like seeing it in a config file that someone's actually worked on.

It's better to explicitly configure this rather than using the heuristic.  The tc-cake keywords are "dual-srchost" if the hosts to be isolated are upstream of Cake, and "dual-dsthost" if they are downstream.  In a typical installation where two instances of Cake are managing both directions of traffic, one will be configured each way.

It's also possible to disable host-isolation entirely, using the "flows" keyword to provide the same kind of flow-isolation as fq_codel (or most other forms of FQ) does.  Cake can also provide only host isolation, relying entirely on statistical multiplexing (and the beneficience of AQM) for managing flows to each host; the keywords are then "srchost" and "dsthost" with the same meaning as above.  If you are seriously worried about the number of active users at critical moments, this could be a useful option for you; the number of hosts Cake can fully handle in this mode is the same as the number of flows it can handle in the normal flow-isolation modes.

 - Jonathan Morton

[Cake] Re: help request for cake on a large network

[Sebastian Moeller]

Hi Jonathan...

thanks for commenting on the details.

> On 30. Sep 2025, at 07:18, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 28 Sep, 2025, at 8:07 pm, dave seddon <dave.seddon.ca@gmail.com> wrote:
>> 
>> cakeConfig = {
>>   Bandwidth = "990M";  # We currently have 1Gb/s, so this is our limit
>>   OverheadBytes = 38;  # Ethernet overhead (preamble + inter-frame gap + FCS)
>>   MPUBytes = 84;       # Minimum packet unit for Ethernet
>>   NAT = true;
>>   FlowIsolationMode = "triple";
>>   PriorityQueueingPreset = "besteffort";
>> };
> 
> It's perhaps worth talking about these parameters a bit.
> 
> You probably realise that the overhead compensation should be set up for the physical bottleneck link.  Not everything is, or behaves exactly like, an Ethernet cable, even if the connection between your router and the transceiver for that link physically is one.  Or it might really be Ethernet, but with extra headers patched into each Ethernet frame (in a domestic context, that's what PPPoE partially is, but VLANs add an extra couple of words to a frame too).  If in doubt, err on the side of assuming there's more overhead than you know about.  These parameters are, however, completely inadequate to describe wireless links, especially WiFi.

+1; over estimating the true overhead (or MPU) slightly has negligible cost (a small reduction in potential maximum throughput), underestimating however can result in noticeable latency under load increases. Some years ago, I retested my assumed de-bloated DSL link (after a few months of uneventful operation) because somewhat felt just not right under load, and lo and behold, my ISP had increased the overhead by 4 bytes (a VLAN tag) in the interim invalidating my overhead settings by 4 bytes... (That was with an ATM/AAL5 link, where measuring the actual overhead is achievable for end-users).

Long story, short: if in doubt rather add a small safety margin to your best overhead estimate... sqm-scripts came down on recommeding 44 bytes as default. Will not cover all possibilities, but goes a long way in covering those encapsulation seen often on ebd user internet access links.


> 
> I'd also like to remind everyone what "triple" flow isolation is.  It's a way to get something like host-and-flow isolation when you're not certain which side of the link the hosts that want isolating from each other are; there's a heuristic which effectively decides, dynamically for each flow, whether to treat it as being src-host or dst-host isolated.  I put it in mainly to serve as a sensible default, since blindly choosing one or the other would get it wrong exactly half the time.  I don't really like seeing it in a config file that someone's actually worked on.

Mmmh, we might want to tweak the tc-cake man page then to make this clearer? ATM it reads:

dual-srchost
Flows are defined by the 5-tuple, and fairness is applied
first over source addresses, then over individual flows.
Good for use on egress traffic from a LAN to the internet,
where it'll prevent any one LAN host from monopolising the
uplink, regardless of the number of flows they use.

dual-dsthost
Flows are defined by the 5-tuple, and fairness is applied
first over destination addresses, then over individual
flows. Good for use on ingress traffic to a LAN from the
internet, where it'll prevent any one LAN host from
monopolising the downlink, regardless of the number of
flows they use.

triple-isolate (default)
Flows are defined by the 5-tuple, and fairness is applied
over source *and* destination addresses intelligently (ie.
not merely by host-pairs), and also over individual flows.
Use this if you're not certain whether to use dual-srchost
or dual-dsthost; it'll do both jobs at once, preventing any
one host on *either* side of the link from monopolising it
with a large number of flows.


Reading this I would have the impression that selecting triple even on a bespoke config would be fine, no?


> 
> It's better to explicitly configure this rather than using the heuristic.  The tc-cake keywords are "dual-srchost" if the hosts to be isolated are upstream of Cake, and "dual-dsthost" if they are downstream.  In a typical installation where two instances of Cake are managing both directions of traffic, one will be configured each way.
> 
> It's also possible to disable host-isolation entirely, using the "flows" keyword to provide the same kind of flow-isolation as fq_codel (or most other forms of FQ) does.  Cake can also provide only host isolation, relying entirely on statistical multiplexing (and the beneficience of AQM) for managing flows to each host; the keywords are then "srchost" and "dsthost" with the same meaning as above.  If you are seriously worried about the number of active users at critical moments, this could be a useful option for you; the number of hosts Cake can fully handle in this mode is the same as the number of flows it can handle in the normal flow-isolation modes.

I would really like to see some tc -s qdisc output from this network under load, as that should be informative if any twaeks are necessary at all.


> 
> - Jonathan Morton
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net

[Cake] Re: help request for cake on a large network

[Jonathan Morton]

> On 30 Sep, 2025, at 9:09 am, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> I would really like to see some tc -s qdisc output from this network under load, as that should be informative if any twaeks are necessary at all.

I think the point is that this is a conference/event network that will be set up transiently.  By the time you get a realistic workload to measure the behaviour of, you're in production.

 - Jonathan Morton

[Cake] Re: help request for cake on a large network

[Sebastian Moeller]

Hi Jonathan,


> On 30. Sep 2025, at 10:59, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 30 Sep, 2025, at 9:09 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> I would really like to see some tc -s qdisc output from this network under load, as that should be informative if any twaeks are necessary at all.
> 
> I think the point is that this is a conference/event network that will be set up transiently.  By the time you get a realistic workload to measure the behaviour of, you're in production.

Indeed, this approach would do little for the up-coming SCALE N, but it should be instructive for SCALA N+1.

Regards
	Sebastian

> 
> - Jonathan Morton

[Cake] Re: help request for cake on a large network

[Toke Høiland-Jørgensen]

David Lang <david@lang.hm> writes:

> Sebastian Moeller wrote:
>
>> Hi David,
>>
>> while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.
>
> I will do that, however the network is only up under load for 4 days a year, so 
> it's a slow feedback loop :-)
>
> I would welcome any other suggestions for data to gather.

Having queue statistics at a scale as granular as you can manage would
be cool. It's around ~400 bytes of raw data per sample Capturing that
every 100ms for four days is only around 1.4 GB of data; should
theoretically be manageable? :)

Note that the 400 bytes is the in-kernel binary representation; the
output of `tc -s` is somewhat more; using JSON output (`tc -j -s`) and
compressing the output may get within something that server-grade
hardware should handle just fine.

>>> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
>>>
>>> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
>>>
>>> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)
>>
>
>> So one issue might be that with several 100 users the default compile-time 
>> size of queues (1024, IIRC) that cake will entertain might be too little, even 
>> in light of the 8 way assoziative hashing design. I believe this can be 
>> changed (within limits) only by modifying at source and recompilation of the 
>> kernel, if that should be needed at all.
>
> custom compiling a kernel is very much an option (and this sort of tweaking is 
> the sort of thing I'm expecting to need to do)
>
> The conference is in March, so we have some time to think about this and 
> customize things, just no chance to test before the show.
>
>> I wonder whether multi-queue cake would not solve this to some degree, as I 
>> assume each queue's instance would bring its own independent set of 1024 bins?
>
> good thought

While I certainly wouldn't mind having a large-scale test of the
multi-queue variant of cake, I don't really think it's necessary at 1G.
Assuming you're using server/desktop-grade hardware for the gateways,
CAKE should scale just fine to 1Gbit.

Sebastian is right that the MQ variant will install independent CAKE
instances on each hardware queue, which will give you more flow queues.
However, the round-robin dequeueing among those queues will also be
completely independent, so you won't get fairness among them either
(only between the flows that share a HWQ).

As for collision probability, we actually have a calculation of this in
the original CAKE paper[0], in figure 1. With set-associative hashing,
collision probability only start to rise around 500 simultaneous flows.
And bear in mind that these need to be active flows *from the PoV of the
router*. I.e., they need to all be actively transmitting data at the
same time; even with lots of users with active connections as seen from
the endpoint, the number of active flows in the router should be way
smaller (there's a paper discussing this that I can't find right now).
Having some data about this would be interesting, of course (and should
be part of the tc statistics).

-Toke

[0] https://arxiv.org/pdf/1804.07617

[Cake] Re: help request for cake on a large network

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
>
>> On 30. Sep 2025, at 11:04, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> David Lang <david@lang.hm> writes:
>> 
>>> Sebastian Moeller wrote:
>>> 
>>>> Hi David,
>>>> 
>>>> while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.
>>> 
>>> I will do that, however the network is only up under load for 4 days a year, so 
>>> it's a slow feedback loop :-)
>>> 
>>> I would welcome any other suggestions for data to gather.
>> 
>> Having queue statistics at a scale as granular as you can manage would
>> be cool. It's around ~400 bytes of raw data per sample Capturing that
>> every 100ms for four days is only around 1.4 GB of data; should
>> theoretically be manageable? :)
>> 
>> Note that the 400 bytes is the in-kernel binary representation; the
>> output of `tc -s` is somewhat more; using JSON output (`tc -j -s`) and
>> compressing the output may get within something that server-grade
>> hardware should handle just fine.
>> 
>>>>> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
>>>>> 
>>>>> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
>>>>> 
>>>>> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)
>>>> 
>>> 
>>>> So one issue might be that with several 100 users the default compile-time 
>>>> size of queues (1024, IIRC) that cake will entertain might be too little, even 
>>>> in light of the 8 way assoziative hashing design. I believe this can be 
>>>> changed (within limits) only by modifying at source and recompilation of the 
>>>> kernel, if that should be needed at all.
>>> 
>>> custom compiling a kernel is very much an option (and this sort of tweaking is 
>>> the sort of thing I'm expecting to need to do)
>>> 
>>> The conference is in March, so we have some time to think about this and 
>>> customize things, just no chance to test before the show.
>>> 
>>>> I wonder whether multi-queue cake would not solve this to some degree, as I 
>>>> assume each queue's instance would bring its own independent set of 1024 bins?
>>> 
>>> good thought
>> 
>> While I certainly wouldn't mind having a large-scale test of the
>> multi-queue variant of cake, I don't really think it's necessary at 1G.
>> Assuming you're using server/desktop-grade hardware for the gateways,
>> CAKE should scale just fine to 1Gbit.
>> 
>> Sebastian is right that the MQ variant will install independent CAKE
>> instances on each hardware queue, which will give you more flow queues.
>> However, the round-robin dequeueing among those queues will also be
>> completely independent, so you won't get fairness among them either
>> (only between the flows that share a HWQ).
>> 
>> As for collision probability, we actually have a calculation of this in
>> the original CAKE paper[0], in figure 1. With set-associative hashing,
>> collision probability only start to rise around 500 simultaneous flows.
>> And bear in mind that these need to be active flows *from the PoV of the
>> router*. I.e., they need to all be actively transmitting data at the
>> same time; even with lots of users with active connections as seen from
>> the endpoint, the number of active flows in the router should be way
>> smaller (there's a paper discussing this that I can't find right now).
>
> Maybe Luca's "Evaluating the Number of Active Flows in a Scheduler
> Realizing Fair Statistical Bandwidth Sharing" from 2013?
> https://team.inria.fr/rap/files/2013/12/KMOR05b.pdf

Yup, that was exactly the one I was thinking about - thank you for
digging up the link!

(Adding back the list to Cc so others can see it too).

-Toke

[Cake] Re: help request for cake on a large network

[David Lang]

Jonathan Morton wrote:

> On Sunday, 28 September 2025, David Lang wrote:
>> I'm starting to prepare for the next Scale conference and we are switching from
>> Juniper routers to Linux routers. This gives me the ability to implement cake.
>>
>> One problem we have is classes that tell everyone 'go download this' that
>> trigger hundreds of people to hammer the network at the same time (this is both
>> a wifi and a network bandwidth issue, wifi is being worked on)
>>
>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>
>> Any suggestions on how to configure cake for this sort of environment where
>> there are so many devices?
>
> So far as Cake is concerned, the normal setup should work fine even under 
> stress conditions.  If there are too many simultaneous flows to achieve full 
> flow isolation, it degrades gracefully to statistical multiplexing, and you 
> still have the AQM working to keep everything responsive.  Or rather, a 
> thousand AQMs working in parallel...

what would need to be done to increase resources to allow for full isolation of 
more flows?

> Of course, this only matters when Cake is set up to be the bottleneck.  If 
> wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel, 
> which I believe still applies to only some hardware since it needs to manage 
> the MAC queue which some devices don't expose.  This has the extra smarts 
> needed to adapt gracefully to wifi's foibles, and might already be enough to 
> convert an effectively nonfunctional wifi into one that feels, if not fast, 
> then at least reliable.

I am doing everything I can to eliminate wifi as the bottleneck and overload our 
uplink ;-p

mostly the wifi bottleneck is being eliminated by having lots of separate APs 
for people to connect to.

David Lang

[Cake] Re: help request for cake on a large network

[David Lang]

I'll read this in more detail later (it's almost 6am after I've been driving all 
night), but large number of simultanious flows in not unlikely.

Several classes that start at the same time, each with 100-300 people in it.

one of the first slides that go up tell the students to fetch something (over 
https, and for example with nix, with certificate pinning, so no ability to have 
a local cache for it), or more commonly, fetch a script tht then downloads a 
bunch of somethings.

all of the sudden, 500 simulanious flows doesn't look that unlikely :-)

David Lang

Toke Høiland-Jørgensen wrote:

> David Lang <david@lang.hm> writes:
>
>> Sebastian Moeller wrote:
>>
>>> Hi David,
>>>
>>> while I have no real answer for your questions (due to never having had that kind of load in my home network ;) ) I would like to ask you to make take scripted captures of tc -s qdisc for the wan interface is reasonable short intervals (say every 10 minutes?) as that might be just what we need to actually answer your question.
>>
>> I will do that, however the network is only up under load for 4 days a year, so
>> it's a slow feedback loop :-)
>>
>> I would welcome any other suggestions for data to gather.
>
> Having queue statistics at a scale as granular as you can manage would
> be cool. It's around ~400 bytes of raw data per sample Capturing that
> every 100ms for four days is only around 1.4 GB of data; should
> theoretically be manageable? :)
>
> Note that the 400 bytes is the in-kernel binary representation; the
> output of `tc -s` is somewhat more; using JSON output (`tc -j -s`) and
> compressing the output may get within something that server-grade
> hardware should handle just fine.
>
>>>> On 28. Sep 2025, at 13:06, David Lang <david@lang.hm> wrote:
>>>>
>>>> I'm starting to prepare for the next Scale conference and we are switching from Juniper routers to Linux routers. This gives me the ability to implement cake.
>>>>
>>>> One problem we have is classes that tell everyone 'go download this' that trigger hundreds of people to hammer the network at the same time (this is both a wifi and a network bandwidth issue, wifi is being worked on)
>>>
>>
>>> So one issue might be that with several 100 users the default compile-time
>>> size of queues (1024, IIRC) that cake will entertain might be too little, even
>>> in light of the 8 way assoziative hashing design. I believe this can be
>>> changed (within limits) only by modifying at source and recompilation of the
>>> kernel, if that should be needed at all.
>>
>> custom compiling a kernel is very much an option (and this sort of tweaking is
>> the sort of thing I'm expecting to need to do)
>>
>> The conference is in March, so we have some time to think about this and
>> customize things, just no chance to test before the show.
>>
>>> I wonder whether multi-queue cake would not solve this to some degree, as I
>>> assume each queue's instance would bring its own independent set of 1024 bins?
>>
>> good thought
>
> While I certainly wouldn't mind having a large-scale test of the
> multi-queue variant of cake, I don't really think it's necessary at 1G.
> Assuming you're using server/desktop-grade hardware for the gateways,
> CAKE should scale just fine to 1Gbit.
>
> Sebastian is right that the MQ variant will install independent CAKE
> instances on each hardware queue, which will give you more flow queues.
> However, the round-robin dequeueing among those queues will also be
> completely independent, so you won't get fairness among them either
> (only between the flows that share a HWQ).
>
> As for collision probability, we actually have a calculation of this in
> the original CAKE paper[0], in figure 1. With set-associative hashing,
> collision probability only start to rise around 500 simultaneous flows.
> And bear in mind that these need to be active flows *from the PoV of the
> router*. I.e., they need to all be actively transmitting data at the
> same time; even with lots of users with active connections as seen from
> the endpoint, the number of active flows in the router should be way
> smaller (there's a paper discussing this that I can't find right now).
> Having some data about this would be interesting, of course (and should
> be part of the tc statistics).
>
> -Toke
>
> [0] https://arxiv.org/pdf/1804.07617
>

[Cake] Re: help request for cake on a large network

[David Lang]

dave seddon wrote:

> Agreed.
>
> I nearly replied to suggest getting Crown Castle to just provide dark fiber
> to One Wilshire and fire it up at 10GE. ( I assume any of the major
> sponsors could provide a 10ge internet link for a few days without any
> concern.)

that would be fantastic if it could be done, but in our experience, getting a 
hotel/convention center to allow a new network service to be setup is increadily 
hard.

> This way the bottle neck moves to the APs, which have a significant smaller
> number of flows each.

the user may move from one AP to another in the middle of a flow, so the shaping 
needs to move further from the APs

> The other option, if upgrading the Internets isn't an option, could be to
> use 100mbs links to the APs. This sounds counter intuitive, is essentially
> moving the bottleneck to the APs.
>
> I assume you have traffic graphs from the APs from last year? I guess none
> really exceeded 100mbs?

If they did, it wasn't by much, we only use single channels on each band on each 
AP, isn't that going to limit each radio to 56k? (we do this instead of wider 
channels so that we can reuse the channel spacially sooner)

so limiting the connection to 100m isn't going to add any noticable 
restriction/bottleneck

David Lang

>
> On Mon, Sep 29, 2025, 20:56 Jonathan Morton <chromatix99@gmail.com> wrote:
>
>>
>>
>> On Sunday, 28 September 2025, David Lang wrote:
>>> I'm starting to prepare for the next Scale conference and we are
>> switching from
>>> Juniper routers to Linux routers. This gives me the ability to implement
>> cake.
>>>
>>> One problem we have is classes that tell everyone 'go download this'
>> that
>>> trigger hundreds of people to hammer the network at the same time (this
>> is both
>>> a wifi and a network bandwidth issue, wifi is being worked on)
>>>
>>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>>
>>> Any suggestions on how to configure cake for this sort of environment
>> where
>>> there are so many devices?
>>
>> So far as Cake is concerned, the normal setup should work fine even under
>> stress conditions.  If there are too many simultaneous flows to achieve
>> full flow isolation, it degrades gracefully to statistical multiplexing,
>> and you still have the AQM working to keep everything responsive.  Or
>> rather, a thousand AQMs working in parallel...
>>
>> Of course, this only matters when Cake is set up to be the bottleneck.  If
>> wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel,
>> which I believe still applies to only some hardware since it needs to
>> manage the MAC queue which some devices don't expose.  This has the extra
>> smarts needed to adapt gracefully to wifi's foibles, and might already be
>> enough to convert an effectively nonfunctional wifi into one that feels, if
>> not fast, then at least reliable.
>>
>>  - Jonathan Morton
>> _______________________________________________
>> Cake mailing list -- cake@lists.bufferbloat.net
>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>>
>

[Cake] Re: help request for cake on a large network

[David Lang]

Jonathan Morton wrote:

>> On 30 Sep, 2025, at 9:09 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> I would really like to see some tc -s qdisc output from this network under load, as that should be informative if any twaeks are necessary at all.
>
> I think the point is that this is a conference/event network that will be set up transiently.  By the time you get a realistic workload to measure the behaviour of, you're in production.

Yep, but that means that figring out what stats we want to gather this year, in 
order to analyze over the next 12 months to configure things for next year is 
very important.

We start plugging everything in on monday, on Wednesday night, the registration 
desk is opened, and the network is now production. The first significant load is 
Thursday morning when the classes start.

David Lang

[Cake] Re: help request for cake on a large network

[Sebastian Moeller]

Hi David,


> On 30. Sep 2025, at 14:37, David Lang <david@lang.hm> wrote:
> 
> Jonathan Morton wrote:
> 
>> On Sunday, 28 September 2025, David Lang wrote:
>>> I'm starting to prepare for the next Scale conference and we are switching from
>>> Juniper routers to Linux routers. This gives me the ability to implement cake.
>>> 
>>> One problem we have is classes that tell everyone 'go download this' that
>>> trigger hundreds of people to hammer the network at the same time (this is both
>>> a wifi and a network bandwidth issue, wifi is being worked on)
>>> 
>>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>> 
>>> Any suggestions on how to configure cake for this sort of environment where
>>> there are so many devices?
>> 
>> So far as Cake is concerned, the normal setup should work fine even under stress conditions.  If there are too many simultaneous flows to achieve full flow isolation, it degrades gracefully to statistical multiplexing, and you still have the AQM working to keep everything responsive.  Or rather, a thousand AQMs working in parallel...
> 
> what would need to be done to increase resources to allow for full isolation of more flows?

I believe editing net/sched/sch_cake.c and increase the number of CAKE_QUEUES:
#define CAKE_QUEUES (1024)

to something larger...

however I do not know what a realistic maximum is... and IIRC cake will search through this repeatedly, 1024 is one full memory page (4k pages) anything larger will require multiple pages (likely as single contiguous allocation). I remember dimmly someone ages ago playing with this and finding that the number can be increased, but not arbitrarily so at least on your typical somewhat memory-poor router. But that should be easy to figure out.
Mind you, given Toke's reminder on the small number of concurrently active flows and cake's 8-way associativity, maybe this is not necessary at all.


> 
>> Of course, this only matters when Cake is set up to be the bottleneck.  If wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel, which I believe still applies to only some hardware since it needs to manage the MAC queue which some devices don't expose.  This has the extra smarts needed to adapt gracefully to wifi's foibles, and might already be enough to convert an effectively nonfunctional wifi into one that feels, if not fast, then at least reliable.
> 
> I am doing everything I can to eliminate wifi as the bottleneck and overload our uplink ;-p
> 
> mostly the wifi bottleneck is being eliminated by having lots of separate APs for people to connect to.
> 
> David Lang
> _______________________________________________
> Cake mailing list -- cake@lists.bufferbloat.net
> To unsubscribe send an email to cake-leave@lists.bufferbloat.net

[Cake] Re: help request for cake on a large network

[David Lang]

Sebastian Moeller wrote:

> Hi David,
>
>
>> On 30. Sep 2025, at 14:37, David Lang <david@lang.hm> wrote:
>>
>> Jonathan Morton wrote:
>>
>>> On Sunday, 28 September 2025, David Lang wrote:
>>>> I'm starting to prepare for the next Scale conference and we are switching from
>>>> Juniper routers to Linux routers. This gives me the ability to implement cake.
>>>>
>>>> One problem we have is classes that tell everyone 'go download this' that
>>>> trigger hundreds of people to hammer the network at the same time (this is both
>>>> a wifi and a network bandwidth issue, wifi is being worked on)
>>>>
>>>> The network is pretty flat, a couple of subnets each on ipv4 and ipv6.
>>>>
>>>> Any suggestions on how to configure cake for this sort of environment where
>>>> there are so many devices?
>>>
>>> So far as Cake is concerned, the normal setup should work fine even under stress conditions.  If there are too many simultaneous flows to achieve full flow isolation, it degrades gracefully to statistical multiplexing, and you still have the AQM working to keep everything responsive.  Or rather, a thousand AQMs working in parallel...
>>
>> what would need to be done to increase resources to allow for full isolation of more flows?
>
> I believe editing net/sched/sch_cake.c and increase the number of CAKE_QUEUES:
> #define CAKE_QUEUES (1024)
>
> to something larger...
>
> however I do not know what a realistic maximum is... and IIRC cake will search through this repeatedly, 1024 is one full memory page (4k pages) anything larger will require multiple pages (likely as single contiguous allocation). I remember dimmly someone ages ago playing with this and finding that the number can be increased, but not arbitrarily so at least on your typical somewhat memory-poor router. But that should be easy to figure out.

In this case, we are using Ryzon systems with at least 32G of ram (need to look 
up the detailed specs)

David Lang

> Mind you, given Toke's reminder on the small number of concurrently active flows and cake's 8-way associativity, maybe this is not necessary at all.
>
>
>>
>>> Of course, this only matters when Cake is set up to be the bottleneck.  If wifi is the bottleneck, you'll want a wifi stack with integrated fq_codel, which I believe still applies to only some hardware since it needs to manage the MAC queue which some devices don't expose.  This has the extra smarts needed to adapt gracefully to wifi's foibles, and might already be enough to convert an effectively nonfunctional wifi into one that feels, if not fast, then at least reliable.
>>
>> I am doing everything I can to eliminate wifi as the bottleneck and overload our uplink ;-p
>>
>> mostly the wifi bottleneck is being eliminated by having lots of separate APs for people to connect to.
>>
>> David Lang
>> _______________________________________________
>> Cake mailing list -- cake@lists.bufferbloat.net
>> To unsubscribe send an email to cake-leave@lists.bufferbloat.net
>
>
>