[Cake] Fwd: [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Cake] Fwd: [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 8 bytes --]

FYI :)


[-- Attachment #2: Type: message/rfc822, Size: 1414 bytes --]

From: Jamal Hadi Salim via people <people@netdevconf.info>
To: people <people@netdevconf.info>
Cc: Kimberley Jeffries <kimberleyjeffries@gmail.com>, Lael Santos <lael.santos@expertisesolutions.com.br>, program-committee@netdevconf.info, Bruno Banelli <bruno.banelli@sartura.hr>, j.koeppeler@tu-berlin.de, stefan.schmid@tu-berlin.de, Jamal Hadi Salim <jhs@mojatatu.com>
Subject: [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores
Date: Sun, 9 Feb 2025 08:37:41 -0500

Qdiscs rely on global lock to sync state across CPUs and therefore
dont scale in presence of many cores (or in presence of very high
bandwidth).
Jonas Köppeler, Toke Høiland-Jørgensen, and Stefan Schmid implemented
a multi-queue variant of sch_cake that can scale its rate limiting
across hardware queues (and thus CPU cores) by sharing a bit of state
on top of the mq qdisc.

In this talk, they will present the implementation, performance
evaluation, as well as discuss their proposal for an API that will
make this work upstreamable, and applicable to other qdiscs as well.

Details: https://netdevconf.info/0x19/16

cheers,
jamal
_______________________________________________
people mailing list -- people@netdevconf.info
To unsubscribe send an email to people-leave@netdevconf.info

Re: [Cake] [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Sebastian Moeller]

That sounds most excellent...

This also means that now we need cheap router hardware with > 2 NIC queues and > 2 CPUs ;) (with ingress and egress shaping the current state is that 2 CPUs can be utilised). I wonder, does this work for IFBs as well or only for real hardware NIC queues?

Regards
	Sebastian


> On 10. Feb 2025, at 10:30, Toke Høiland-Jørgensen via Cake <cake@lists.bufferbloat.net> wrote:
> 
> FYI :)
> 
> 
> From: Jamal Hadi Salim via people <people@netdevconf.info>
> Subject: [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores
> Date: 9. February 2025 at 14:37:41 CET
> To: people <people@netdevconf.info>
> Cc: Kimberley Jeffries <kimberleyjeffries@gmail.com>, Lael Santos <lael.santos@expertisesolutions.com.br>, program-committee@netdevconf.info, Bruno Banelli <bruno.banelli@sartura.hr>, j.koeppeler@tu-berlin.de, stefan.schmid@tu-berlin.de, Jamal Hadi Salim <jhs@mojatatu.com>
> 
> 
> Qdiscs rely on global lock to sync state across CPUs and therefore
> dont scale in presence of many cores (or in presence of very high
> bandwidth).
> Jonas Köppeler, Toke Høiland-Jørgensen, and Stefan Schmid implemented
> a multi-queue variant of sch_cake that can scale its rate limiting
> across hardware queues (and thus CPU cores) by sharing a bit of state
> on top of the mq qdisc.
> 
> In this talk, they will present the implementation, performance
> evaluation, as well as discuss their proposal for an API that will
> make this work upstreamable, and applicable to other qdiscs as well.
> 
> Details: https://netdevconf.info/0x19/16
> 
> cheers,
> jamal
> _______________________________________________
> people mailing list -- people@netdevconf.info
> To unsubscribe send an email to people-leave@netdevconf.info
> 
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> That sounds most excellent...
>
> This also means that now we need cheap router hardware with > 2 NIC
> queues and > 2 CPUs ;) (with ingress and egress shaping the current
> state is that 2 CPUs can be utilised).

Well, even if there are only 2 CPUs, the MQ version can utilise both of
those in one direction, which helps for asymmetrical traffic :)

> I wonder, does this work for IFBs as well or only for real hardware
> NIC queues?

Yup, you can specify the number of TX queues on an ifb interface when
creating it (ip link add numtxqueues 2 type ifb).

-Toke

Re: [Cake] [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Sebastian Moeller]

Hi Toke,


> On 10. Feb 2025, at 11:35, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> 
> Sebastian Moeller <moeller0@gmx.de> writes:
> 
>> That sounds most excellent...
>> 
>> This also means that now we need cheap router hardware with > 2 NIC
>> queues and > 2 CPUs ;) (with ingress and egress shaping the current
>> state is that 2 CPUs can be utilised).
> 
> Well, even if there are only 2 CPUs, the MQ version can utilise both of
> those in one direction, which helps for asymmetrical traffic :)

Fair point, I was thinking bi directional saturating traffic on a symmetric link, which I guess is far from the typical use case ;)

> 
>> I wonder, does this work for IFBs as well or only for real hardware
>> NIC queues?
> 
> Yup, you can specify the number of TX queues on an ifb interface when
> creating it (ip link add numtxqueues 2 type ifb).

Great, so this will then work even if the true NIC hardware is not multi-queue capable or does numtxqueues need to be <= true number of NIC queues?

> 
> -Toke
> 

Re: [Cake] [NetDev-People] 0x19: Talk, mq-cake: Scaling software rate limiting across CPU cores

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
>
>> On 10. Feb 2025, at 11:35, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> 
>> Sebastian Moeller <moeller0@gmx.de> writes:
>> 
>>> That sounds most excellent...
>>> 
>>> This also means that now we need cheap router hardware with > 2 NIC
>>> queues and > 2 CPUs ;) (with ingress and egress shaping the current
>>> state is that 2 CPUs can be utilised).
>> 
>> Well, even if there are only 2 CPUs, the MQ version can utilise both of
>> those in one direction, which helps for asymmetrical traffic :)
>
> Fair point, I was thinking bi directional saturating traffic on a
> symmetric link, which I guess is far from the typical use case ;)

Exactly :)

>> 
>>> I wonder, does this work for IFBs as well or only for real hardware
>>> NIC queues?
>> 
>> Yup, you can specify the number of TX queues on an ifb interface when
>> creating it (ip link add numtxqueues 2 type ifb).
>
> Great, so this will then work even if the true NIC hardware is not
> multi-queue capable or does numtxqueues need to be <= true number of
> NIC queues?

Hmm, no, ifb queues are independent, so yeah, I guess ingress shaping
could scale regardless of the number of HWQs...

-Toke