[Codel] Do codel or fq-codel need to know the link bandwidth?

[Codel] Do codel or fq-codel need to know the link bandwidth?

[Evan Mesterhazy]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

As far as I can tell, the fq-codel algorithm (
https://www.rfc-editor.org/rfc/rfc8290.html) doesn't use link bandwidth as
a parameter. From reading the description, it seems like it should be able
to work without knowing the link bandwidth since it decides to drop packets
based on the amount of time they have waited in a queue. In fact, the
original codel rfc (https://www.rfc-editor.org/rfc/rfc8289#section-4.1)
even says that it's designed to be non-starving and work over variable
bandwidth links.

However, I have noticed that the commercial implementations for at least
two firewalls that I have used (OPNsense and a Unifi gateway) both require
the user to input the upload and download bandwidths of the link.

In OPNsense I tried setting the bandwidth to a large number above the
actual link bandwidth and then manually setting the fq-codel parameters to
their Linux defaults, but that resulted in poor performance. My concern, of
course, is that I'm leaving throughput on the table by artificially
limiting the bandwidth at a value that may be less than the actual
available bandwidth depending on the time of day, etc.

I'm not sure what exactly OPNsense is doing with the bandwidth parameters,
and I've asked on its forum (
https://forum.opnsense.org/index.php?topic=44501.0). My question for all of
you is whether I am misunderstanding something about fq-codel. Does it need
to be configured with the link bandwidth, or can it work nicely in its
default configuration with variable bandwidth links (let's consider a
typical home fiber connection between 300 and 1000 Mbps).

Thanks for any insight you can share.

[-- Attachment #2: Type: text/html, Size: 2225 bytes --]

Re: [Codel] Do codel or fq-codel need to know the link bandwidth?

[Dave Taht]

Concurrent with the rise of codel was the bql algorithm in linux, and
the combination of the two led to sufficient backpressure for it to
operate at the native rate of the interface. (AQL does a similar job
for wifi). This keeps queuing in the device ring down to what can be
serviced in a single interrupt.

"Bufferbloat Systematic Analysis":
https://d1wqtxts1xzle7.cloudfront.net/109517412/its2014_bb-libre.pdf
Byte queue limits: https://lwn.net/Articles/469652/

BSD has not evolved an equivalent mechanism. Theoretically you could
run it natively but it would have a lot of jitter from a per-packet
device ring.

Further most ISPs use a non-native rate for their customer interfaces,
either using a policer or FIFO shaper and thus the rise of combatting
that with shaping via fq_codel to slightly below the ISPs' rate to
move the bottleneck to your own hardware.

We designed CAKE to use a deficit based shaper to be (roughly) the
inverse of a token bucket shaper to get very close to the ISP rate yet
shape well

LibreQos (Preseem, Paraqum, Bequant) give ISPs a fq_codel or cake based shaper.
The rightest answer was to have the isp correctly shape the download,
and the uplink device shape the upload.

The bufferbloat project has never had the resources to tackle BSD.

These days all of Linux (and OSX) run fq_codel natively. The FIFO is dead.

Re: [Codel] Do codel or fq-codel need to know the link bandwidth?

[Dave Taht]

"variable bandwidth" requires that the queuing below fq_codel be
minimal. For example 5g has enormous device buffers all over the place
which we haven't found an answer to aside from nagging the 5g folk to
fix it, and "cake-autorate"

Linux WiFi had similar problems, fixed now.

https://www.cs.kau.se/tohojo/airtime-fairness/



On Sat, Dec 7, 2024 at 2:25 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> Concurrent with the rise of codel was the bql algorithm in linux, and
> the combination of the two led to sufficient backpressure for it to
> operate at the native rate of the interface. (AQL does a similar job
> for wifi). This keeps queuing in the device ring down to what can be
> serviced in a single interrupt.
>
> "Bufferbloat Systematic Analysis":
> https://d1wqtxts1xzle7.cloudfront.net/109517412/its2014_bb-libre.pdf
> Byte queue limits: https://lwn.net/Articles/469652/
>
> BSD has not evolved an equivalent mechanism. Theoretically you could
> run it natively but it would have a lot of jitter from a per-packet
> device ring.
>
> Further most ISPs use a non-native rate for their customer interfaces,
> either using a policer or FIFO shaper and thus the rise of combatting
> that with shaping via fq_codel to slightly below the ISPs' rate to
> move the bottleneck to your own hardware.
>
> We designed CAKE to use a deficit based shaper to be (roughly) the
> inverse of a token bucket shaper to get very close to the ISP rate yet
> shape well
>
> LibreQos (Preseem, Paraqum, Bequant) give ISPs a fq_codel or cake based shaper.
> The rightest answer was to have the isp correctly shape the download,
> and the uplink device shape the upload.
>
> The bufferbloat project has never had the resources to tackle BSD.
>
> These days all of Linux (and OSX) run fq_codel natively. The FIFO is dead.



-- 
Dave Täht CSO, LibreQos

Re: [Codel] Do codel or fq-codel need to know the link bandwidth?

[Evan Mesterhazy]

[-- Attachment #1: Type: text/plain, Size: 3516 bytes --]

Thanks Dave, your reply is very informative. I definitely understand that
FreeBSD is somewhat less sophisticated than Linux when it comes to this
since I've read through a few of the bugs and other messages you've posted
regarding its implementation. However the Unifi routers run on Linux, and
they still require the user to specify the ISP's service bandwidth. I think
this paragraph from your reply may be the crux of it:

> Further most ISPs use a non-native rate for their customer interfaces,
> either using a policer or FIFO shaper and thus the rise of combatting
> that with shaping via fq_codel to slightly below the ISPs' rate to
> move the bottleneck to your own hardware.

If the router is able to keep pushing bytes over the ISP's link even in
excess of the allowed bandwidth, then the local router's packets probably
aren't sitting in their queues long enough to be dropped by fq-codel. It
sounds like proper traffic shaping needs to happen wherever the bottleneck
is (I suppose this should have been obvious to me), which in the case of a
300 Mbp/s connection over GPON is somewhere at the ISP. I guess if the ISP
does proper traffic shaping the end user shouldn't need to do anything.

Assuming I didn't say anything inaccurate in my reply, I think I understand
why setting a bandwidth limit is necessary to make fq-codel work properly
on a router with a less than line speed connection from the ISP.

Thank you!

On Sat, Dec 7, 2024 at 5:32 PM Dave Taht <dave.taht@gmail.com> wrote:

> "variable bandwidth" requires that the queuing below fq_codel be
> minimal. For example 5g has enormous device buffers all over the place
> which we haven't found an answer to aside from nagging the 5g folk to
> fix it, and "cake-autorate"
>
> Linux WiFi had similar problems, fixed now.
>
> https://www.cs.kau.se/tohojo/airtime-fairness/
>
>
>
> On Sat, Dec 7, 2024 at 2:25 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > Concurrent with the rise of codel was the bql algorithm in linux, and
> > the combination of the two led to sufficient backpressure for it to
> > operate at the native rate of the interface. (AQL does a similar job
> > for wifi). This keeps queuing in the device ring down to what can be
> > serviced in a single interrupt.
> >
> > "Bufferbloat Systematic Analysis":
> > https://d1wqtxts1xzle7.cloudfront.net/109517412/its2014_bb-libre.pdf
> > Byte queue limits: https://lwn.net/Articles/469652/
> >
> > BSD has not evolved an equivalent mechanism. Theoretically you could
> > run it natively but it would have a lot of jitter from a per-packet
> > device ring.
> >
> > Further most ISPs use a non-native rate for their customer interfaces,
> > either using a policer or FIFO shaper and thus the rise of combatting
> > that with shaping via fq_codel to slightly below the ISPs' rate to
> > move the bottleneck to your own hardware.
> >
> > We designed CAKE to use a deficit based shaper to be (roughly) the
> > inverse of a token bucket shaper to get very close to the ISP rate yet
> > shape well
> >
> > LibreQos (Preseem, Paraqum, Bequant) give ISPs a fq_codel or cake based
> shaper.
> > The rightest answer was to have the isp correctly shape the download,
> > and the uplink device shape the upload.
> >
> > The bufferbloat project has never had the resources to tackle BSD.
> >
> > These days all of Linux (and OSX) run fq_codel natively. The FIFO is
> dead.
>
>
>
> --
> Dave Täht CSO, LibreQos
>

[-- Attachment #2: Type: text/html, Size: 5006 bytes --]

Re: [Codel] Do codel or fq-codel need to know the link bandwidth?

[Jonathan Morton]

> On 7 Dec, 2024, at 11:27 pm, Evan Mesterhazy <evan.mesterhazy@gmail.com> wrote:
> 
> However, I have noticed that the commercial implementations for at least two firewalls that I have used (OPNsense and a Unifi gateway) both require the user to input the upload and download bandwidths of the link.

That's because they combine three algorithms into a coherent package.  All three algorithms do something different which is a crucial part of the whole.

FQ_Codel is a combination of two of these algorithms: the Codel AQM (which signals to traffic when it's passing through a congested link) and the DRR++ Fair Queuing algorithm, which essentially isolates different flows from interfering with each other.

The third algorithm is called a "shaper".  It moves the bottleneck out of the link hardware itself (which isn't smart enough to run FQ_Codel) and into the CPU (which is).  It is the shaper that needs to know what bandwidth to use, because it can't directly sense the link characteristics from where it is.

 - Jonathan Morton