[Bloat] Testbed using containers

[Bloat] Testbed using containers

[O. P.]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Hi there,

I'm trying to set up a testbed to evaluate different AQM techniques using
docker containers.
My first idea to create congestion was to use netem. However I later came
across
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
which discourages using netem. Since the document is from 2014 and also
states that "netem has been improving", my question was wether the current
netem has improved sufficiently to be used to get realistic results.
If netem has improved sufficienly, what would be the correct way to use
netem along fq, fq-codel or codel for example ?
If not, is HTB still the best way to perform rate limiting ? Is there a
prefered way to combine HTB with AQMs ?

Best regards

John

[-- Attachment #2: Type: text/html, Size: 1027 bytes --]

[Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Joerg Deutschmann]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

Dear all,

bringing up again the question from a previous message to this list...

https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#the-netem-qdisc-does-not-work-in-conjunction-with-other-qdiscs
says to not use NetEm together with other qdiscs.

Is this still true?

How could one emulate bottlenecks together with fq_codel?

Best regards
Joerg


On 15.01.24 13:24, O. P. via Bloat wrote:
> 
> Hi there,
> 
> I'm trying to set up a testbed to evaluate different AQM techniques 
> using docker containers.
> My first idea to create congestion was to use netem. However I later 
> came across 
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/ <https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/> which discourages using netem. Since the document is from 2014 and also states that "netem has been improving", my question was wether the current netem has improved sufficiently to be used to get realistic results.
> If netem has improved sufficienly, what would be the correct way to use 
> netem along fq, fq-codel or codel for example ?
> If not, is HTB still the best way to perform rate limiting ? Is there a 
> prefered way to combine HTB with AQMs ?
> 
> Best regards
> 
> John
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4843 bytes --]

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2250 bytes --]

What I would typically do is setup netem on an inbound qdisc and whatever
is under test on the outbound.

even then, care is needed to ensure netem itself is not the bottleneck, has
an appropriate packet limit (I use millions), and that you are not running
out of cpu. For example it is really hard to pull 4gbit on cheap hw with
default qdiscs.

On Fri, Oct 25, 2024 at 11:54 AM Joerg Deutschmann via Bloat <
bloat@lists.bufferbloat.net> wrote:

> Dear all,
>
> bringing up again the question from a previous message to this list...
>
>
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#the-netem-qdisc-does-not-work-in-conjunction-with-other-qdiscs
> says to not use NetEm together with other qdiscs.
>
> Is this still true?
>
> How could one emulate bottlenecks together with fq_codel?
>
> Best regards
> Joerg
>
>
> On 15.01.24 13:24, O. P. via Bloat wrote:
> >
> > Hi there,
> >
> > I'm trying to set up a testbed to evaluate different AQM techniques
> > using docker containers.
> > My first idea to create congestion was to use netem. However I later
> > came across
> >
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
> <
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/>
> which discourages using netem. Since the document is from 2014 and also
> states that "netem has been improving", my question was wether the current
> netem has improved sufficiently to be used to get realistic results.
> > If netem has improved sufficienly, what would be the correct way to use
> > netem along fq, fq-codel or codel for example ?
> > If not, is HTB still the best way to perform rate limiting ? Is there a
> > prefered way to combine HTB with AQMs ?
> >
> > Best regards
> >
> > John
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>


-- 
Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 3680 bytes --]

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Joerg Deutschmann]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

> What I would typically do is setup netem on an inbound qdisc and 
> whatever is under test on the outbound.
Thanks, this is good advice.

> even then, care is needed to ensure netem itself is not the bottleneck, 
> has an appropriate packet limit (I use millions), and that you are not 
> running out of cpu. For example it is really hard to pull 4gbit on cheap 
> hw with default qdiscs.
👍

As I'm not an expert regarding qdiscs, this command line example was 
useful showing how to configure an ingress qdisc:
https://serverfault.com/questions/984822/how-to-remove-the-ingress-delay-introduced-using-tc
(Note that it does not set a higher packet limit as you correctly 
recommended.)

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4843 bytes --]

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Maximilian Bachl]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

From my experience (experimented with it last year), it still behaves
weirdly. You can use htb for the shaping and you can create delay using
netem by using it on another (virtual) host on a link that does not have
any other qdiscs and where the link is not the bottleneck.

Best regards,
Max

On Fri 25. Oct 2024 at 20:54, Joerg Deutschmann via Bloat <
bloat@lists.bufferbloat.net> wrote:

> Dear all,
>
> bringing up again the question from a previous message to this list...
>
>
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#the-netem-qdisc-does-not-work-in-conjunction-with-other-qdiscs
> says to not use NetEm together with other qdiscs.
>
> Is this still true?
>
> How could one emulate bottlenecks together with fq_codel?
>
> Best regards
> Joerg
>
>
> On 15.01.24 13:24, O. P. via Bloat wrote:
> >
> > Hi there,
> >
> > I'm trying to set up a testbed to evaluate different AQM techniques
> > using docker containers.
> > My first idea to create congestion was to use netem. However I later
> > came across
> >
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
> <
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/>
> which discourages using netem. Since the document is from 2014 and also
> states that "netem has been improving", my question was wether the current
> netem has improved sufficiently to be used to get realistic results.
> > If netem has improved sufficienly, what would be the correct way to use
> > netem along fq, fq-codel or codel for example ?
> > If not, is HTB still the best way to perform rate limiting ? Is there a
> > prefered way to combine HTB with AQMs ?
> >
> > Best regards
> >
> > John
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 3530 bytes --]

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Stephen Hemminger]

On Fri, 25 Oct 2024 21:55:21 +0200
Maximilian Bachl via Bloat <bloat@lists.bufferbloat.net> wrote:

> From my experience (experimented with it last year), it still behaves
> weirdly. You can use htb for the shaping and you can create delay using
> netem by using it on another (virtual) host on a link that does not have
> any other qdiscs and where the link is not the bottleneck.
> 
> Best regards,
> Max

Yes, netem has expectations about how inner qdisc behaves,
and other qdisc used as inner have expectations about how/when packets
are sent. There is a mismatch, not sure if it is fixable with in the
architecture of how Linux queue disciplines operate.

The best way is to use netem on an intermediate hop and not expect
it to work perfectly when used on an endpoint. The same is true of Dummynet
and other network emulators; they are uses as man-in-the-middle systems.

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Joerg Deutschmann]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

Thanks Max and Stephen,

>>  From my experience (experimented with it last year), it still behaves
>> weirdly. You can use htb for the shaping and you can create delay using
>> netem by using it on another (virtual) host on a link that does not have
>> any other qdiscs and where the link is not the bottleneck.
Yes, I did some quick tests and saw that this works:
> sudo tc qdisc add dev eth0 root handle 2: tbf rate 200Mbit burst 4542 limit 500000
> sudo tc qdisc add dev eth0 parent 2: handle 3: fq_codel
but this does not work:
> sudo tc qdisc add dev eth0 root handle 1: netem delay 20ms limit 10000000
> sudo tc qdisc add dev eth0 parent 1: handle 2: tbf rate 200Mbit burst 4542 limit 500000
> sudo tc qdisc add dev eth0 parent 2: handle 3: fq_codel

Adding netem delay to a separate host or adding it to the ingress qdisc 
(as described by Dave) does the trick.


> Yes, netem has expectations about how inner qdisc behaves,
> and other qdisc used as inner have expectations about how/when packets
> are sent. There is a mismatch, not sure if it is fixable with in the
> architecture of how Linux queue disciplines operate.
For the sake of completeness, here's the quote from the netem man page:
> Combining netem with other qdisc is possible but may not always work because netem use skb control block to set delays.
:-)

> 
> The best way is to use netem on an intermediate hop and not expect
> it to work perfectly when used on an endpoint. The same is true of Dummynet
> and other network emulators; they are uses as man-in-the-middle systems.
Still I guess it can be tricky if netem delay + tbf shaping + fq_codel 
shall be used altogether.

This paper mentions that the order tbf/netem or netem/tbf matters 
(section 4.1.1), but does not mention fq_codel:
https://atlas.cs.uni-tuebingen.de/~menth/papers/Menth17-Sub-2.pdf

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4843 bytes --]

Re: [Bloat] The NetEm qdisc does not work in conjunction with other qdiscs

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

To add to discrepancies tbf so far as I remember broke up gso
superpackets into packets, htb did not.

How all the aqms behave with superpackets is insufficiently explored. This
is why cake made breaking
up gso packets into regular packets the default.


On Fri, Oct 25, 2024 at 3:47 PM Joerg Deutschmann via Bloat <
bloat@lists.bufferbloat.net> wrote:

> Thanks Max and Stephen,
>
> >>  From my experience (experimented with it last year), it still behaves
> >> weirdly. You can use htb for the shaping and you can create delay using
> >> netem by using it on another (virtual) host on a link that does not have
> >> any other qdiscs and where the link is not the bottleneck.
> Yes, I did some quick tests and saw that this works:
> > sudo tc qdisc add dev eth0 root handle 2: tbf rate 200Mbit burst 4542
> limit 500000
> > sudo tc qdisc add dev eth0 parent 2: handle 3: fq_codel
> but this does not work:
> > sudo tc qdisc add dev eth0 root handle 1: netem delay 20ms limit 10000000
> > sudo tc qdisc add dev eth0 parent 1: handle 2: tbf rate 200Mbit burst
> 4542 limit 500000
> > sudo tc qdisc add dev eth0 parent 2: handle 3: fq_codel
>
> Adding netem delay to a separate host or adding it to the ingress qdisc
> (as described by Dave) does the trick.
>
>
> > Yes, netem has expectations about how inner qdisc behaves,
> > and other qdisc used as inner have expectations about how/when packets
> > are sent. There is a mismatch, not sure if it is fixable with in the
> > architecture of how Linux queue disciplines operate.
> For the sake of completeness, here's the quote from the netem man page:
> > Combining netem with other qdisc is possible but may not always work
> because netem use skb control block to set delays.
> :-)
>
> >
> > The best way is to use netem on an intermediate hop and not expect
> > it to work perfectly when used on an endpoint. The same is true of
> Dummynet
> > and other network emulators; they are uses as man-in-the-middle systems.
> Still I guess it can be tricky if netem delay + tbf shaping + fq_codel
> shall be used altogether.
>
> This paper mentions that the order tbf/netem or netem/tbf matters
> (section 4.1.1), but does not mention fq_codel:
> https://atlas.cs.uni-tuebingen.de/~menth/papers/Menth17-Sub-2.pdf
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>


-- 
Dave Täht CSO, LibreQos

[-- Attachment #2: Type: text/html, Size: 3441 bytes --]