[Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jan Ceuleers]

Disappointing...


-------- Forwarded Message --------
Subject: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc
for networking
Date: Wed, 08 Jul 2015 10:50:05 -0000
From: Andy Whitcroft <apw@canonical.com>
Reply-To: Bug 1436945 <1436945@bugs.launchpad.net>
To: jan.ceuleers@computer.org

** Changed in: linux (Ubuntu)
    Milestone: ubuntu-15.03 => ubuntu-15.07

-- 
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1436945

Title:
  devel: consider fq_codel as the default qdisc for networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1436945/+subscriptions

[Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jan Ceuleers]

Took 3 years after Dave approached them, but Ubuntu is finally adopting
fq_codel as the default qdisc.


-------- Forwarded Message --------
Subject: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc
for networking
Date: Thu, 24 May 2018 14:50:09 -0000
From: Laurent Bonnaud <L.Bonnaud@laposte.net>
Reply-To: Bug 1436945 <1436945@bugs.launchpad.net>
To: jan.ceuleers@computer.org

I also see fq_codel used as default:

# cat /proc/sys/net/core/default_qdisc
fq_codel

# ip addr
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state
UP group default qlen 1000


** Changed in: linux (Ubuntu)
       Status: Confirmed => Fix Released

-- 
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1436945

Title:
  devel: consider fq_codel as the default qdisc for networking

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1436945/+subscriptions

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Bless, Roland (TM)]

Hi,

Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
> Took 3 years after Dave approached them, but Ubuntu is finally adopting
> fq_codel as the default qdisc.

Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
but I don't understand why it make sense for ordinary end-systems.
Didn't Byte Queue Limits (BQL) suffice? Just curious...

Regards
 Roland

> -------- Forwarded Message --------
> Subject: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc
> for networking
> Date: Thu, 24 May 2018 14:50:09 -0000
> From: Laurent Bonnaud <L.Bonnaud@laposte.net>
> Reply-To: Bug 1436945 <1436945@bugs.launchpad.net>
> To: jan.ceuleers@computer.org
> 
> I also see fq_codel used as default:
> 
> # cat /proc/sys/net/core/default_qdisc
> fq_codel
> 
> # ip addr
> [...]
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state
> UP group default qlen 1000
> 
> 
> ** Changed in: linux (Ubuntu)
>        Status: Confirmed => Fix Released
> 

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Mon, Jun 4, 2018 at 1:28 PM, Bless, Roland (TM) <roland.bless@kit.edu>
wrote:

> Hi,
>
> Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
> > Took 3 years after Dave approached them, but Ubuntu is finally adopting
> > fq_codel as the default qdisc.
>
> Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
> but I don't understand why it make sense for ordinary end-systems.
> Didn't Byte Queue Limits (BQL) suffice? Just curious...
>

Actually, for a long time now the codel wiki has recommended using sch_fq
instead of fq_codel for servers.

It seems Ubuntu defaulting to fq_codel is a result of using systemd which
adopted it as default almost 4 years ago. There was a request to change the
default in systemd from fq_codel to sch_fq in order to support pacing but
the issue was closed when tcp internal pacing was introduced in linux 4.13:

https://github.com/systemd/systemd/issues/5090

/Jonas

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Dave Taht]

On Mon, Jun 4, 2018 at 6:16 AM, Jonas Mårtensson
<martensson.jonas@gmail.com> wrote:
>
>
> On Mon, Jun 4, 2018 at 1:28 PM, Bless, Roland (TM) <roland.bless@kit.edu>
> wrote:
>>
>> Hi,
>>
>> Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
>> > Took 3 years after Dave approached them, but Ubuntu is finally adopting
>> > fq_codel as the default qdisc.
>>
>> Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
>> but I don't understand why it make sense for ordinary end-systems.
>> Didn't Byte Queue Limits (BQL) suffice? Just curious...
>
>
> Actually, for a long time now the codel wiki has recommended using sch_fq
> instead of fq_codel for servers.

That recommendation predates the tcp pacing change you note below.

I'm pretty convinced fq_codel is now the best all around "default"
qdisc, and we should rework our recomendation.

> It seems Ubuntu defaulting to fq_codel is a result of using systemd which
> adopted it as default almost 4 years ago. There was a request to change the
> default in systemd from fq_codel to sch_fq in order to support pacing but
> the issue was closed when tcp internal pacing was introduced in linux 4.13:
>
> https://github.com/systemd/systemd/issues/5090
>
> /Jonas
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]

Speaking about systemd defaults, they just enabled ecn for outgoing
connections:

https://github.com/systemd/systemd/pull/9143

/Jonas

On Mon, Jun 4, 2018 at 7:08 PM, Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Jun 4, 2018 at 6:16 AM, Jonas Mårtensson
> <martensson.jonas@gmail.com> wrote:
> >
> >
> > On Mon, Jun 4, 2018 at 1:28 PM, Bless, Roland (TM) <roland.bless@kit.edu
> >
> > wrote:
> >>
> >> Hi,
> >>
> >> Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
> >> > Took 3 years after Dave approached them, but Ubuntu is finally
> adopting
> >> > fq_codel as the default qdisc.
> >>
> >> Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
> >> but I don't understand why it make sense for ordinary end-systems.
> >> Didn't Byte Queue Limits (BQL) suffice? Just curious...
> >
> >
> > Actually, for a long time now the codel wiki has recommended using sch_fq
> > instead of fq_codel for servers.
>
> That recommendation predates the tcp pacing change you note below.
>
> I'm pretty convinced fq_codel is now the best all around "default"
> qdisc, and we should rework our recomendation.
>
> > It seems Ubuntu defaulting to fq_codel is a result of using systemd which
> > adopted it as default almost 4 years ago. There was a request to change
> the
> > default in systemd from fq_codel to sch_fq in order to support pacing but
> > the issue was closed when tcp internal pacing was introduced in linux
> 4.13:
> >
> > https://github.com/systemd/systemd/issues/5090
> >
> > /Jonas
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 3052 bytes --]

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jonathan Morton]

> On 4 Jun, 2018, at 9:22 pm, Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
> 
> Speaking about systemd defaults, they just enabled ecn for outgoing connections:

That is also good news.  With Apple *and* Ubuntu using it by default, we should finally get critical mass of ECN traffic and any remaining blackholes fixed, making it easy for everyone else to justify turning it on as well.

 - Jonathan Morton

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]


Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
    > Speaking about systemd defaults, they just enabled ecn for outgoing
    > connections:

    > https://github.com/systemd/systemd/pull/9143

What about PLPMTU?  Do you think they might tweak that too?

     net.ipv4.tcp_mtu_probing=2
     (despite name, applies to IPv6 too)

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 464 bytes --]

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On Tue, Jun 5, 2018 at 2:22 AM, Michael Richardson <mcr@sandelman.ca> wrote:

>
> Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
>     > Speaking about systemd defaults, they just enabled ecn for outgoing
>     > connections:
>
>     > https://github.com/systemd/systemd/pull/9143
>
> What about PLPMTU?  Do you think they might tweak that too?
>
>      net.ipv4.tcp_mtu_probing=2
>      (despite name, applies to IPv6 too)


Maybe, suggest it on their github. But I would maybe propose instead
net.ipv4.tcp_mtu_probing=1.

/Jonas

[-- Attachment #2: Type: text/html, Size: 2327 bytes --]

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 461 bytes --]

On Tue, 5 Jun 2018, Jonas Mårtensson wrote:

>> What about PLPMTU?  Do you think they might tweak that too?
>>
>>      net.ipv4.tcp_mtu_probing=2
>>      (despite name, applies to IPv6 too)
>
>
> Maybe, suggest it on their github. But I would maybe propose instead
> net.ipv4.tcp_mtu_probing=1.

MTU probing would be awsome. I am great fan of PLPMTU and this should be 
default-on everywhere in all protocols.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[David Lang]

On Mon, 4 Jun 2018, Bless, Roland (TM) wrote:

> Hi,
>
> Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
>> Took 3 years after Dave approached them, but Ubuntu is finally adopting
>> fq_codel as the default qdisc.
>
> Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
> but I don't understand why it make sense for ordinary end-systems.
> Didn't Byte Queue Limits (BQL) suffice? Just curious...

no, BQL makes things much better (and make it possible for more advanced quueing 
to take place), but you can still run into problems where a bulk stream can 
flood the output queue so that other traffic suffers badly.

with fq_codel, the available bandwidth is distributed in a way that ends up 
being much more functional.

It turns out that the behavior to prioritize new and sparse connections 
significantly improves perceived performance (no more long delays in DNS lookups 
before you start doing any real work for example)

without BQL, you can't even see the rest of the problems, but BQL doesn't solve 
everything.

David Lang

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Mario Hock]

Am 05.06.2018 um 01:00 schrieb David Lang:
> On Mon, 4 Jun 2018, Bless, Roland (TM) wrote:
> 
>> Hi,
>>
>> Am 24.05.2018 um 17:38 schrieb Jan Ceuleers:
>>> Took 3 years after Dave approached them, but Ubuntu is finally adopting
>>> fq_codel as the default qdisc.
>>
>> Yes, if the Linux kernel is forwarding packets it makes a lot of sense,
>> but I don't understand why it make sense for ordinary end-systems.
>> Didn't Byte Queue Limits (BQL) suffice? Just curious...
> 
> no, BQL makes things much better (and make it possible for more advanced 
> quueing to take place), but you can still run into problems where a bulk 
> stream can flood the output queue so that other traffic suffers badly.
> 
> with fq_codel, the available bandwidth is distributed in a way that ends 
> up being much more functional.
> 
> It turns out that the behavior to prioritize new and sparse connections 
> significantly improves perceived performance (no more long delays in DNS 
> lookups before you start doing any real work for example)
> 
> without BQL, you can't even see the rest of the problems, but BQL 
> doesn't solve everything.

Just to make sure that I got your answer correctly. The benefit for 
endsystems comes from the "fq" (flow queuing) part, not from the "codel" 
part of fq_codel?

Mario Hock

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Jonathan Morton]

> On 5 Jun, 2018, at 10:44 am, Mario Hock <mario.hock@kit.edu> wrote:
> 
> Just to make sure that I got your answer correctly. The benefit for endsystems comes from the "fq" (flow queuing) part, not from the "codel" part of fq_codel?

That's a fair characterisation, yes.

In fact, even for middleboxes, the "flow isolation" semantics of FQ have the most impact on reducing inter-flow induced latency.  The "codel" part (AQM) helps with intra-flow latency, which is usually much less important once flow isolation is in place, but is still worth having.

 - Jonathan Morton

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Mario Hock]

Am 05.06.2018 um 09:49 schrieb Jonathan Morton:
>> On 5 Jun, 2018, at 10:44 am, Mario Hock <mario.hock@kit.edu> wrote:
>>
>> Just to make sure that I got your answer correctly. The benefit for endsystems comes from the "fq" (flow queuing) part, not from the "codel" part of fq_codel?
> 
> That's a fair characterisation, yes.
> 
> In fact, even for middleboxes, the "flow isolation" semantics of FQ have the most impact on reducing inter-flow induced latency.  The "codel" part (AQM) helps with intra-flow latency, which is usually much less important once flow isolation is in place, but is still worth having.

Thanks for the confirmation.

A potential drawback of using the codel part (of fq_codel) in the 
endsystems is that it can cause packet drops already at the sender.

I could actually confirm this assumption with a very simple experiment 
consisting of two servers connected over a 1Gbit/s link and 100 parallel 
flows (iperf3). With fq_codel I had 5,000-10,000 retransmissions within 
60s. With fq (or pfifo_fast) no packets are dropped. (I presume either 
"TCP small queues" or backpressure keeps the queues from overflowing.)

Also, ping times (delays for short flows) were similar with fq and 
fq_codel (mostly <= 1ms).

Is there any advantage of using fq_codel over fq at the endsystems?

Mario Hock

Re: [Bloat] Fwd: [Bug 1436945] Re: devel: consider fq_codel as the default qdisc for networking

[Dave Taht]

On Tue, Jun 5, 2018 at 12:49 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 5 Jun, 2018, at 10:44 am, Mario Hock <mario.hock@kit.edu> wrote:
> >
> > Just to make sure that I got your answer correctly. The benefit for endsystems comes from the "fq" (flow queuing) part, not from the "codel" part of fq_codel?
>
> That's a fair characterisation, yes.
>
> In fact, even for middleboxes, the "flow isolation" semantics of FQ have the most impact on reducing inter-flow induced latency.  The "codel" part (AQM) helps with intra-flow latency, which is usually much less important once flow isolation is in place, but is still worth having.

So, jonathan, this portion of the debate leaked over into
https://github.com/systemd/systemd/issues/9725

And I lost a great deal of hair over it. the codel portion is way
worth it on "end-systems".

>
>  - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619