[Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

I could hardly contain my delight in seeing sqm-scripts and luci-app-sqm
enter the openwrt mainline packages repository.

thx hnyman, toke, sebastian, et al for pushing this up!

In other news, after stephen hemminger gave his talk at linux plumbers...

http://lwn.net/Articles/616241/

fedora and systemd are considering a change in their distro's default
of pfifo_fast to fq_codel:

http://osdir.com/ml/general/2014-10/msg34011.html

It's been a rather good week, and all I've had to do is sit back and
munch popcorn while
it happened. (well, I've been off hacking at make-wifi-fast and the
inbound rate limiting
problem)

-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

On Fri, 17 Oct 2014 11:56:01 -0700, Dave Taht said:

> fedora and systemd are considering a change in their distro's default
> of pfifo_fast to fq_codel:
> 
> http://osdir.com/ml/general/2014-10/msg34011.html

systemd is a distro now?  Damn, talk about mission creep.. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

See:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=e6c253e363dee77ef7e5c5f44c4ca55cded3fd47

There is a scene in Monty Python's "Life of Brian" that comes to mind.
A bunch of revolutionaries are discussing various forms of paper
protest, and world domination, and Judith shows up full of the news
that Brian is being crucified  that day...

"Reg, for God's sake, it's perfectly simple. All you've got to do is
to go out of that door now, and try to stop the Romans' nailing him
up! It's happening, Reg! Something's actually happening, Reg! Can't
you understand?"

Frustrated she leaves the room, and the group takes up a new motion
according to parlamentary rules...

(for the full scene, see: http://www.montypython.net/brianmm3.php )

That's what the last couple years of endless talks and ietf meetings
have felt like....

All that said, doubtless we'll have various snags and difficulties on
the way to more full deployment, various new techniques will appear
and evolve, and

/me goes back to bed whistling "always look on the bright side of life"

On Fri, Oct 17, 2014 at 12:30 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Fri, 17 Oct 2014 11:56:01 -0700, Dave Taht said:
>
>> fedora and systemd are considering a change in their distro's default
>> of pfifo_fast to fq_codel:
>>
>> http://osdir.com/ml/general/2014-10/msg34011.html
>
> systemd is a distro now?  Damn, talk about mission creep.. :)



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

On Mon, 20 Oct 2014 10:03:55 -0700, Dave Taht said:
> See:
> 
> http://cgit.freedesktop.org/systemd/systemd/commit/?id=e6c253e363dee77ef7e5c5f44c4ca55cded3fd47

Those guys must be stopped.  Yet another time they tweak a kernel parameter
without bothering to check the return code.

(Yes, I'm still pissed at them for suddenly deciding to require cgroup support
in the kernel, and not even adding a check at startup and a printf "Sorry,
no cgroup support found".  It just kept going, trying to use cgroups support
taht wasn't there, and hilarity and hijinks ensued...


[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Michal Schmidt]

On 10/20/2014 07:41 PM, Valdis.Kletnieks@vt.edu wrote:
> On Mon, 20 Oct 2014 10:03:55 -0700, Dave Taht said:
>> See:
>>
>> http://cgit.freedesktop.org/systemd/systemd/commit/?id=e6c253e363dee77ef7e5c5f44c4ca55cded3fd47
> 
> Those guys must be stopped.  Yet another time they tweak a kernel parameter
> without bothering to check the return code.

Well, the return code is checked, but in case of -ENOENT the error
message is intentionally downgraded to debug level:
http://cgit.freedesktop.org/systemd/systemd/tree/src/sysctl/sysctl.c?id=14f27b4e3b009d10bb9a3b43b74585c73a7c7626#n105

I guess the intention is to not bother warning about sysctls that get
removed from new kernels. Unfortunately, writing an invalid
default_qdisc returns -ENOENT, causing write_string_file() to return the
same, confusing apply_sysctl() into thinking the sysctl does not exist
at all. We can certainly improve that.

> (Yes, I'm still pissed at them for suddenly deciding to require cgroup
> support in the kernel, and not even adding a check at startup and a
> printf "Sorry, no cgroup support found".  It just kept going, trying
> to use cgroups support taht wasn't there, and hilarity and hijinks
> ensued...

Suddenly? CONFIG_CGROUPS was systemd's requirement from its very
beginning. It used to print:

   Failed to mount /sys/fs/cgroup: No such file or directory.

and stop booting at this point.
This was changed in 2011 in response to Ingo Molnar's request to be
able to test !CGROUPS kernels. So systemd try to boot after printing
this warning:

   CONFIG_CGROUPS was not set when your kernel was compiled.
   Systems without control groups are not supported.
   We will now sleep for 10s, and then continue boot-up.
   Expect breakage and please do not file bugs.
   Instead fix your kernel and enable CONFIG_CGROUPS.

Since running on !CGROUPS kernels was not in anybody's test plans,
the ability to boot without cgroups later bitrotted. When it was noticed
in May 2014, the code was reverted to again consider the failure to
mount cgroups as fatal and print the original error message.

I don't know what hilarity you observed, but at no point should the
failure to mount cgroups have been silent.

Regards,
Michal

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On Tue, 21 Oct 2014 16:50:59 +0200, Michal Schmidt said:

> Suddenly? CONFIG_CGROUPS was systemd's requirement from its very
> beginning. It used to print:
>
>    Failed to mount /sys/fs/cgroup: No such file or directory.
>
> and stop booting at this point.

Actually, no. It *wasn't* there from "its very beginning".

See Fedora bug https://bugzilla.redhat.com/show_bug.cgi?id=628004

There was a period of time when it did *NOT* print that message and give
you a hint of what was wrong.

And an indicator of the problems with the systemd design team mentality
is that the comment was "be nice to Ingo Molnar", rather than admitting
that yes, printing that the kernel didn't have a required feature might
be a good idea....

(Yes, I personally got bit by this one myself...)


[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

This is not the place to engage in the systemd debate.

We use procd in openwrt because it fits in teeny routers, and it has its
own problems.

I would, in fact, like it if more devs, engaged in auditing, coding for,
and improving *all* the init replacements. The flamage vs fixage ratio
elsewhere is way out of hand, and it bugs me a lot.

The kernel to userspace transition is one of the most finicky, difficult,
and painfully visible areas in which to work, and I would prefer
constructive coding and tips shared here (and elsewhere) between people
trying to make things better at boot, and to work together to make
meaningful change, rather than rehashing old problems and scars.

I note:

I would appreciate some interest from the systemd folk into the work of
ietf  homenet, and into  things like hnetd and mdns proxy. (and the reverse
from those working on those concepts) The excessively dynamic nature of the
modern ipv6 rollout has led to a need to tightly integrate address
assignment,  routing and naming, which helped delay barrier breaker a year
and it's not clear to me to what extent the modern linux server or desktop
can deal with that and still provide useful services, nor concepts like
source specific routing, and better integration with 802.11 and 802.14.

[-- Attachment #2: Type: text/html, Size: 1390 bytes --]

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Michal Schmidt]

On 10/21/2014 06:57 PM, Dave Taht wrote:
> This is not the place to engage in the systemd debate.

I agree. I wrote one more reply on that topic, but did not and will not
send it.

> I would appreciate some interest from the systemd folk into the work
> of ietf  homenet, and into  things like hnetd and mdns proxy. (and the
> reverse from those working on those concepts) The excessively dynamic
> nature of the modern ipv6 rollout has led to a need to tightly
> integrate address assignment,  routing and naming, which helped delay
> barrier breaker a year and it's not clear to me to what extent the
> modern linux server or desktop can deal with that and still provide
> useful services, nor concepts like source specific routing, and better
> integration with 802.11 and 802.14.

Adding Tom Gundersen (the author of systemd-networkd) to CC in case he
has any comments about this.

Thanks,
Michal

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Tom Gundersen]

On Tue, Oct 21, 2014 at 7:05 PM, Michal Schmidt <mschmidt@redhat.com> wrote:
> On 10/21/2014 06:57 PM, Dave Taht wrote:
>> This is not the place to engage in the systemd debate.
>
> I agree. I wrote one more reply on that topic, but did not and will not
> send it.
>
>> I would appreciate some interest from the systemd folk into the work
>> of ietf  homenet, and into  things like hnetd and mdns proxy. (and the
>> reverse from those working on those concepts) The excessively dynamic
>> nature of the modern ipv6 rollout has led to a need to tightly
>> integrate address assignment,  routing and naming, which helped delay
>> barrier breaker a year and it's not clear to me to what extent the
>> modern linux server or desktop can deal with that and still provide
>> useful services, nor concepts like source specific routing, and better
>> integration with 802.11 and 802.14.
>
> Adding Tom Gundersen (the author of systemd-networkd) to CC in case he
> has any comments about this.

Thanks for the CC Michal.

I have now subscribed to cerowrt-devel (long overdue), and I would
very much appreciate any comments you guys may have on our networking
work in systemd. In particular, if there are any more tweaks like
making fq_codel the deafult, which would be the reasonable choice for
95% of users (most of whom don't know about these things and would
otherwise never touch them), we are very open to suggestions.

As pointed out earlier in this thread, systemd is obviously not a
distro, and the distros have the final say, but I think it makes sense
for us to provide reasonable default sysctl entries wherever we can
(and rather let distros override this if they want).

Cheers,

Tom

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Michal Schmidt]

On 10/21/2014 07:24 PM, Tom Gundersen wrote:
> I have now subscribed to cerowrt-devel (long overdue), and I would
> very much appreciate any comments you guys may have on our networking
> work in systemd. In particular, if there are any more tweaks like
> making fq_codel the deafult, which would be the reasonable choice for
> 95% of users (most of whom don't know about these things and would
> otherwise never touch them), we are very open to suggestions.

An idea: Can networkd configure interfaces' txqueuelen?
(Though with BQL and codel maybe it's not that important anymore.)

Michal

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[David Personette]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

BQL is still only on a relative handful of all the Ethernet drivers IIRC
(If I Recall Correctly). So it could still help out considerably...

-- 
David Personette
Network & System Engineer
434.260.1337


On Tue, Oct 21, 2014 at 1:44 PM, Michal Schmidt <mschmidt@redhat.com> wrote:

> On 10/21/2014 07:24 PM, Tom Gundersen wrote:
> > I have now subscribed to cerowrt-devel (long overdue), and I would
> > very much appreciate any comments you guys may have on our networking
> > work in systemd. In particular, if there are any more tweaks like
> > making fq_codel the deafult, which would be the reasonable choice for
> > 95% of users (most of whom don't know about these things and would
> > otherwise never touch them), we are very open to suggestions.
>
> An idea: Can networkd configure interfaces' txqueuelen?
> (Though with BQL and codel maybe it's not that important anymore.)
>
> Michal
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 1776 bytes --]

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Michal Schmidt]

On 10/21/2014 07:44 PM, Michal Schmidt wrote:
> An idea: Can networkd configure interfaces' txqueuelen?

... or ethtool options such as ring sizes, various offloads?

Michal

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Tom Gundersen]

On Tue, Oct 21, 2014 at 7:44 PM, Michal Schmidt <mschmidt@redhat.com> wrote:
> On 10/21/2014 07:24 PM, Tom Gundersen wrote:
>> I have now subscribed to cerowrt-devel (long overdue), and I would
>> very much appreciate any comments you guys may have on our networking
>> work in systemd. In particular, if there are any more tweaks like
>> making fq_codel the deafult, which would be the reasonable choice for
>> 95% of users (most of whom don't know about these things and would
>> otherwise never touch them), we are very open to suggestions.
>
> An idea: Can networkd configure interfaces' txqueuelen?
> (Though with BQL and codel maybe it's not that important anymore.)

Hm, the way I read the docs, figuring out the "good" values is not
that straight-forward, and doing this will anyway be obsolete soon, so
not sure we should be setting anything by default. However, we
probably should make it much simpler to configure. We could add
support for both ringbuffer and quelength sizes to our link files [0],
so admins colud implement the bufferbloat recommendations by doing
something like:

----8<------
/etc/systemd/network/00-wlan.link
[Match]
Type=wlan

[Link]
TransmitRingBuffer=4
TransmitQueueLength=16
----8<------

(suggestions welcome for the naming of the variables and also for man
page sections).

These settings would then be applied by udev to any udev interface as
it appears (and before libudev notifies applications about its
existence).

Does something like this make sense?

Cheers,

Tom

[0]: <http://www.freedesktop.org/software/systemd/man/systemd.link.html>
[1]: <http://www.bufferbloat.net/projects/bloat/wiki/Linux_Tips>

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

On Tue, Oct 21, 2014 at 11:06 AM, Tom Gundersen <teg@jklm.no> wrote:
> On Tue, Oct 21, 2014 at 7:44 PM, Michal Schmidt <mschmidt@redhat.com> wrote:
>> On 10/21/2014 07:24 PM, Tom Gundersen wrote:
>>> I have now subscribed to cerowrt-devel (long overdue), and I would

I am curious if you or michal are also openwrt or cerowrt users? Or
are running things like sch_fq or fq_codel on your desktops and
servers?

Having native, first hand experience with this stuff would be a good
guideline. There
is a lot to like about the new fq scheduler for servers and maybe for hosts.

And "cake" continues to progress.

>>> very much appreciate any comments you guys may have on our networking
>>> work in systemd. In particular, if there are any more tweaks like
>>> making fq_codel the deafult, which would be the reasonable choice for
>>> 95% of users (most of whom don't know about these things and would
>>> otherwise never touch them), we are very open to suggestions.
>>
>> An idea: Can networkd configure interfaces' txqueuelen?
>> (Though with BQL and codel maybe it's not that important anymore.)

One thing that is missed by people that calculate BDP is that they
usually do the math one way, with the biggest packet size or an
average packet size. There are several problems with this:

1) With the advent of TSO and GSO offloads, the packet size
on servers can bloat up to 64k each. Multiply this by 1256 (txqueuelen +
the typical size of a tx ring) packets and you
can see all the pre-BQL, pre codel latency in all it's glory,
particularly at lower rates.

There's a paper on this...

2) Most client workloads are ack dominated, tending towards 66 bytes each
with some larger packets for http get requests, dns and voip. At this level
a queue with only 1000 small packets is 3 orders of magnitude smaller,
and until some recent work, could be starved by other processing on the system.

3) txqueuelen only has effect on certain qdiscs. In the case of pfifo_fast, you
can and do actually hit that limit, but in the aqms (*codel, pie, red,
ared, sfqred), the
limit is just there to keep from running out of resources - it's
otherwise really
hard to hit in those qdiscs as they start shooting or marking packets
long before
the limit is hit.

So... fiddling with txqueuelen or the ring buffer sizes is something
of a losing game. A qdisc
(like bfifo or *codel) that buffered up acks or big packets with a
byte, rather than packet limit,
is saner, along with BQL underneath.

> Hm, the way I read the docs, figuring out the "good" values is not
> that straight-forward, and doing this will anyway be obsolete soon, so
> not sure we should be setting anything by default.

Tend to agree.

I am generally allergic to TSO/GSO/GRO/LFO offloads at speeds below
100mbit, (although, sigh, I found one still shipping box from alix
with a geode in it that benefits from gso slightly (being able to push
out 60Mbit rather than 40))

and certainly txqueuelen is just plain too big at these speeds, but
you'd have to detect the link rate in order to change it to something
saner.

These show the difference in pfifo_fast on the current beagle at
100mbit with txqueuelen 1000 and 100, offloads off.


http://snapon.lab.bufferbloat.net/~d/beagle_nobql/pfifo_nobql_tsq3028txqueue1000.svg
http://snapon.lab.bufferbloat.net/~d/beagle_nobql/pfifo_nobql_tsq3028txqueue100.svg

(there are a ton of results on the beagle here in this directory, at
different speeds, and buffering, before I got around to actually
adding bql to it (even more results in this dir, data sets easily
compared with netperf-wrapper))

http://snapon.lab.bufferbloat.net/~d/beagle_bql/pfifo_bql_tsq3028txqueue100.svg
http://snapon.lab.bufferbloat.net/~d/beagle_bql/fq_bql_tsq3028.svg
http://snapon.lab.bufferbloat.net/~d/beagle_bql/fq_codel_bql_tsq3028.svg

http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png

You can see that BQL makes the most difference in the latency.

I keep hoping for saner tuning of these offloads at higher speeds on
better hardware, but it appears as of the last kernel version I tested
thoroughly TSO/GSO is still needed on devices with gigE interfaces.

http://snapon.lab.bufferbloat.net/~cero2/nuc-to-puck/results.html

And then there's (sigh) wifi.

> However, we
> probably should make it much simpler to configure. We could add
> support for both ringbuffer and quelength sizes to our link files [0],
> so admins colud implement the bufferbloat recommendations by doing
> something like:
>
> ----8<------
> /etc/systemd/network/00-wlan.link
> [Match]
> Type=wlan

As for wifi, there is much now published on all the problems there.

A recent summary of what seems to be needed I did at ieee: (see pp 23- )

http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf

There is no ring buffer. Often tuning down txqueuelen is a very good idea, with
today's wifi drivers being MASSIVELY overbuffered. Better to apply
fq_codel, for now, and work on restructuring that entire subsystem.

> [Link]
> TransmitRingBuffer=4
> TransmitQueueLength=16

Regrettably many devices do not respond to tuning such as this.
Example the e1000e
doesn't let you get below 64 entries in the ring buffer, and the
ar71xx allows it, but crashes...

(thankfully both have BQL)

> ----8<------
>
> (suggestions welcome for the naming of the variables and also for man
> page sections).
>
> These settings would then be applied by udev to any udev interface as
> it appears (and before libudev notifies applications about its
> existence).
>
> Does something like this make sense?

Regettably, no. I think printing a warning somewhere, when BQL is not detected
on an interface going up would clue more towards getting BQL adopted more fully.

"BQL not detected on interface X, latency may be compromised, beg your
vendor for BQL support" -

>
> Cheers,
>
> Tom
>
> [0]: <http://www.freedesktop.org/software/systemd/man/systemd.link.html>
> [1]: <http://www.bufferbloat.net/projects/bloat/wiki/Linux_Tips>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

> http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png
>
> You can see that BQL makes the most difference in the latency.

And ALSO that these fixes also improved system throughput enormously.
This is partially due to the improvement in ack clocking you get from
reduced RTTs, partially due to improved cache behavior (shorter
queues), and partially continual improvements elsewhere in the tcp
portions of the stack.

With more recent kernels...

I now get full throughput from the beagles in both directions with the
3.16 kernel,
the stil out of tree bql patch, and either fq or fq_codel. I haven't
got around to plotting all those results (they are from kathie's new
lab), but they are here:
http://snapon.lab.bufferbloat.net/~d/pollere/

There is a small buffered tail dropping switch in the way, on these
later data sets. There was some puzzling behavior on the e1000e that I
need to poke into in a more controlled setting.

As for other tunables on hosts, TCP small queues might be amiable to
some tuning, but that too may well evolve further in kernelspace.

-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

On Tue, Oct 21, 2014 at 12:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png
>>
>> You can see that BQL makes the most difference in the latency.
>
> And ALSO that these fixes also improved system throughput enormously.

Meant to include that plot.

http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewithbql.png

You can disregard the decline in download bandwidth (as we are also
sending 5x as many acks and measurement data, which is not counted
in that part of the plot)

> This is partially due to the improvement in ack clocking you get from
> reduced RTTs, partially due to improved cache behavior (shorter
> queues), and partially continual improvements elsewhere in the tcp
> portions of the stack.
>
> With more recent kernels...
>
> I now get full throughput from the beagles in both directions with the
> 3.16 kernel,
> the stil out of tree bql patch, and either fq or fq_codel. I haven't
> got around to plotting all those results (they are from kathie's new
> lab), but they are here:
> http://snapon.lab.bufferbloat.net/~d/pollere/

The latency spikes are generally due to not having BQL, probably:
http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle-3.8-nobql-fq-fq_codel.png

This is using the new fq scheduler on both sides, with BQL enabled.

http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle_3.16-fq-fq-no-offloads.png

The switch most likely is prioritizing EF marked packets. (as is
sch_fq). Most of the buffering is in the switch, not the host, now.
(the prior results I showed had no switch in the way)

> There is a small buffered tail dropping switch in the way, on these
> later data sets. There was some puzzling behavior on the e1000e that I
> need to poke into in a more controlled setting.
>
> As for other tunables on hosts, TCP small queues might be amiable to
> some tuning, but that too may well evolve further in kernelspace.

So I have now drowned you in data on one architecture and setup. The
most thoroughly publicly analyzed devices and drivers are the ar71xx,
e1000e, and beaglebone at this point.

The use of fq_codel in a qos system (artificially rate limited using
htb, hfsc, or tbf) is pretty well proven to be a huge win at this
point.

At line rates fq and fq_codel still help quite a bit without a BQL
enabled driver, BQL is needed for best results. I don't know to what
extent the BQL enabled drivers already cover the marketplace, it was
generally my assumption that the e1000e counted for a lot...

http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers

And thus with all the positive results so far, more wider distribution
of the new stuff
on more devices outside the sample set of those on the bloat,
cerowrt-devel and codel lists (about 500 people all told),

and all ya gotta do is turn it on.

This gets me to the stopping point that we hit a whlle back, which was
reliably determining if a good clocksource was present in the system.
Somehow. clock_getres, perhaps?

> --
> Dave Täht
>
> http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Tom Gundersen]

On Tue, Oct 21, 2014 at 10:59 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Tue, Oct 21, 2014 at 12:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>> http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png
>>>
>>> You can see that BQL makes the most difference in the latency.
>>
>> And ALSO that these fixes also improved system throughput enormously.
>
> Meant to include that plot.
>
> http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewithbql.png
>
> You can disregard the decline in download bandwidth (as we are also
> sending 5x as many acks and measurement data, which is not counted
> in that part of the plot)
>
>> This is partially due to the improvement in ack clocking you get from
>> reduced RTTs, partially due to improved cache behavior (shorter
>> queues), and partially continual improvements elsewhere in the tcp
>> portions of the stack.
>>
>> With more recent kernels...
>>
>> I now get full throughput from the beagles in both directions with the
>> 3.16 kernel,
>> the stil out of tree bql patch, and either fq or fq_codel. I haven't
>> got around to plotting all those results (they are from kathie's new
>> lab), but they are here:
>> http://snapon.lab.bufferbloat.net/~d/pollere/
>
> The latency spikes are generally due to not having BQL, probably:
> http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle-3.8-nobql-fq-fq_codel.png
>
> This is using the new fq scheduler on both sides, with BQL enabled.
>
> http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle_3.16-fq-fq-no-offloads.png
>
> The switch most likely is prioritizing EF marked packets. (as is
> sch_fq). Most of the buffering is in the switch, not the host, now.
> (the prior results I showed had no switch in the way)
>
>> There is a small buffered tail dropping switch in the way, on these
>> later data sets. There was some puzzling behavior on the e1000e that I
>> need to poke into in a more controlled setting.
>>
>> As for other tunables on hosts, TCP small queues might be amiable to
>> some tuning, but that too may well evolve further in kernelspace.
>
> So I have now drowned you in data on one architecture and setup. The
> most thoroughly publicly analyzed devices and drivers are the ar71xx,
> e1000e, and beaglebone at this point.
>
> The use of fq_codel in a qos system (artificially rate limited using
> htb, hfsc, or tbf) is pretty well proven to be a huge win at this
> point.
>
> At line rates fq and fq_codel still help quite a bit without a BQL
> enabled driver, BQL is needed for best results. I don't know to what
> extent the BQL enabled drivers already cover the marketplace, it was
> generally my assumption that the e1000e counted for a lot...
>
> http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers
>
> And thus with all the positive results so far, more wider distribution
> of the new stuff
> on more devices outside the sample set of those on the bloat,
> cerowrt-devel and codel lists (about 500 people all told),
>
> and all ya gotta do is turn it on.

Thanks for the very detailed info Dave.

What I'm taking from this is that at the moment we should not be doing
anything more than what we already do (the sysctl changes that Michal
pushed), and that any improvements should be made in the kernel
drivers (BQL support in particular).

What we could do in the future, is in case you have kernel features
that should be enabled by default as they benefit the general user,
but where you cannot change the kernel defaults due to backwards
compat, we can (as we did with fq) add these to the recommended sysctl
files shipped with systemd. This is of course just a gentle nudge for
people to use features, distros/admins can still override them.

> This gets me to the stopping point that we hit a whlle back, which was
> reliably determining if a good clocksource was present in the system.
> Somehow. clock_getres, perhaps?

I appear to have missed the context on this. In what situation do we
(userspace) need to care about the existence of a high quality
clocksource?

Cheers,

Tom

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Dave Taht]

On Thu, Dec 4, 2014 at 8:09 AM, Tom Gundersen <teg@jklm.no> wrote:
> On Tue, Oct 21, 2014 at 10:59 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Tue, Oct 21, 2014 at 12:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>> http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png
>>>>
>>>> You can see that BQL makes the most difference in the latency.
>>>
>>> And ALSO that these fixes also improved system throughput enormously.
>>
>> Meant to include that plot.
>>
>> http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewithbql.png
>>
>> You can disregard the decline in download bandwidth (as we are also
>> sending 5x as many acks and measurement data, which is not counted
>> in that part of the plot)
>>
>>> This is partially due to the improvement in ack clocking you get from
>>> reduced RTTs, partially due to improved cache behavior (shorter
>>> queues), and partially continual improvements elsewhere in the tcp
>>> portions of the stack.
>>>
>>> With more recent kernels...
>>>
>>> I now get full throughput from the beagles in both directions with the
>>> 3.16 kernel,
>>> the stil out of tree bql patch, and either fq or fq_codel. I haven't
>>> got around to plotting all those results (they are from kathie's new
>>> lab), but they are here:
>>> http://snapon.lab.bufferbloat.net/~d/pollere/
>>
>> The latency spikes are generally due to not having BQL, probably:
>> http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle-3.8-nobql-fq-fq_codel.png
>>
>> This is using the new fq scheduler on both sides, with BQL enabled.
>>
>> http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle_3.16-fq-fq-no-offloads.png
>>
>> The switch most likely is prioritizing EF marked packets. (as is
>> sch_fq). Most of the buffering is in the switch, not the host, now.
>> (the prior results I showed had no switch in the way)
>>
>>> There is a small buffered tail dropping switch in the way, on these
>>> later data sets. There was some puzzling behavior on the e1000e that I
>>> need to poke into in a more controlled setting.
>>>
>>> As for other tunables on hosts, TCP small queues might be amiable to
>>> some tuning, but that too may well evolve further in kernelspace.
>>
>> So I have now drowned you in data on one architecture and setup. The
>> most thoroughly publicly analyzed devices and drivers are the ar71xx,
>> e1000e, and beaglebone at this point.
>>
>> The use of fq_codel in a qos system (artificially rate limited using
>> htb, hfsc, or tbf) is pretty well proven to be a huge win at this
>> point.
>>
>> At line rates fq and fq_codel still help quite a bit without a BQL
>> enabled driver, BQL is needed for best results. I don't know to what
>> extent the BQL enabled drivers already cover the marketplace, it was
>> generally my assumption that the e1000e counted for a lot...
>>
>> http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers
>>
>> And thus with all the positive results so far, more wider distribution
>> of the new stuff
>> on more devices outside the sample set of those on the bloat,
>> cerowrt-devel and codel lists (about 500 people all told),
>>
>> and all ya gotta do is turn it on.
>
> Thanks for the very detailed info Dave.
>
> What I'm taking from this is that at the moment we should not be doing
> anything more than what we already do (the sysctl changes that Michal
> pushed), and that any improvements should be made in the kernel
> drivers (BQL support in particular).

I would like to see more pushback from everyone in the universe on new
ethernet drivers in particular, to always have BQL.

it is very useful at all speeds and there has been a big push for it
in the 10GigE+ space, but not enough at gigE and below.

So we are always asking for it when a new
driver arrives, or new work on an old driver arrives. (recently the nvneta
for example). It would be nice if there was a big push to add BQL
to older drivers (talk to jesper about that)

Recently xmit_more support arrived in the kernel mainline,
it needs BQL, and it's *very good*.

> What we could do in the future, is in case you have kernel features
> that should be enabled by default as they benefit the general user,
> but where you cannot change the kernel defaults due to backwards
> compat, we can (as we did with fq) add these to the recommended sysctl
> files shipped with systemd. This is of course just a gentle nudge for
> people to use features, distros/admins can still override them.

Cool. But I don't mind testing stuff at this level for a year or more before
inflicting it on users. :)

Relative to the kernel options, IPV6_SUBTREES is needed for good
quality source specific routing support. So far as I know that has defaulted
to on in most distros for a long time.

Relative to device naming, we are increasingly living in a world where
IP addresses come and go, and firewalls are having a lot of trouble
dealing with that - (two recent examples:

http://www.bufferbloat.net/issues/438

https://github.com/sbyx/odhcp6c/issues/27
)

My own take on this was to name devices after their security level
and then type in cerowrt, which is not an idea that has been taken up
anywhere else - but that allows for never having to reload the firewall and
loss conntrack state on an address change or interface add, if more work was put
into it. It does look like nftables has some simpler means for adding/
deleting interfaces from a security model, but many solutions
seem to require changing a lot of state and introducing brief windows
of vulnerability or network disconnectivity.

In my case I regularly run with two or more interfaces always connected
to the internet, I never need to down one and up the other, or change
default routes, I just unplug in one place, and go to another, and
plug in again... and all my sessions stay up.... [1]. Admittedly now that
I use mosh more often (has ipv6 support in git head), I am not
as bothered by disconnectivity as I used to be.....

Lastly I would like to see the dhcp-related work in systemd
paying attention to the kinds of issues the ipv6 homenet wg in the
ietf and openwrt (and early distributors of ipv6 native addressing,
like comcast)

are dealing with... and things like hnetd looked at.

>> This gets me to the stopping point that we hit a whlle back, which was
>> reliably determining if a good clocksource was present in the system.
>> Somehow. clock_getres, perhaps?
>
> I appear to have missed the context on this. In what situation do we
> (userspace) need to care about the existence of a high quality
> clocksource?

There are a (very few) cpus/arches that lack a fast and high quality timesource.

way older x86 boxes lacking hpet, for example. Network performance on these
devices tends to be pretty bad in the first place, and it is simply unknown at
this point how much the per-packet timestamping in codel would change things.

Nearly all modern chips have a fast, onchip, single cycle timesource, so this
is not a problem for those.
>
> Cheers,
>
> Tom

[1] In fact I just noticed my ethernet was unplugged, and had been for a
while....

d@ganesha:~/git/tinc/src$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr d8:50:e6:a0:b6:32
          inet addr:172.26.16.2  Bcast:172.26.16.255  Mask:255.255.255.0
          inet6 addr: fe80::da50:e6ff:fea0:b632/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:9508669 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8321719 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9637122680 (9.6 GB)  TX bytes:5583512622 (5.5 GB)

d@ganesha:~/git/tinc/src$ ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr 24:0a:64:cc:24:7d
          inet addr:172.26.17.228  Bcast:255.255.255.255  Mask:0.0.0.0
          inet6 addr: fe80::260a:64ff:fecc:247d/64 Scope:Link
          inet6 addr: 2601:9:3600:b397:260a:64ff:fecc:242d/128 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1743729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1489781 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1077109813 (1.0 GB)  TX bytes:504903872 (504.9 MB)

d@ganesha:~/git/tinc/src$ ip route
default via 172.26.17.224 dev wlan0  proto 42 onlink
69.181.216.0/22 via 172.26.17.224 dev wlan0  proto 42 onlink
169.254.0.0/16 dev eth0  scope link  metric 1000
172.26.16.0/24 dev eth0  proto kernel  scope link  src 172.26.16.2
172.26.16.1 via 172.26.17.227 dev wlan0  proto 42 onlink
172.26.16.3 via 172.26.17.227 dev wlan0  proto 42 onlink
172.26.17.0/24 via 172.26.17.224 dev wlan0  proto 42 onlink
172.26.17.3 via 172.26.17.227 dev wlan0  proto 42 onlink
172.26.17.227 via 172.26.17.227 dev wlan0  proto 42 onlink
192.168.7.2 via 172.26.17.227 dev wlan0  proto 42 onlink
d@ganesha:~/git/tinc/src$





-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Toke Høiland-Jørgensen]

Tom Gundersen <teg@jklm.no> writes:

> I have now subscribed to cerowrt-devel (long overdue), and I would
> very much appreciate any comments you guys may have on our networking
> work in systemd. In particular, if there are any more tweaks like
> making fq_codel the deafult, which would be the reasonable choice for
> 95% of users (most of whom don't know about these things and would
> otherwise never touch them), we are very open to suggestions.

One thing that has gone into openwrt recently but is not supported in
systemd-networkd is source-specific routing. Since I got an internet
connection too fast for the WNDR to keep up, I've transitioned to an x86
box running Arch Linux for that link. It uses systemd-networkd to setup
most of the networking (which works very well!), but one thing missing
is support for source-specific routing.

Right now, I have a systemd unit to set up my IPv6 tunnel with this in
it:

ExecStart=/usr/bin/ip tunnel add he-ipv6 mode sit remote 216.66.80.90 ttl 255 dev enp2s0
ExecStart=/usr/bin/ip link set he-ipv6 up
ExecStart=/usr/bin/ip addr add 2001:470:xx::2/64 dev he-ipv6
ExecStart=/usr/bin/ip route add default via 2001:470:xx::1 from 2001:470:yy::/48 proto static
ExecStart=/usr/bin/ip route add default via 2001:470:xx::1 from 2001:470:xx::2/128 proto static
ExecStart=/usr/bin/ip route add default from ::/128 dev he-ipv6

The top three lines I can replace by a file in /etc/systemd/network, but
not the bottom three.


Also, having a way to make systemd units depend on network interface
availability (and configuration state) would be neat; to do things like
start up a VPN daemon when the WAN connection becomes available. :)

-Toke

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Tom Gundersen]

Hi Toke,

On Wed, Oct 22, 2014 at 1:18 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Tom Gundersen <teg@jklm.no> writes:
>
>> I have now subscribed to cerowrt-devel (long overdue), and I would
>> very much appreciate any comments you guys may have on our networking
>> work in systemd. In particular, if there are any more tweaks like
>> making fq_codel the deafult, which would be the reasonable choice for
>> 95% of users (most of whom don't know about these things and would
>> otherwise never touch them), we are very open to suggestions.
>
> One thing that has gone into openwrt recently but is not supported in
> systemd-networkd is source-specific routing. Since I got an internet
> connection too fast for the WNDR to keep up, I've transitioned to an x86
> box running Arch Linux for that link. It uses systemd-networkd to setup
> most of the networking (which works very well!), but one thing missing
> is support for source-specific routing.
>
> Right now, I have a systemd unit to set up my IPv6 tunnel with this in
> it:
>
> ExecStart=/usr/bin/ip tunnel add he-ipv6 mode sit remote 216.66.80.90 ttl 255 dev enp2s0
> ExecStart=/usr/bin/ip link set he-ipv6 up
> ExecStart=/usr/bin/ip addr add 2001:470:xx::2/64 dev he-ipv6
> ExecStart=/usr/bin/ip route add default via 2001:470:xx::1 from 2001:470:yy::/48 proto static
> ExecStart=/usr/bin/ip route add default via 2001:470:xx::1 from 2001:470:xx::2/128 proto static
> ExecStart=/usr/bin/ip route add default from ::/128 dev he-ipv6
>
> The top three lines I can replace by a file in /etc/systemd/network, but
> not the bottom three.

I finally got around to have a look at this, and I now added support
for source routing. It appears to work for me (current git) with:


wireless.network ---8<-----------------------

[Match]
Name=wlp3s0

[Network]
DHCP=yes
Tunnel=he

he.netdev ---8<-----------------------------------------

[NetDev]
Name=he
Kind=sit

[Tunnel]
Remote=222.333.444.555
TTL=255

he.network ---8<-----------------------------------------

[Match]
Name=he-ipv6

[Network]
Address=2001:470:11::2/64

[Route]
Gateway=2001:470:11::1
Source=2001:470:22::/48

[Route]
Gateway=2001:470:11::1
Source=2001:470:11::2/128

[Route]
Source=::/128

---8<-----------------------------------------

Let me know if you experience any issues!

> Also, having a way to make systemd units depend on network interface
> availability (and configuration state) would be neat; to do things like
> start up a VPN daemon when the WAN connection becomes available. :)

We don't have a nice solution for this yet, but it is something we are
keeping in mind and will have to solve somehow (much of the common
things will be integrated with networkd directly).

Cheers,

Tom

Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default

[Toke Høiland-Jørgensen]

Tom Gundersen <teg@jklm.no> writes:

> I finally got around to have a look at this, and I now added support
> for source routing. It appears to work for me (current git) with:

Cool, thanks! Will give it a spin when I get a chance and let you know
if I experience any problems :)

-Toke