[Bloat] Some updates on hacking on AQMS

[Bloat] Some updates on hacking on AQMS

[Dave Taht]

I'm going to make some notes against my original posting 'Some notes
on hacking on AQM's' here, because in less than a day, much progress
been made.

On Wed, Jun 8, 2011 at 6:12 AM, Dave Taht <dave.taht@gmail.com> wrote:
> So in addition to hacking on the switch, I've been poking into the behavior of multiple AQM systems in the kernel

...elided...

> Some notes:
>
> There are as many philosophies to AQM as there are shapers and classifiers.
>
> None of the Linux shaper scripts in the field handle ipv6 traffic.

I've got some fixes for this in the 3 (soon to be 4) shapers I'm playing with.

According to the netfilter guys, DDR scheduling + HFSC is the new
'hotness', and I was supplied a script that uses it... (which I think
was noted in the previous thread, if it was private email, I'll
repost)

I have not (as yet) patched DDR support into the openwrt testbed.

>
> HTB is the most commonly used qdisc, handles it's bandwidth limits by packet
> drop
> and doesn't do ECN. It's usually used in conjunction with other qdiscs, too.
>
> An explanation of how diffserv (dsmark) and GRED are supposed to play ball
> together
> (starting here: http://www.opalsoft.net/qos/DS-27.htm) is so amazingly not
> opaque.
>
> SFB remains promising, but until I get a ported tc for it,
> I can't play with it much.

Next week iproute2-2.6.39 may be released, and I'll slam it into
openwrt, assuming it can be made to work with 2.6.37.6.

>
> SFQ is the second most commonly used qdisc, but doesn't balance in ways ESFQ
> could.
>
> ESFQ really looked like a winner and I'm sorry it never made the mainline
> kernel.

ESFQ features were added to SFQ 4 years ago. Few use them, but they
are there. It's actually more flexible than ESFQ was. Core among them
is the ability to not match against flows but against distinct IP
addresses, which makes sense in this bittorrent-ed age.


>
> HFSC is mind-bending as to what it tries to do.

It really seems that ECN support could be added generically for all
qdiscs that currently do packet drop. Creating a generic mark_or_drop
function is easy. The difficulties lie in adding 'marking' counters to
every qdisc and userspace.

The overall effacy of ECN when more widely deployed
remains an unknown, but given the heroic efforts given
packet delivery today across 3g/4g/wireless, providing
a postitive congestion signaling mechanism across those links seems
ever more important.

I've added an ipod, an ipad, and a windows 7 and windows vista box to
the list of machines I can test ECN with.

>
> Any form of fair queuing is useful for ethernet, but actually knowing the
> link rate and port on the switch per dest macaddr would help in load
> balancing streams.
>
> Fair queueing is very bad on wireless when packet aggregation is used.

I currently find the mismatch between wired and wireless to be so
extreme that I'm no longer bridging the two wherever possible, thus
reducing the effects of multicast.

Naturally this introduces problems of its own. As one example, windows
needs a wins proxy supplied in order to 'see' machines on the other
side of a router. MDNS seems to be another issue.

Openwrt, seems to generally come with PIMv1 and PIMv2 disabled, and I
don't know to what extent these protocols are actually used.

IGMPproxy seems to work, but there are probably many more gotchas with
short haul multicast left to be exposed.

Another problem is that the wndr3700 comes without a distinct mac for
one ethernet device, and the switch is either unassigned one or my
code for finding it is wrong.

Still, I'm loving not having wired and wireless bridged,
a busy wired link (lots of arp requests, for example) seemingly does
terrible things to a wireless one when bridged.

>
> PFIFO_FAST is tied to TOS bits, not diffserv bits.
>
> RED is, well, RED.
>
> GRED is far less opaque than RED, as noted earlier.
>

> 802.11e does its prioritization at the vlan layer, not at the TOS or
> diffserv bits. Getting from tos or diffserv to mq* seems painful but I
> haven't looked into it too hard.

Here's a really obscure bit of info gleaned from nbd (and I have not
tried this yet, but it seems really important to do so)

In the mac80211 code....

If on entry you mark a packet's skb->priority field with a value that
matches the 802.1d priority fields (0..7) + 256 - the mac80211 layer
interprets that  and wedges the result into the 4 bands available in
802.11e. (regrettably it's not linear, both 0 and 3 = BE)

Those are BK (background), BE, Video, and Voice.

(without classification, everything ends up in BE)

The interesting thing about the (rather underused) video and voice
catagories is that they actually work at the
physical layer in separate bands across the Cwin.

So, in addition to voice, I'd like to try and wedge all the
non-multicast, low volume 'mice' packets like NTP, ND, etc into the
Voice category, treating 'voice' more like a control plane, than just
voice. This will probably tickle more bugs, but the idea is kind of
fun.

I'll write more about this later.

It's really not clear to me how to properly use 'BK' vs 'BE' given how
it 'sounds on the air'. It seems like a lose from a game theory
perspective, where repurposing voice and video catagories seems more
like a win.

> MQ and MQPrio are horribly underdocumented. I still don't 'get' how to use
> them
> properly (I'm more focused on writing a good classifier at the moment)

I still have no idea what MQ and MQprio are really good for. Perhaps the above?

> iptables seems to think ecn can only be looked at in TCP streams, where (for
> example),
> ecn bits can be copied to the outer header of a udp vpn stream, and marked
> when needed.

This was incorrect. There is a specific ecn-ip-ect (0x.0x3) match that
works against IP.

> ip6tables has no support for looking at ecn except through a u32 match.

Patrick Mchardy added ipv6 support for iptables matches a few minutes
ago, and after review I figure they will end up in net-next...


> iptables -t mangle -A Wireless -p tcp -m tcp --tcp-flags ALL SYN,ACK -m ecn
> --ecn-tcp-ece -m recent --name ecn_enabled --set -m comment --comment 'ECN
> enabled streams'
> iptables -t mangle -A Wireless -p tcp -m tcp --tcp-flags ALL SYN,ACK -m ecn
> ! --ecn-tcp-ece -m recent --name ecn_disabled --set -m comment --comment
> 'ECN disab
> led streams'
>
> iptables -t mangle -F POSTROUTING
> iptables -t mangle -A POSTROUTING -j Wireless
>
> You can see what ips managed to do ECN or not via
>
> cat /proc/net/xt_recent/ecn_*


The patches just submitted to the netfilter list also fix an inversion
match problem.


> But that's just a distraction from trying to converge on a
> decent set of solutions for AQM. I AM happy to report that after getting
> buffer sizes down (via ethtool, a switch patch, txqueuelen) I am finally
> able to reliably see sub 10ms latencies on the wndr3700... but I wake up
> these days, feeling doomed.

I feel less doomed now. Thanks everybody!

Sometimes crying out in frustration (and thoroughly documenting your
problems), really, really works.

My thanks to everybody that jumped in to help, and to find the right
places to find the help, etc.

Amusingly, while I was testing a new build of cerowrt, my link to
comcast went down and I spent 20 minutes blaming my stuff before
figuring out the real cause of the
problem.

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Bloat] Some updates on hacking on AQMS

[Juliusz Chroboczek]

> It really seems that ECN support could be added generically for all
> qdiscs that currently do packet drop. Creating a generic mark_or_drop
> function is easy.

As hinted in a previous message -- please don't.  Every qdisc must be
examined individually to check if it is suitable for ECN.

Consider also the following.  If the administrator specifies a maximum
rate of 100 Mbit/s, he probably doesn't expect the outgoing traffic to
exceed that rate under any circumstances.  If you start marking instead
of dropping, you must make sure that the resulting traffic is still
under 100 Mbit/s, including the marked packets.

-- Juliusz

Re: [Bloat] Some updates on hacking on AQMS

[Dave Taht]

On Thu, Jun 23, 2011 at 4:45 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
>> It really seems that ECN support could be added generically for all
>> qdiscs that currently do packet drop. Creating a generic mark_or_drop
>> function is easy.
>
> As hinted in a previous message -- please don't.  Every qdisc must be
> examined individually to check if it is suitable for ECN.
>
> Consider also the following.  If the administrator specifies a maximum
> rate of 100 Mbit/s, he probably doesn't expect the outgoing traffic to
> exceed that rate under any circumstances.  If you start marking instead
> of dropping, you must make sure that the resulting traffic is still
> under 100 Mbit/s, including the marked packets.

In designing this particular concept I made sure that was an option,
in the mildly recorrected:

http://www.bufferbloat.net/projects/bloat/wiki/Save_the_Ants

Try harder to not shoot the ants, but make ecn marking a first class
option throughout the qdiscs...

if possible..

>
> -- Juliusz
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Bloat] Some updates on hacking on AQMS

[Eric Dumazet]

Le jeudi 23 juin 2011 à 17:10 -0600, Dave Taht a écrit :
> On Thu, Jun 23, 2011 at 4:45 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
> >> It really seems that ECN support could be added generically for all
> >> qdiscs that currently do packet drop. Creating a generic mark_or_drop
> >> function is easy.
> >
> > As hinted in a previous message -- please don't.  Every qdisc must be
> > examined individually to check if it is suitable for ECN.
> >
> > Consider also the following.  If the administrator specifies a maximum
> > rate of 100 Mbit/s, he probably doesn't expect the outgoing traffic to
> > exceed that rate under any circumstances.  If you start marking instead
> > of dropping, you must make sure that the resulting traffic is still
> > under 100 Mbit/s, including the marked packets.
> 
> In designing this particular concept I made sure that was an option,
> in the mildly recorrected:
> 
> http://www.bufferbloat.net/projects/bloat/wiki/Save_the_Ants
> 
> Try harder to not shoot the ants, but make ecn marking a first class
> option throughout the qdiscs...
> 
> if possible..

I believe it was done.

qdisc implementations are not random, but follow extensive research
works.

Before doing a change like that, you must redo all the experiments and
show the pro/cons ;)

Once qdisc queue is _full_, its too late, you have to drop packets
anyway. And its not because at least one flow is not responsive :
It might be because one thousand flows began their life at the very same
moment :(

Re: [Bloat] Some updates on hacking on AQMS

[Dave Taht]

On Thu, Jun 23, 2011 at 11:01 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le jeudi 23 juin 2011 à 17:10 -0600, Dave Taht a écrit :
>> On Thu, Jun 23, 2011 at 4:45 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
>> >> It really seems that ECN support could be added generically for all
>> >> qdiscs that currently do packet drop. Creating a generic mark_or_drop
>> >> function is easy.
>> >
>> > As hinted in a previous message -- please don't.  Every qdisc must be
>> > examined individually to check if it is suitable for ECN.
>> >
>> > Consider also the following.  If the administrator specifies a maximum
>> > rate of 100 Mbit/s, he probably doesn't expect the outgoing traffic to
>> > exceed that rate under any circumstances.  If you start marking instead
>> > of dropping, you must make sure that the resulting traffic is still
>> > under 100 Mbit/s, including the marked packets.
>>
>> In designing this particular concept I made sure that was an option,
>> in the mildly recorrected:
>>
>> http://www.bufferbloat.net/projects/bloat/wiki/Save_the_Ants
>>
>> Try harder to not shoot the ants, but make ecn marking a first class
>> option throughout the qdiscs...
>>
>> if possible..
>
> I believe it was done.
>
> qdisc implementations are not random, but follow extensive research
> works.

All of which is either invalidated, obsolete, or newly revalidated
since the advent of wireless, the increase in using TCP for short
transfers, and nat becoming predominant, the increase in traffic
volume overall, and the introduction of both voice and video calling
on the web, and the introduction of massive amounts of video traffic
to the Internet, sadly most of which are unclassifyable and on port
80...

... The workloads have changed, the characteristics of traffic have
changed, and experiments need to be re-run.

> Before doing a change like that, you must redo all the experiments and
> show the pro/cons ;)

I'm kind of hoping to be building a platform where others can do that easier.

> Once qdisc queue is _full_, its too late, you have to drop packets
> anyway. And its not because at least one flow is not responsive :
> It might be because one thousand flows began their life at the very same
> moment :(

Painfully aware of that.

>
>
>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Bloat] Some updates on hacking on AQMS

[Dave Taht]

On Fri, Jun 24, 2011 at 3:10 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Jun 23, 2011 at 11:01 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> Once qdisc queue is _full_, its too late, you have to drop packets
>> anyway. And its not because at least one flow is not responsive :
>> It might be because one thousand flows began their life at the very same
>> moment :(

Oh, and while I'm touching upon this subject, by default openwt
enables 'syn flood support', which (by default) rate limits syn
attempts to 25/second, bursting to 50, dropping new attempts long
before they can introduce the TCP mice problem, and introducing long
retries for the re-attempted connections.

That's not bandwidth or workload relative, or related to the number of
users, but a flat limit.

While the syn flood rate is configurable (and I've either disabled it
entirely or bumped it up a lot in my own testing - it's easy to see if
you try to, say, access the top 100 web sites all at once - try 100,
see 50 fail..... retry 50, see 25 fail...) ...

Arbitrarily dropping syn attempts at a level well below what a single
browser can introduce today has a dramatic effect on perceived
performance and latency.

This again points to the need for in depth knowledge of what packets
are being dropped, where, why, and when, when evaluating the behavior
of todays networking subsystems. I'm looking forward to being able to
analyze packet drops extensively in the future with the packet drop
analysis tool recently discussed here.

No doubt other routers, gateways and firewalls are making similarly
desparate attempts to smooth the flow of traffic above and beyond the
basic effects of the qdiscs.

by default Openwrt also clamps MSS... blocks icmpv6 'etoobig' messages
to internal networks... and limits 6in4 ipv6 mtu to 1280.

In addition to all the other heroic measures being done elsewhere in
the networking subsystems (forget about wireless for a bit... cable
modems often do synack optimization as one example, and a recent paper
by cablelabs pointed to overbuffering in most modems incorrect by an
order of magnitude) it would be good to have a clearer picture of what
is REALLY going on before making any clear determinations as to the
real effacy of any qos subsystem.

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com