QoS for system critical packets on wireless

QoS for system critical packets on wireless

[Dave Taht]

The biggest fallout of the diffserv work I was trying was observing
that most packets fell into one of 3 buckets:

1) System control and 'MICE' are < less than 1% of all packets. Mice
includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
portion of the stack, in what I'm doing currently. Mice are
desperately needed for the network to continue to function.

2) unresponsive streams and udp (varying, but generally pretty low)
3) responsive tcp streams (closer to 99%)

So applying some level of packet prioritization to babel appears to
make sense - AFTER getting the excessive buffers under control, more
on that further below - so it can do it's work.

At the moment I am arbitrarily classifying babel packets into the CS6
diffserv class, which in the end, is then classifying the result into
the the 802.11e VI or VO classes. (VO being mildly more underused)

I haven't the faintest idea if this actually does anything to the
window in which frames are transmitted via multicast on wireless-b, g
or n. Theoretically the 802.11e VI and VO classes have their own tiny
timeslots to broadcast in separate from BE/BK.

There is no need to use diffserv, merely prioritization using some
other classification scheme be that outright
tc queues, or iptables marking would help, to jump the queues internally.

(I am getting a spectrum analyzer shortly)

There were several (long term!!!) thoughts here:

A) wireless devices are currently making heroic efforts (deep
buffering, exorbitant retries) to get packets through. Seeing a big
delay between transmit time and reception is more an indicator of
congestion than actual packet loss is right now. By the time you see
actual packet loss, the network has often already collapsed
completely.

B) Theoretically using a different 802.11e class reduces the heroism
by some unknown amount.

C) QoS, Packet marking and prioritization of any sort makes babel
control packets jump closer to the head of the internal queues of the
transmitting clients, thus speeding up routing change propagation. By
all means, don't shoot mice, but elephants.

Once you do all this stuff, packet loss comes closer to being a
measure of actual problems in the air, instead of deep in the stack.

D) The ECN bit could be used to indicate congestion on links that
aren't losing packets but are experiencing congestion.

Moving back to item C, I would like to recomend to babel users on
Linux (at least) that they try:

Reduce txqueuelen on their ethernet and wireless devices by a lot! I'm
using 4-16 at present.
Reduce driver buffering by a lot! I cut one driver from 512 packets
buffered to 3, which made voip feasible.

I note that reducing driver buffering currently reduces wireless-n in
single threaded situations by a lot, however, in more real-world
scenarios it's hardly noticable - and can be fixed one day, after we
get better classification and feedback mechanisms. And by all means,
if you are using a wireless-n device on a mostly-g network, excessive
buffering hurts a lot.

For wired, we are also using ethtool to reduce dma tx values that are
often set optimally for GigE (64-256) to what is optimal for the real
world of far less than 100Mbit, which appears to be in the range 4-16,
where possible. The correct values need to be derived from further
experimentation

Once these changes are made, Qos actually starts to have some effect
on overall network performance again. Without reducing buffer sizes
dramatically, it doesn't.

Apply QoS and packet priorization to system critical 'mice' packets at
the very least. Rate limit but exclude them from being shot down by
other bandwidth control mechanisms. Few people are doing QoS to ipv6
packets at all, and many icmp messages (in addition to babel's udp
multicast) should be prioritized...

I'm pretty sure, based on the results gathered thus far, that this
will improve the quality of most mesh networks out there
and I'd love for more people to be trying these things on a wider
scale and let us know what the effects are.

I've also written elsewhere about the effect of multicast traffic on
wireless and am trying hard to stop bridging gigE (1000Mbit) and
Wireless (a,b,g,n) together wherever possible, as the huge disparity
between the multicast rates is not accounted for in any QoS scheme
available to date in Linux. Addressing large scale usage multicast
effectively is going to take a great deal of work, and even arp can
cause headaches.

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Babel-users] QoS for system critical packets on wireless

[Juliusz Chroboczek]

Dave,

My points below are tainted by the fact that I deeply dislike
classification -- I'm hoping for solutions in which there's no higher
layer knowledge in the routers.

> 1) System control and 'MICE' are < less than 1% of all packets. Mice
> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
> portion of the stack, in what I'm doing currently.

Once you've fought the bloat, there's hopefully no further need to
classify these packets.  On a working network, you should be able to
achieve less than 5% packet loss, even without ECN, and all protocols
should be able to support that level of loss.

> A) wireless devices are currently making heroic efforts (deep
> buffering, exorbitant retries) to get packets through. Seeing a big
> delay between transmit time and reception is more an indicator of
> congestion than actual packet loss is right now. By the time you see
> actual packet loss, the network has often already collapsed
> completely.

Not for multicast -- there's no link-layer ARQ for multicast in 802.11.
That's why RFC 6126 says that you MUST send hello TLVs over multicast
only.

> C) QoS, Packet marking and prioritization of any sort makes babel
> control packets jump closer to the head of the internal queues of the
> transmitting clients, thus speeding up routing change propagation.

Yes.  However, Babel is designed to support loss rates up to 80% or so,
and therefore should normally only collapse when your network has
already collapsed.

(The reason for that?  Julien used to have a couch in his office, where
the pre-ARQ loss rate to the closest Babel router was well above 50%.
We put a lot of effort into ensuring that Julien could read mail on his
couch.)

> I've also written elsewhere about the effect of multicast traffic on
> wireless and am trying hard to stop bridging gigE (1000Mbit) and
> Wireless (a,b,g,n) together wherever possible,

Hear, hear.

(Somebody please bring that up with the OpenWRT folks.)

-- Juliusz

Re: QoS for system critical packets on wireless

[Jim Gettys]

On 06/22/2011 11:17 AM, Dave Taht wrote:
> The biggest fallout of the diffserv work I was trying was observing
> that most packets fell into one of 3 buckets:
>
> 1) System control and 'MICE' are<  less than 1% of all packets. Mice
> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
> portion of the stack, in what I'm doing currently. Mice are
> desperately needed for the network to continue to function.

I'd not use this term: please call it something else.

Van and Kathy have talked about HTTP "mice" and "elephant" TCP flows for 
a long time, and you'll terminally confuse everyone if you call these 
items "mice".

                 - Jim

Re: QoS for system critical packets on wireless

[Dave Taht]

On Wed, Jun 22, 2011 at 8:48 PM, Jim Gettys <jg@freedesktop.org> wrote:
> On 06/22/2011 11:17 AM, Dave Taht wrote:
>>
>> The biggest fallout of the diffserv work I was trying was observing
>> that most packets fell into one of 3 buckets:
>>
>> 1) System control and 'MICE' are<  less than 1% of all packets. Mice
>> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
>> portion of the stack, in what I'm doing currently. Mice are
>> desperately needed for the network to continue to function.
>
> I'd not use this term: please call it something else.
>
> Van and Kathy have talked about HTTP "mice" and "elephant" TCP flows for a
> long time, and you'll terminally confuse everyone if you call these items
> "mice".

I totally agree that overloading 'mice' is the wrong thing to do. mice
is a very useful category defined as 'short tcp streams', and well
discussed that way in the existing TCP literature.

I think I've settled on ANT, but lack a suitable backronym. Ants do
useful stuff, taking care of cleaning up and organizing the universe.
They are orderly, and hard to see,
and can lift enormous things despite their size.

Given my radio background I dislike it because it's short for antenna,
but 'bee is confusing, and hard to spell, bird is the name of a
routing daemon, and I'm plumb out of ideas lacking other analogies...



>                - Jim
>
> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: QoS for system critical packets on wireless

[Jim Gettys]

On 06/22/2011 11:00 PM, Dave Taht wrote:
> On Wed, Jun 22, 2011 at 8:48 PM, Jim Gettys<jg@freedesktop.org>  wrote:
>> On 06/22/2011 11:17 AM, Dave Taht wrote:
>>> The biggest fallout of the diffserv work I was trying was observing
>>> that most packets fell into one of 3 buckets:
>>>
>>> 1) System control and 'MICE' are<    less than 1% of all packets. Mice
>>> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
>>> portion of the stack, in what I'm doing currently. Mice are
>>> desperately needed for the network to continue to function.
>> I'd not use this term: please call it something else.
>>
>> Van and Kathy have talked about HTTP "mice" and "elephant" TCP flows for a
>> long time, and you'll terminally confuse everyone if you call these items
>> "mice".
> I totally agree that overloading 'mice' is the wrong thing to do. mice
> is a very useful category defined as 'short tcp streams', and well
> discussed that way in the existing TCP literature.
>
> I think I've settled on ANT, but lack a suitable backronym. Ants do
> useful stuff, taking care of cleaning up and organizing the universe.
> They are orderly, and hard to see,
> and can lift enormous things despite their size.
> Given my radio background I dislike it because it's short for antenna,
> but 'bee is confusing, and hard to spell, bird is the name of a
> routing daemon, and I'm plumb out of ideas lacking other analogies...
>
>

I like Bees, myself.   They pollinate everything (a good analogy to what 
DNS, DHCP, et. al. do).
             - Jim

>>                 - Jim
>>
>> _______________________________________________
>> Bloat-devel mailing list
>> Bloat-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat-devel
>>
>
>

Re: QoS for system critical packets on wireless

[Dave Taht]

On Thu, Jun 23, 2011 at 5:35 AM, Jim Gettys <jg@freedesktop.org> wrote:
> On 06/22/2011 11:00 PM, Dave Taht wrote:
>>
>> On Wed, Jun 22, 2011 at 8:48 PM, Jim Gettys<jg@freedesktop.org>  wrote:
>>>
>>> On 06/22/2011 11:17 AM, Dave Taht wrote:
>>>>
>>>> The biggest fallout of the diffserv work I was trying was observing
>>>> that most packets fell into one of 3 buckets:
>>>>
>>>> 1) System control and 'MICE' are<    less than 1% of all packets. Mice
>>>> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
>>>> portion of the stack, in what I'm doing currently. Mice are
>>>> desperately needed for the network to continue to function.
>>>
>>> I'd not use this term: please call it something else.
>>>
>>> Van and Kathy have talked about HTTP "mice" and "elephant" TCP flows for
>>> a
>>> long time, and you'll terminally confuse everyone if you call these items
>>> "mice".
>>
>> I totally agree that overloading 'mice' is the wrong thing to do. mice
>> is a very useful category defined as 'short tcp streams', and well
>> discussed that way in the existing TCP literature.
>>
>> I think I've settled on ANT, but lack a suitable backronym. Ants do
>> useful stuff, taking care of cleaning up and organizing the universe.
>> They are orderly, and hard to see,
>> and can lift enormous things despite their size.
>> Given my radio background I dislike it because it's short for antenna,
>> but 'bee is confusing, and hard to spell, bird is the name of a
>> routing daemon, and I'm plumb out of ideas lacking other analogies...
>>
>>
>
> I like Bees, myself.   They pollinate everything (a good analogy to what
> DNS, DHCP, et. al. do).

Too easily confused with BE, is what I meant by 'hard to spell'. I did
come up with a backronym for BEE, tho - 'Best Effort, Extreme'.

But I think I will go with Ants, at least for now.

>            - Jim
>
>>>                - Jim
>>>
>>> _______________________________________________
>>> Bloat-devel mailing list
>>> Bloat-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat-devel
>>>
>>
>>
>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: [Babel-users] QoS for system critical packets on wireless

[Dave Taht]

On Wed, Jun 22, 2011 at 4:12 PM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
> Dave,
>
> My points below are tainted by the fact that I deeply dislike
> classification -- I'm hoping for solutions in which there's no higher
> layer knowledge in the routers.

It was seeing even the minm

>
>> 1) 'ANTS' are < less than 1% of all packets. ANTS includes system control and
>> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
>> portion of the stack, in what I'm doing currently.
>
> Once you've fought the bloat, there's hopefully no further need to
> classify these packets.  On a working network, you should be able to
> achieve less than 5% packet loss, even without ECN, and all protocols
> should be able to support that level of loss.

I have always needed some minimal level of QoS on all networks I
shared with other people, even prior to bufferbloat.

As for classification, with asymmetric networks, the canonical example
of some level needed is moving interactive packets up in priority over
uploads.

Once you admit there is some need for classication, you rapidly head
down an Aristotelian rathole. Believe me, I'm feeling that - a couple
hundred protocols, two mutually sort-of-similar classification schemes
in linux (tc, iptables)... the chain of functions a linux box has to
call to get a packet from point a to point b is 3 pages long,
even before you start trying to classify things.

Minimal, ad-hoc classification methods as they exist today are
insufficient to handle enough 'normal' cases to accomplish their
intent. Wedging everything into Ants, Mice, and Elephants seems a more
reasonable approach by the day, although I admit some fondness for
diffserv...

An example, also taken from the above - no common shaper script out
there accounts properly for ipv6 packets, and you can clobber upload
bandwidth easily without prioritization for interactive ipv6 packets,
too.

I struggle with the need for any level of classification, but I
reluctantly have given up, and begun to assume that it needs to be
done way better and more consistently in the general case.

As it is, here's a tidbit from last night... after poking into how
802.11e is supposed to work (mac80211 basically treats values in the
skb->priority field in the range 256 0..7 as a map from 802.1d) I went
looking into how 802.1p (vlan prioritization, based on d) worked in
Linux...

from linux-2.6/net/8021q/vlan.h

/**
 *      struct vlan_priority_tci_mapping - vlan egress priority mappings
 *      @priority: skb priority
 *      @vlan_qos: vlan priority: (skb->priority << 13) & 0xE000
 *      @next: pointer to next struct
 */


At which point I decided to go drinking.

(guess what else skb-priority is used for?)

I can't believe anybody is actually using this stuff, the details are
too deeply buried.

>> A) wireless devices are currently making heroic efforts (deep
>> buffering, exorbitant retries) to get packets through. Seeing a big
>> delay between transmit time and reception is more an indicator of
>> congestion than actual packet loss is right now. By the time you see
>> actual packet loss, the network has often already collapsed
>> completely.
>
> Not for multicast -- there's no link-layer ARQ for multicast in 802.11.
> That's why RFC 6126 says that you MUST send hello TLVs over multicast
> only.

The problem is that all the packets in the queues ahead of babel can
be delayed with deep buffering and exorbitant retries.

(I am intensely curious as to the effect of 802.11e on n - if it was
modified to support transmitting in the short windows assigned to VI
and VO....)

>
>> C) QoS, Packet marking and prioritization of any sort makes babel
>> control packets jump closer to the head of the internal queues of the
>> transmitting clients, thus speeding up routing change propagation.
>
> Yes.  However, Babel is designed to support loss rates up to 80% or so,
> and therefore should normally only collapse when your network has
> already collapsed.
>
> (The reason for that?  Julien used to have a couch in his office, where
> the pre-ARQ loss rate to the closest Babel router was well above 50%.
> We put a lot of effort into ensuring that Julien could read mail on his
> couch.)

Heh. Pre-n, I imagine.

>
>> I've also written elsewhere about the effect of multicast traffic on
>> wireless and am trying hard to stop bridging gigE (1000Mbit) and
>> Wireless (a,b,g,n) together wherever possible,
>
> Hear, hear.
>
> (Somebody please bring that up with the OpenWRT folks.)

I have. But I don't want to talk about this now, because then I'd want
to go drinking again. I do think in the long run that better tools and
unifying concepts need to show up that would make routing and
firewalling easier and more effective and performant than bridging and
ebtables...

but if I could toss the bridge code out of the cerowort effort right
now, I would. I  have got 3x the performance out of the wired side of
it by doing pure routing, without DoSing the wireless side, and I get
working multicast throughout, too...

All I need is a way of generating a unique local mac for the second
ethernet interface on first boot. I'm tempted to buy my own mac
address range - I'm that frustrated.

>
> -- Juliusz
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: QoS for system critical packets on wireless

[Rick Jones]

On 06/22/2011 08:00 PM, Dave Taht wrote:
> I think I've settled on ANT, but lack a suitable backronym. Ants do
> useful stuff, taking care of cleaning up and organizing the universe.
> They are orderly, and hard to see,
> and can lift enormous things despite their size.

ANTs
Actually Need Transmitting
Ants Need Transmitting

BEETLEs

Background End-to-End Transmission Liking Entities

GNATs
Globally Necessary And Transmitted
Generally Necessary And Transmitted

rick jones

Re: QoS for system critical packets on wireless

[Justin McCann]

On Thu, Jun 23, 2011 at 1:23 PM, Rick Jones <rick.jones2@hp.com> wrote:
> On 06/22/2011 08:00 PM, Dave Taht wrote:
>>
>> I think I've settled on ANT, but lack a suitable backronym. Ants do
>> useful stuff, taking care of cleaning up and organizing the universe.
>> They are orderly, and hard to see,
>> and can lift enormous things despite their size.
>
> ANTs
> Actually Need Transmitting
> Ants Need Transmitting

+1 on ANTS. The other two aren't really acronyms, but since it's fun:

Administrative Network Traffic [& Services]
Anything Needed (for) Transit Service
Anything Necessary To Start

Re: [Babel-users] QoS for system critical packets on wireless

[Juliusz Chroboczek]

>> Once you've fought the bloat, there's hopefully no further need to
>> classify these packets.

> As for classification, with asymmetric networks, the canonical example
> of some level needed is moving interactive packets up in priority over
> uploads.

Unless your network is badly underprovisioned, some form of roughly
fair share scheduling (say, SFQ) should be all you need.

> Once you admit there is some need for classication,

I certainly don't.  We should be simplifying our networks, not making
them even more complex.  (Dave, I'm seriously worried you're heading in
the wrong direction.)

> I admit some fondness for diffserv...

The first step is to admit you have a problem ;-)

-- Juliusz

Re: [Babel-users] QoS for system critical packets on wireless

[Dave Taht]

On Fri, Jun 24, 2011 at 6:52 AM, Juliusz Chroboczek <jch@pps.jussieu.fr> wrote:
>>> Once you've fought the bloat, there's hopefully no further need to
>>> classify these packets.
>
>> As for classification, with asymmetric networks, the canonical example
>> of some level needed is moving interactive packets up in priority over
>> uploads.
>
> Unless your network is badly underprovisioned, some form of roughly
> fair share scheduling (say, SFQ) should be all you need.

That caveat is crucial. On the dark areas of these maps:

http://www.google.com/search?q=satellite+night&hl=en&client=ubuntu&hs=eu7&channel=fs&prmd=ivnsm&tbm=isch&tbo=u&source=univ&sa=X&ei=S5kETvjwBeWt0AGWip3oCw&ved=0CC8QsAQ&biw=955&bih=427

most of the dark areas will be underprovisioned for many years to come.

Every dense wireless network is underprovisioned. 3G, 4G, wifi, etc.

Lastly, bandwidth requests always expand to be in excess of what is
available. Once, we did youtube at 320x200. Then bittorrent... Now
it's netflix, and HQ video, and facetime, etc.

I have some hope that we've reached the end of the S-curve on normal
usages of bandwidth and that the technology and infrastructure can now
have some time to catch up. Classically under/over provisioning has
been cyclical.

>
>> Once you admit there is some need for classication,
>
> I certainly don't.  We should be simplifying our networks, not making
> them even more complex.  (Dave, I'm seriously worried you're heading in
> the wrong direction.)

Classifying traffic and looking at Ants has proved useful and in some
ways is a fallout from the background bufferbloat detector idea for
ntp of a few months back.

For example, I would never have spotted the syn flood limit a few days
back had I not been looking at mice and noticing a huge difference
between syn attempts and actual connections. Many other examples of
productive analysis exist of various forms of tcp streams vs the
bufferbloat background radiation, and doubtless, many other 'hacks'
exist that are harming the network more than help it.

At the moment I can filter against ant-like traffic and quickly
produce a scatter plot that at a glance shows the overall health of
the network.

As for the complexity, the network is already pretty complex as noted
earlier, and I would hope that being aware of the ants would lead
towards better diagnoses of the root causes of problems.

Raising the Ants to the same level of awareness academic analysis as
TCP mice and elephants have had would be a very good thing.

As for actually doing shaping, I have great hopes that once
bufferbloat is licked, and the fallout, and classic means of dealing
with great bandwidth disparities are more widely implemented, that
shaping techniques such as SFB and SFQ will work well again.

But that's all it is. Hope. I figure, at best, I'll have a network as
responsive and reliable as the networks I had in the mid 2000s.

>> I admit some fondness for diffserv...
>
> The first step is to admit you have a problem ;-)

Heh. Back in the early 80s I was one of the radical packet-heads, in
an all out war with the bell-heads. I've come to thinking that
although we won that battle, we lost the war. Nearly every voip
network uses separate provisioning, and I think that voice quality has
degraded significantly over the last decade.

I had a long cherished dream of one day being able to have a
'jam-o-phone' - a device that would let me play live music with the
housebound old buddy down the street, over the net, but even as
bandwidths have grown hugely, our ability to keep latencies and jitter
down to low levels has also degraded hugely.

>
> -- Juliusz
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: QoS for system critical packets on wireless

[Dave Taht]

On Thu, Jun 23, 2011 at 12:38 PM, Justin McCann <jneilm@gmail.com> wrote:
> On Thu, Jun 23, 2011 at 1:23 PM, Rick Jones <rick.jones2@hp.com> wrote:
>> On 06/22/2011 08:00 PM, Dave Taht wrote:
>>>
>>> I think I've settled on ANT, but lack a suitable backronym. Ants do
>>> useful stuff, taking care of cleaning up and organizing the universe.
>>> They are orderly, and hard to see,
>>> and can lift enormous things despite their size.
>>
>> ANTs
>> Actually Need Transmitting
>> Ants Need Transmitting
>
> +1 on ANTS. The other two aren't really acronyms, but since it's fun:
>
> Administrative Network Traffic [& Services]
> Anything Needed (for) Transit Service
> Anything Necessary To Start

Almost Needed Transfers

OK, I'm happy. We have enough backronyms for this that are meaningful
now, and ANTs can take the rightful place among the mice and elephants
for future discussion...


> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com