Mice [was Re: QoS for system critical packets on wireless]

Mice [was Re: QoS for system critical packets on wireless]

[Justin McCann]

On Wed, Jun 22, 2011 at 11:17 AM, Dave Taht <dave.taht@gmail.com> wrote:
> The biggest fallout of the diffserv work I was trying was observing
> that most packets fell into one of 3 buckets:
>
> 1) System control and 'MICE' are < less than 1% of all packets. Mice
> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
> portion of the stack, in what I'm doing currently. Mice are
> desperately needed for the network to continue to function.
>
> 2) unresponsive streams and udp (varying, but generally pretty low)
> 3) responsive tcp streams (closer to 99%)

Dave,

I want to avoid hijacking your thread, because I think the QoS issues
you're raising are important, but I think you've misused the term
'mice'. Mice are short-lived TCP flows (see
http://www.google.com/search?q=mice+elephants+tcp), not these
control/management-related low-bandwidth protocols. Obviously the
control/management protocols need priority on the network. "Mice"
really belongs in category 3), which often gets broken down into 3a)
mice and 3b) elephants.

To resurrect an ancient thread, what Kathie Nichols was referring to
is a more sinister problem: short-lived TCP flows make up a lot of the
*number of flows* in the Internet--- think of all the little JS, CSS
and image pulls a web page causes--- but not the majority of bytes.
You can make long-lived high-bandwidth flows behave without starvation
using ECN and the like, and a lot (most?) of AQM research has focused
mainly on that.

The problem is, how do you handle tons of flows which each transmit
just a few packets? Head-dropping one or more packets in a
high-bandwidth flow using SACK won't cause that flow much of a
problem. However, if your queue is full of packets from many different
flows that each consist of just a few packets, dropping *even one*
might cause that flow to hit a retransmission timeout. The overall
connection-completion latency (SYN to FIN) goes up significantly,
which is especially bad if the connections are done serially.

When you have a relatively small number of high-bandwidth flows,
dropping packets can quickly change the fill rate at the bottleneck
queue. When you have a lot of small flows, dropping packets from only
a few flows doesn't make much difference--- each flow doesn't
contribute as much to the overall sustained rate, so it can't reduce
it by much either.

I'm not really read up on all the various queueing disciplines. Maybe
when you have a large number of flows in your queue, you need to drop
more packets (from many of them) to get a corresponding drop in the
fill rate. Paying attention to IP-to-IP flows vs. TCP/UDP flows may
also make up the difference when you've got a lot of separate TCP
connections.

I think most consumer-ISPs' approach to elephant flows is to simply
rate-limit them after a period of time. So much for that rate you
thought you were paying for.

      Justin

Re: Mice [was Re: QoS for system critical packets on wireless]

[Dave Taht]

On Wed, Jun 22, 2011 at 1:39 PM, Justin McCann <jneilm@gmail.com> wrote:
> On Wed, Jun 22, 2011 at 11:17 AM, Dave Taht <dave.taht@gmail.com> wrote:
>> The biggest fallout of the diffserv work I was trying was observing
>> that most packets fell into one of 3 buckets:
>>
>> 1) System control and 'some tiny animal' are < less than 1% of all packets. 'some tiny animal'
>> includes a bunch of messages like ARP, NTP, UDP, and most of the icmp6
>> portion of the stack, in what I'm doing currently. 'some tiny animal' are
>> desperately needed for the network to continue to function.
>>
>> 2) unresponsive streams and udp (varying, but generally pretty low)
>> 3) responsive tcp streams (closer to 99%)
>
> Dave,
>
> I want to avoid hijacking your thread, because I think the QoS issues
> you're raising are important, but I think you've misused the term
> 'mice'. Mice are short-lived TCP flows (see
> http://www.google.com/search?q=mice+elephants+tcp), not these
> control/management-related low-bandwidth protocols. Obviously the
> control/management protocols need priority on the network. "Mice"
> really belongs in category 3), which often gets broken down into 3a)
> mice and 3b) elephants.

EXTREMELY VALID POINT.

I utterly agree that I have been mis-using the term 'mice' in this
context. I will stop
immediately to end the confusion. won't even qualify by saying TCP
Mice, gotcha. I have to stop talking to myself more often....

I kind of started extending the term because I'd also lumpled syns,
synacks, and fins into the tiny packets that are "often useless and
even detrimental to shoot" catagory.

Synack optimization is actually a "feature" of many cable modems...
and fins have been a problem going back YEARS where many providers
simply drop the connection instead of waiting for a correct close.

I further make a distinction between control packets and things like
DNS/NTP and VOIP as per the above....

While I'm liking using as much Diffserv terminology as possible (CS6
works fine for 'control' packets (some of icmp6 fits into that
catagory too), EF works for voip, I really need a word for DNS/NTP and
maybe ARP and DHCP that fits the facts somewhat better. They really
aren't BE services anymore...

sooo....

*****

We need another animal or two, smaller than a mouse, yet required for
life, to use as a analogy... e coli? (nah....) Need something with a
good back-acronym... lots of animals to choose from....

????

Ideas?

****

The problem with lumping arp into CS6 is that it can and should be
rate limited elsewhere. Same for DHCP. Similarly CS6 includes BGP
which can lead to enormous transfers....

NTP could go into CS6, except that it's more EF

other sorts of UDP are generally small flows, with vpns being that
particular udp elephant.

So I just want a single word for these sorts of packets that can
compete with the TCP mice and elephant discussion... because in
forgetting about them, we've entered a world of hurt.

>
> To resurrect an ancient thread, what Kathie Nichols was referring to
> is a more sinister problem: short-lived TCP flows make up a lot of the
> *number of flows* in the Internet--- think of all the little JS, CSS
> and image pulls a web page causes--- but not the majority of bytes.
> You can make long-lived high-bandwidth flows behave without starvation
> using ECN and the like, and a lot (most?) of AQM research has focused
> mainly on that.

Yes I'd understood that, and unfortunately extended it past it's
original meaning
in several weeks of talking to myself...

> The problem is, how do you handle tons of flows which each transmit
> just a few packets? Head-dropping one or more packets in a
> high-bandwidth flow using SACK won't cause that flow much of a
> problem. However, if your queue is full of packets from many different
> flows that each consist of just a few packets, dropping *even one*
> might cause that flow to hit a retransmission timeout. The overall
> connection-completion latency (SYN to FIN) goes up significantly,
> which is especially bad if the connections are done serially.
>
> When you have a relatively small number of high-bandwidth flows,
> dropping packets can quickly change the fill rate at the bottleneck
> queue. When you have a lot of small flows, dropping packets from only
> a few flows doesn't make much difference--- each flow doesn't
> contribute as much to the overall sustained rate, so it can't reduce
> it by much either.
>
> I'm not really read up on all the various queueing disciplines. Maybe
> when you have a large number of flows in your queue, you need to drop
> more packets (from many of them) to get a corresponding drop in the
> fill rate. Paying attention to IP-to-IP flows vs. TCP/UDP flows may
> also make up the difference when you've got a lot of separate TCP
> connections.
>
> I think most consumer-ISPs' approach to elephant flows is to simply
> rate-limit them after a period of time. So much for that rate you
> thought you were paying for.
>
>      Justin
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: Mice [was Re: QoS for system critical packets on wireless]

[Dave Taht]

OK, so I've gone and updated the related wiki page to indicate more
of what I'd actually meant. Still need a good analogy though...

Secondly you've given me an idea towards actively tracking mice-y
short flows using something like conntrack or simply getting stats on
syns, synacks,fins, finacks which ought to be interesting in itself.
One problem I'd already found in the openwrt defaults was that it does
severe rate limiting of syns by default - 25!!!??? which I only found
by trying to access the top 100 web sites all at the same time and
being totally befuddled by the results. (this test is also in my
Diffserv repo)

Thirdly, although the iptables implementation of diffserv was
straightforward and
easy to describe, and the proposal for enhancing iptables to be
performant equally clear and straightforward, it's beginning to appear
that most of the functionality I proposed in the first draft of the
diffserv rfc belongs in the 'tc' subsystem, not in iptables.

I'm really glad I wrote it down and am grateful for the feedback both
public and private thus far.


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: Mice [was Re: QoS for system critical packets on wireless]

[Jonathan Morton]

On 22 Jun, 2011, at 11:11 pm, Dave Taht wrote:

> I further make a distinction between control packets and things like
> DNS/NTP and VOIP as per the above....

Tiny Information Transfers?

http://www.visordown.com/members/images/179524/gallery/ist2_2553987-great-tits.jpg

:-D

 - Jonathan

Re: Mice [was Re: QoS for system critical packets on wireless]

[Dave Taht]

On Wed, Jun 22, 2011 at 2:59 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> On 22 Jun, 2011, at 11:11 pm, Dave Taht wrote:
>
>> I further make a distinction between control packets and things like
>> DNS/NTP and VOIP as per the above....
>
> Tiny Information Transfers?
>
> http://www.visordown.com/members/images/179524/gallery/ist2_2553987-great-tits.jpg

NICE ***s. I would so love to be able to talk about that that way, but
in today's politically correct environment it seems impossible, as per
the weirdly charged discussion of Mark Pesce's "smoke signals" [1] -
which while containing far more controversial content - touched upon
this subject and caused a firestorm of comment.

(mildly) safer names, please.

> :-D
>
>  - Jonathan
>
>


1: http://linuxconfau.blip.tv/file/4851926

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

Re: Mice [was Re: QoS for system critical packets on wireless]

[Dave Taht]

Mice, Elephants, and Fleas?

Mice, Elephants, and Bees?

Need a < than five letter animal, smaller than a mouse, useful for stuff...

Mice, Elephants, and Ants?

Birds?

Of these I like ants and bees the best... but I'm low on backronyms here...

On Wed, Jun 22, 2011 at 3:06 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, Jun 22, 2011 at 2:59 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> On 22 Jun, 2011, at 11:11 pm, Dave Taht wrote:
>>
>>> I further make a distinction between control packets and things like
>>> DNS/NTP and VOIP as per the above....
>>
>> Tiny Information Transfers?
>>
>> http://www.visordown.com/members/images/179524/gallery/ist2_2553987-great-tits.jpg
>
> NICE ***s. I would so love to be able to talk about that that way, but
> in today's politically correct environment it seems impossible, as per
> the weirdly charged discussion of Mark Pesce's "smoke signals" [1] -
> which while containing far more controversial content - touched upon
> this subject and caused a firestorm of comment.
>
> (mildly) safer names, please.
>
>> :-D
>>
>>  - Jonathan
>>
>>
>
>
> 1: http://linuxconfau.blip.tv/file/4851926
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://the-edge.blogspot.com
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com