[Cerowrt-devel] Coping with wireless-n [#305]

[Cerowrt-devel] Coping with wireless-n [#305]

[Dave Taht]

This is the 6th in a series of mails doing a post-mortem and rethink
of what is in cerowrt.

Basically I think the 2.4ghz spectrum is beyond hope. Everywhere I've
been there are dozens of radios all on all the channels available, all
competing, all interfering, and the difference in performance results
I get minute to minute, hour to hour varies so wildly that in order to
get a good picture of how basic 2.4ghz performance worked I had to go
to remote valley in france where there was no competing signals to
deal with.

There I got pretty equivalent results between 2.4ghz and 5. The rest
of the time, not so much.

As for wireless-g vs n, the current architecture of the mac80211 stack
buries packet aggregation so deeply within it that the two
technologies are very incompatible. It also bothers me that management
frames are buried so deeply, too.

The only places I've been this year that had a functional wireless
network, rigorously divided up the SSIDs into g, n, ipv4 and ipv6
specific parts, and were using Cisco hardware.

I don't see the need to have separate SSIDs for 4 and 6, but I think
g+QFQ, and n+SFQ might do better in the general case on an AP.

QFQ would work well on client machines, particularly with n, to break
up packet bursts more sanely into multiple streams. SFQ, less so. I'd
rather pursue QFQ on the clients as it better 'shreds' competing
bursts. I'd like others to play with it too.

and in all cases getting to where something BQL-like + Time in Queue
on top of those two techniques will help most of all,
but better feedback loops and means of determining bandwidth
differences between destinations are kind of required.

And doing MUCH better classification into the hardware queues would be
good, too.

So in the next release of cerowrt I'm planning on having a separate
n-only or g-only SSID available on the 5ghz channel, and
to experiment with the above qdiscs. I sort of have some major ANT and
DSCP specific classifier code that I plan to polish up.

as for BQL and Time in Queue, that work is only just beginning and I
hope to track it closely.  As to how to make it apply to wireless,
I look forward to the engineering debate(s). It's a ton of work.

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[david]

On Thu, 8 Dec 2011, Dave Taht wrote:

> This is the 6th in a series of mails doing a post-mortem and rethink
> of what is in cerowrt.
>
> Basically I think the 2.4ghz spectrum is beyond hope. Everywhere I've
> been there are dozens of radios all on all the channels available, all
> competing, all interfering, and the difference in performance results
> I get minute to minute, hour to hour varies so wildly that in order to
> get a good picture of how basic 2.4ghz performance worked I had to go
> to remote valley in france where there was no competing signals to
> deal with.
>
> There I got pretty equivalent results between 2.4ghz and 5. The rest
> of the time, not so much.
>
> As for wireless-g vs n, the current architecture of the mac80211 stack
> buries packet aggregation so deeply within it that the two
> technologies are very incompatible. It also bothers me that management
> frames are buried so deeply, too.
>
> The only places I've been this year that had a functional wireless
> network, rigorously divided up the SSIDs into g, n, ipv4 and ipv6
> specific parts, and were using Cisco hardware.
>
> I don't see the need to have separate SSIDs for 4 and 6, but I think
> g+QFQ, and n+SFQ might do better in the general case on an AP.
>
> QFQ would work well on client machines, particularly with n, to break
> up packet bursts more sanely into multiple streams. SFQ, less so. I'd
> rather pursue QFQ on the clients as it better 'shreds' competing
> bursts. I'd like others to play with it too.
>
> and in all cases getting to where something BQL-like + Time in Queue
> on top of those two techniques will help most of all,
> but better feedback loops and means of determining bandwidth
> differences between destinations are kind of required.
>
> And doing MUCH better classification into the hardware queues would be
> good, too.
>
> So in the next release of cerowrt I'm planning on having a separate
> n-only or g-only SSID available on the 5ghz channel, and
> to experiment with the above qdiscs. I sort of have some major ANT and
> DSCP specific classifier code that I plan to polish up.
>
> as for BQL and Time in Queue, that work is only just beginning and I
> hope to track it closely.  As to how to make it apply to wireless,
> I look forward to the engineering debate(s). It's a ton of work.

this puzzles me.

splitting 2.4G and 5G into different different networks (broadcast 
domains) is a huge win. cince I can't find any open implementation fo band 
steering, this requires putting the two bands on different SSIDs.

but I don't understand why there is a big problem with G and N sharing the 
same SSID.

there is some grief with having different speeds on the same channel, but 
only in that the same amount of data will take longer to transmit (causing 
problems with predicting how long the queue is in terms of time as it will 
vary on the destintation), but even if you stick with G for example, it 
can transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some 
higher speeds to this. If the devices are configured sanely, they should 
be transmitting the header for a G frame to reserve the air time and then 
sending the N frame inside of that. this has a slight overhead compared to 
a pure N network, but it doesn't matter if the G network is on the same 
SSID or on a different one, the problem is sharing the airtime on the 
channel.

David Lang

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[Dave Taht]

On Thu, Dec 8, 2011 at 12:51 PM,  <david@lang.hm> wrote:
> On Thu, 8 Dec 2011, Dave Taht wrote:

> this puzzles me.
>
> splitting 2.4G and 5G into different different networks (broadcast domains)
> is a huge win. cince I can't find any open implementation fo band steering,
> this requires putting the two bands on different SSIDs.

Oh, god no, I'm not dropping that. Having those split AND off the wired
network is staying in...

> but I don't understand why there is a big problem with G and N sharing the
> same SSID.

Because you can fully FQ G, and if you do that to N, it messes up aggregation.

>
> there is some

Some?

> grief with having different speeds on the same channel, but
> only in that the same amount of data will take longer to transmit (causing
> problems with predicting how long the queue is in terms of time as it will
> vary on the destintation), but even if you stick with G for example, it can
> transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some
> higher speeds to this. If the devices are configured sanely, they should be
> transmitting the header for a G frame to reserve the air time and then
> sending the N frame inside of that. this has a slight overhead compared to a
> pure N network, but it doesn't matter if the G network is on the same SSID
> or on a different one, the problem is sharing the airtime on the channel.

It's a packet scheduler test more than anything else.

> David Lang



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[david]

On Thu, 8 Dec 2011, Dave Taht wrote:

> On Thu, Dec 8, 2011 at 12:51 PM,  <david@lang.hm> wrote:
>> On Thu, 8 Dec 2011, Dave Taht wrote:
>
>> this puzzles me.
>>
>> splitting 2.4G and 5G into different different networks (broadcast domains)
>> is a huge win. cince I can't find any open implementation fo band steering,
>> this requires putting the two bands on different SSIDs.
>
> Oh, god no, I'm not dropping that. Having those split AND off the wired
> network is staying in...
>
>> but I don't understand why there is a big problem with G and N sharing the
>> same SSID.
>
> Because you can fully FQ G, and if you do that to N, it messes up aggregation.

I don't recognize the term "FQ".

when you say it messes up aggregation, do you mean combining two channels 
togeather for higher throughput or something else?

I was assuming that if you are running a mized network you only use a 
single channel for N. If you are using multiple channels for N they should 
be a separate SSID, just from the fact that you are using two channels for 
N but only one for G (which one would be the question)

>>
>> there is some
>
> Some?

some, but it's an unavoidable feature of wireless communication. You can 
consider turning off some modulation types, but since the clients 
automatically fall back to slower modulation types when there is a 
problem, the result will be failed connections.

now, this may still be the right thing to do, because the failback to a 
slower modulation type works well for weak signal situations, but in a 
high density situation (which is basically every 2.4G deployment in the 
real world nowdays), taking longer to send the same data just means that 
you are more vunerable to another transmitter clobbering you, so it 
actually decreases reliability.

David Lang

>> grief with having different speeds on the same channel, but
>> only in that the same amount of data will take longer to transmit (causing
>> problems with predicting how long the queue is in terms of time as it will
>> vary on the destintation), but even if you stick with G for example, it can
>> transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some
>> higher speeds to this. If the devices are configured sanely, they should be
>> transmitting the header for a G frame to reserve the air time and then
>> sending the N frame inside of that. this has a slight overhead compared to a
>> pure N network, but it doesn't matter if the G network is on the same SSID
>> or on a different one, the problem is sharing the airtime on the channel.
>
> It's a packet scheduler test more than anything else.
>
>> David Lang
>
>
>
>

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[Dave Taht]

On Thu, Dec 8, 2011 at 1:25 PM,  <david@lang.hm> wrote:
> On Thu, 8 Dec 2011, Dave Taht wrote:
>
>> On Thu, Dec 8, 2011 at 12:51 PM,  <david@lang.hm> wrote:
>>>
>>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>
>>
>>> this puzzles me.
>>>
>>> splitting 2.4G and 5G into different different networks (broadcast
>>> domains)
>>> is a huge win. cince I can't find any open implementation fo band
>>> steering,
>>> this requires putting the two bands on different SSIDs.
>>
>>
>> Oh, god no, I'm not dropping that. Having those split AND off the wired
>> network is staying in...
>>
>>> but I don't understand why there is a big problem with G and N sharing
>>> the
>>> same SSID.
>>
>>
>> Because you can fully FQ G, and if you do that to N, it messes up
>> aggregation.
>
>
> I don't recognize the term "FQ".

Fair Queue

http://info.iet.unipi.it/~luigi/qfq/ in this case.

>
> when you say it messes up aggregation, do you mean combining two channels
> togeather for higher throughput or something else?

No, the way the driver is structured it swallows as many packets as are
aggregatable for a given destination, then ships them. If you instead
try to do the right thing - which is to break up packet bursts into
as tiny pieces as possible, aggregation goes to heck. What you want
to do is aggregate fair queued packets for a given destination, at a
size that will fit (up to 64 packets or 64kbytes) at the rate the wireless
interface is running at.

As a result nobody does FQ, nor AQM, on wireless n, where it is
so desparately needed.

>
> I was assuming that if you are running a mized network you only use a single
> channel for N. If you are using multiple channels for N they should be a
> separate SSID, just from the fact that you are using two channels for N but
> only one for G (which one would be the question)

One channel for both N and G in this case. Only one radio for 5ghz.
>
>>>
>>> there is some
>>
>>
>> Some?
>
>
> some, but it's an unavoidable feature of wireless communication. You can
> consider turning off some modulation types, but since the clients
> automatically fall back to slower modulation types when there is a problem,
> the result will be failed connections.

To give you an idea, at 5ghz I'm capable with cerowrt at achieving 150Mbits
with TCP - in the clean air here.

At 2.4, it's rare I can get more than 20, and fairly often much less than that.

Any given test I run regarding wireless simply is not repeatable if I do it
on 2.4ghz.

I can usually 'hear' more than 30 access points at my apt, as another
example.


>
> now, this may still be the right thing to do, because the failback to a
> slower modulation type works well for weak signal situations, but in a high
> density situation (which is basically every 2.4G deployment in the real
> world nowdays), taking longer to send the same data just means that you are
> more vunerable to another transmitter clobbering you, so it actually
> decreases reliability.

yes, minstrel rocks.
>
> David Lang
>
>
>>> grief with having different speeds on the same channel, but
>>> only in that the same amount of data will take longer to transmit
>>> (causing
>>> problems with predicting how long the queue is in terms of time as it
>>> will
>>> vary on the destintation), but even if you stick with G for example, it
>>> can
>>> transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some
>>> higher speeds to this. If the devices are configured sanely, they should
>>> be
>>> transmitting the header for a G frame to reserve the air time and then
>>> sending the N frame inside of that. this has a slight overhead compared
>>> to a
>>> pure N network, but it doesn't matter if the G network is on the same
>>> SSID
>>> or on a different one, the problem is sharing the airtime on the channel.
>>
>>
>> It's a packet scheduler test more than anything else.
>>
>>> David Lang
>>
>>
>>
>>
>>
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[Dave Taht]

note, that I do try on occasion to capture stuff into the bug tracker
When you see something like [#305] in the subject
or cerowrt ccd, it goes there...

That said, I have to not surprise people with that 'feature'.


On Thu, Dec 8, 2011 at 1:46 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Dec 8, 2011 at 1:25 PM,  <david@lang.hm> wrote:
>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>
>>> On Thu, Dec 8, 2011 at 12:51 PM,  <david@lang.hm> wrote:
>>>>
>>>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>>
>>>
>>>> this puzzles me.
>>>>
>>>> splitting 2.4G and 5G into different different networks (broadcast
>>>> domains)
>>>> is a huge win. cince I can't find any open implementation fo band
>>>> steering,
>>>> this requires putting the two bands on different SSIDs.
>>>
>>>
>>> Oh, god no, I'm not dropping that. Having those split AND off the wired
>>> network is staying in...
>>>
>>>> but I don't understand why there is a big problem with G and N sharing
>>>> the
>>>> same SSID.
>>>
>>>
>>> Because you can fully FQ G, and if you do that to N, it messes up
>>> aggregation.
>>
>>
>> I don't recognize the term "FQ".
>
> Fair Queue
>
> http://info.iet.unipi.it/~luigi/qfq/ in this case.
>
>>
>> when you say it messes up aggregation, do you mean combining two channels
>> togeather for higher throughput or something else?
>
> No, the way the driver is structured it swallows as many packets as are
> aggregatable for a given destination, then ships them. If you instead
> try to do the right thing - which is to break up packet bursts into
> as tiny pieces as possible, aggregation goes to heck. What you want
> to do is aggregate fair queued packets for a given destination, at a
> size that will fit (up to 64 packets or 64kbytes) at the rate the wireless
> interface is running at.
>
> As a result nobody does FQ, nor AQM, on wireless n, where it is
> so desparately needed.
>
>>
>> I was assuming that if you are running a mized network you only use a single
>> channel for N. If you are using multiple channels for N they should be a
>> separate SSID, just from the fact that you are using two channels for N but
>> only one for G (which one would be the question)
>
> One channel for both N and G in this case. Only one radio for 5ghz.
>>
>>>>
>>>> there is some
>>>
>>>
>>> Some?
>>
>>
>> some, but it's an unavoidable feature of wireless communication. You can
>> consider turning off some modulation types, but since the clients
>> automatically fall back to slower modulation types when there is a problem,
>> the result will be failed connections.
>
> To give you an idea, at 5ghz I'm capable with cerowrt at achieving 150Mbits
> with TCP - in the clean air here.
>
> At 2.4, it's rare I can get more than 20, and fairly often much less than that.
>
> Any given test I run regarding wireless simply is not repeatable if I do it
> on 2.4ghz.
>
> I can usually 'hear' more than 30 access points at my apt, as another
> example.
>
>
>>
>> now, this may still be the right thing to do, because the failback to a
>> slower modulation type works well for weak signal situations, but in a high
>> density situation (which is basically every 2.4G deployment in the real
>> world nowdays), taking longer to send the same data just means that you are
>> more vunerable to another transmitter clobbering you, so it actually
>> decreases reliability.
>
> yes, minstrel rocks.
>>
>> David Lang
>>
>>
>>>> grief with having different speeds on the same channel, but
>>>> only in that the same amount of data will take longer to transmit
>>>> (causing
>>>> problems with predicting how long the queue is in terms of time as it
>>>> will
>>>> vary on the destintation), but even if you stick with G for example, it
>>>> can
>>>> transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some
>>>> higher speeds to this. If the devices are configured sanely, they should
>>>> be
>>>> transmitting the header for a G frame to reserve the air time and then
>>>> sending the N frame inside of that. this has a slight overhead compared
>>>> to a
>>>> pure N network, but it doesn't matter if the G network is on the same
>>>> SSID
>>>> or on a different one, the problem is sharing the airtime on the channel.
>>>
>>>
>>> It's a packet scheduler test more than anything else.
>>>
>>>> David Lang
>>>
>>>
>>>
>>>
>>>
>>
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> FR Tel: 0638645374
> http://www.bufferbloat.net



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Cerowrt-devel] Coping with wireless-n [#305]

[david]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6211 bytes --]

On Thu, 8 Dec 2011, Dave Taht wrote:

> On Thu, Dec 8, 2011 at 1:25 PM,  <david@lang.hm> wrote:
>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>
>>> On Thu, Dec 8, 2011 at 12:51 PM,  <david@lang.hm> wrote:
>>>>
>>>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>>
>>>
>>>> but I don't understand why there is a big problem with G and N sharing
>>>> the
>>>> same SSID.
>>>
>>>
>>> Because you can fully FQ G, and if you do that to N, it messes up
>>> aggregation.
>>
>>
>> I don't recognize the term "FQ".
>
> Fair Queue
>
> http://info.iet.unipi.it/~luigi/qfq/ in this case.

thanks, I''ll read up on that

>>
>> when you say it messes up aggregation, do you mean combining two channels
>> togeather for higher throughput or something else?
>
> No, the way the driver is structured it swallows as many packets as are
> aggregatable for a given destination, then ships them. If you instead
> try to do the right thing - which is to break up packet bursts into
> as tiny pieces as possible, aggregation goes to heck. What you want
> to do is aggregate fair queued packets for a given destination, at a
> size that will fit (up to 64 packets or 64kbytes) at the rate the wireless
> interface is running at.
>
> As a result nobody does FQ, nor AQM, on wireless n, where it is
> so desparately needed.

this doesn't sound like a unique problem for N. there are other networks 
where the per-transmission overhead is significant and there are benefits 
in combining multiple packets in one transmission.

even on ethernet you have a fairly significant per-packet overhead and I 
remember reading of some cases where people have implemented things that 
combine multiple packets into a larger packet to send at once just to 
avoid the inter-packet gap overhead

this is not a problem to abandon, but one to track down. The result of 
this problem is that you need to be able to control

1. the aggregation parameters (how long will the subsystem wait for more 
traffic, and how much is it willing to combine).

2. the amount of data that is sent to the driver at once. You want to send 
enough data in as short a time period as possible to fill the aggregation 
chunk, even if that means sending packets that are futher up in your queue 
and skipping over more 'deserving' packets. The reason for this is that 
the additional packets included in the aggregation are MUCH cheaper to 
send.

the ideal situation is that the stack aggredation doesn't actually wait 
for additional packets going to this destination, it just looks in it's 
queue and sends all (up to a limit) of the packets going to this 
destination when it sends the first one.


going slightly off topic with a relavent example.

rsyslog has the ability to write log messages to a database. It has the 
ability to do so using transactions so that it gets confirmation that the 
logs are really safe (not just in some memory buffer somewhere that can be 
lost in a power failure)

with a database, the overhead of a transaction is large enough that you 
can frequently insert 100 or 1000 entries in the same wall clock time that 
it takes you to insert one entry (within about a 10% error margin, at 
least at the high end of the range). Rsyslog gained the ability to batch 
message delivery by taking the approach:

each time you look at the queue, grab up to N (configurable) messages and 
deliver them all at once

how this works in the real world is that the first message that arrives 
gets delivered by itself, but while that message is getting delivered 
inefficently, the queue builds up a bit. when it goes back to look for 
more work, there is now a set of multiple messages to be delivered.

this approch 'wastes' bandwidth, but it results in all messages being 
delivered with the minimum possible latency (yes, waiting a little bit to 
get new messages would reduce the latency of a few messages, but it would 
increase the latency for far more messages), and is self-tuning as the 
rate of new messages arriving in the queue and the available bandwidth for 
outputting the messages varies (because, just like with the RF airtime, 
the disk I/O bandwidth is a scarce resource that you are in competition 
for)


If the kernel stack uses a similar approach, then what you want to do is 
to try to do the same and feed it chunks of the right size. If it doesn't 
use this sort of algorithm, it may be that it needs to be changed to 
eliminate the timeout-based mechanism that it has.


>> I was assuming that if you are running a mized network you only use a single
>> channel for N. If you are using multiple channels for N they should be a
>> separate SSID, just from the fact that you are using two channels for N but
>> only one for G (which one would be the question)
>
> One channel for both N and G in this case. Only one radio for 5ghz.

many APs with only one radio in a band can still configure that radio to 
use a 40MHz wide channel for N rather than the standard 20MHz wide channel 
that G uses. Many people try to do this on 2.4Ghz (which is only about 
60MHz wide for the entire band)

>>>>
>>>> there is some
>>>
>>>
>>> Some?
>>
>>
>> some, but it's an unavoidable feature of wireless communication. You can
>> consider turning off some modulation types, but since the clients
>> automatically fall back to slower modulation types when there is a problem,
>> the result will be failed connections.
>
> To give you an idea, at 5ghz I'm capable with cerowrt at achieving 150Mbits
> with TCP - in the clean air here.
>
> At 2.4, it's rare I can get more than 20, and fairly often much less than that.
>
> Any given test I run regarding wireless simply is not repeatable if I do it
> on 2.4ghz.
>
> I can usually 'hear' more than 30 access points at my apt, as another
> example.
>
>
>>
>> now, this may still be the right thing to do, because the failback to a
>> slower modulation type works well for weak signal situations, but in a high
>> density situation (which is basically every 2.4G deployment in the real
>> world nowdays), taking longer to send the same data just means that you are
>> more vunerable to another transmitter clobbering you, so it actually
>> decreases reliability.
>
> yes, minstrel rocks.

minstrel?


David Lang