[Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 8663 bytes --]

I would like to have somehow assembled all the focused resources to make a
go at fixing wifi, or at least having a f2f with a bunch of people in the
late march timeframe. This message of mine to linux-wireless bounced for
some reason and I am off to log out for 10 days, so...

see relevant netdev thread also for ore details.

---------- Forwarded message ----------
From: Dave Taht <dave.taht@gmail.com>
Date: Sat, Jan 31, 2015 at 12:29 PM
Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
To: Arend van Spriel <arend@broadcom.com>
Cc: linux-wireless <linux-wireless@vger.kernel.org>, Michal Kazior <
michal.kazior@tieto.com>, Eyal Perry <eyalpe@dev.mellanox.co.il>, Network
Development <netdev@vger.kernel.org>, Eric Dumazet <eric.dumazet@gmail.com>


The wifi industry as a whole has vastly bigger problems than achieving
1500Mbits in a faraday cage on a single flow.

I encourage you to try tests in netperf-wrapper that explicitly test for
latency under load, and in particular, the RTT_FAIR tests against 4 or more
stations on a single wifi AP. You will find the results very depressing.
Similarly, on your previous test series, a latency figure would have been
nice to have. I just did a talk at nznog, where I tested the local wifi
with less than ambits of throughput, and 3 seconds of latency, filmed here:

https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt

Do wish more folk were testing in the busy real world environments, like
coffee shops, cities... really, anywhere outside a faraday cage!

I am not attending netconf - I was unable to raise funds to go, and the
program committee wanted something "new",

instead of the preso I gave the IEEE 802.11 working group back in
september. (
http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
)

I was very pleased with the results of that talk - the day after I gave it,
the phrase "test for latency" showed up in a bunch of 802.11ax (the next
generation after ac) documents. :) Still, we are stuck with the train wreck
that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g, in
terms of queue management, terrible uses of airtime, rate control and other
stuff. Aruba and Meraki, in particular took a big interest in what I'd
outlined in the preso above (we have a half dozen less well baked ideas -
that's just the easy stuff that can be done to improve wifi).  I gave a
followup at meraki but I don't think that's online.

Felix (nbd) is on vacation right now, as I am I. In fact I am going
somewhere for a week totally lacking internet access.

Presently the plan, with what budget (none) we have and time (very little)
we have is to produce a pair of proof of concept implementations for per
tid queuing (see relevant commit by nbd),  leveraging the new minstrel
stats, the new minstrel-blues stuff, and an aggregation aware codel with a
calculated target based on the most recently active stations, and a bunch
of the other stuff outlined above at IEEE.

It is my hope that this will start to provide accurate back pressure (or
sufficient lack thereof for TSQ), to also improve throughput while still
retaining low latency. But it is a certainty that we will run into more
cross layer issues that will be a pita to resolve.

If we can put together a meet up around or during ELC in california in
march?

I am really not terribly optimistic on anything other than the 2 chipsets
we can hack on (ath9k, mt76). Negotiations to get qualcomm to open up their
ath10k firmware have thus far failed, nor has a ath10k-lite got anywhere.
Perhaps broadcom would be willing to open up their firmware sufficiently to
build in a better API?

A bit more below.


On Jan 30, 2015 5:59 AM, "Arend van Spriel" <arend@broadcom.com> wrote:
>
> On 01/30/15 14:19, Eric Dumazet wrote:
>>
>> On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
>>
>>> Hi Eric,
>>>
>>> Your suggestions are still based on the fact that you consider wireless
>>> networking to be similar to ethernet, but as Michal indicated there are
>>> some fundamental differences starting with CSMA/CD versus CSMA/CA. Also
>>> the medium conditions are far from comparable.

The analogy i now use for it is that switched ethernet is generally your
classic "dumbbell"

topology. Wifi is more like a "taxi-stand" topology. If you think about how
people

queue up at a taxi stand (and sometimes agree to share a ride), the inter
arrival

and departure times of a taxi stand make for a better mental model.

Admittedly, I seem to spend a lot of time, waiting for taxies, thinking
about

wifi.

>> There is no shielding so
>>> it needs to deal with interference and dynamically drops the link rate
>>> so transmission of packets can take several milliseconds. Then with 11n
>>> they came up with aggregation with sends up to 64 packets in a single
>>> transmit over the air at worst case 6.5 Mbps (if I am not mistaken). The
>>> parameter value for tcp_limit_output_bytes of 131072 means that it
>>> allows queuing for about 1ms on a 1Gbps link, but I hope you can see
>>> this is not realistic for dealing with all variances of the wireless
>>> medium/standard. I suggested this as topic for the wireless workshop in
>>> Otawa [1], but I can not attend there. Still hope that there will be
>>> some discussions to get more awareness.

I have sometimes hoped that TSQ could be made more a function of the

number of active flows exiting an interface, but eric tells me that's
impossible.

This is possibly another case where TSQ could use to be a callback
function...

but frankly I care not a whit about maximizing single flow tcp throughput
on wifi

in a faraday cage.


>>
>> Ever heard about bufferbloat ?
>
>
> Sure. I am trying to get awareness about that in our wireless
driver/firmware development teams. So bear with me.
>
>
>> Have you read my suggestions and tried them ?
>>
>> You can adjust the limit per flow to pretty much you want. If you need
>> 64 packets, just do the math. If in 2018 you need 128 packets, do the
>> math again.
>>
>> I am very well aware that wireless wants aggregation, thank you.

I note that a lot of people testing this are getting it backwards. Usually
it is the AP that is sending lots and lots of big packets, where the return
path is predominately acks from the station.

I am not a huge fan of stretch acks, but certainly a little bit of thinning
doesn't bother me on the return path there.

Going the other way, particularly in a wifi world that insists on treating
every packet as sacred (which I don't agree with at all), thinning acks can
help, but single stream throughput is of interest only on benchmarks, FQing
as much as possible all the flows destined the station in each aggregate
masks loss and reduces the need to protect everything so much.

>
> Sorry if I offended you. I was just giving these as example combined with
effective rate usable on the medium to say that the bandwidth is more
dynamic in wireless and as such need dynamic change of queue depth. Now
this can be done by making the fraction size as used in your suggestion
adaptive to these conditions.

Well... see above. Maybe this technique will do more of the right thing,
but... go test.


>
>> 131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing, and
>> we get line rate nevertheless.
>
>
> I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are
moving into that direction in 11ac.
>
>
>> We need this level of shallow queues (BQL, TSQ), to get very precise rtt
>> estimations so that TCP has good entropy for its pacing, even in the 50
>> usec rtt ranges.
>>
>> If we allowed 1ms of queueing, then a 40Gbit flow would queue 5 MBytes.
>>
>> This was terrible, because it increased cwnd and all sender queues to
>> insane levels.
>
>
> Indeed and that is what we would like to address in our wireless drivers.
I will setup some experiments using the fraction sizing and post my
findings. Again sorry if I offended you.

You really, really, really need to test at rates below 50mbit and with
other stations, also while doing this. It's not going to be a linear curve.


>
> Regards,
> Arend
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

[-- Attachment #2: Type: text/html, Size: 10857 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 14070 bytes --]


I think we need to create an Internet focused 802.11 working group that would be to the "OS wireless designers and IEEE 802.11 standards groups" as the WHATML group was to W3C.
 
W3C was clueless about the real world at the point WHATML was created.  And WHATML was a "revenge of the real" against W3C - advancing a wide variety of important practical innovations rather than attending endless standards meetings with people who were not focused on solving actually important problems.
 
It took a bunch of work to get WHATML going, and it offended W3C, who became unhelpful.  But the approach actually worked - we now have a Web that really uses browser-side expressivity and that would never have happened if W3C were left to its own devices.
 
The WiFi consortium was an attempt to wrest control of pragmatic direction from 802.11 and the proprietary-divergence folks at Qualcomm, Broadcom, Cisco, etc.  But it failed, because it became thieves on a raft, more focused on picking each others' pockets than on actually addressing the big issues.
 
Jim has seen this play out in the Linux community around X.  Though there are lots of interests who would benefit by moving the engineering ball forward, everyone resists action because it means giving up the chance at dominance, and the central group is far too weak to do anything beyond adjudicating the worst battles.
 
When I say "we" I definitely include myself (though my time is limited due to other commitments and the need to support my family), but I would only play with people who actually are committed to making stuff happen - which includes raising hell with the vendors if need be, but also effective engineering steps that can achieve quick adoption.
 
Sadly, and I think it is manageable at the moment, there are moves out there being made to get the FCC to "protect" WiFi from "interference".  The current one was Marriott, who requested the FCC for a rule to make it legal to disrupt and block use of WiFi in people's rooms in their hotels, except with their access points.  This also needs some technical defense.  I believe any issues with WiFi performance in actual Marriott hotels are due to bufferbloat in their hotel-wide systems, just as the issues with GoGo are the same.  But it's possible that queueing problems in their own WiFi gear are bad as well.
 
I mention this because it is related, and to the layperson, or non-radio-knowledgeable executive, indistinguishable.  It will take away the incentive to actually fix the 802.11 implementations to be better performing, making the problem seem to be a "management" issue that can be solved by making WiFi less interoperable and less flexible by rules, rather than by engineering.
 
However, solving the problems of hotspot networks and hotel networks are definitely "real world" issues, and quite along the same lines you mention, Dave.  FQ is almost certainly a big deal both in WiFi and in the distribution networks behind WiFi. Co-existence is also a big deal (RTS/CTS-like mechanisms can go a long way to remediate hidden-terminal disruption of the basic protocols). Roaming and scaling need work as well.
 
It would even be a good thing to invent pragmatic ways to provide "low rate" subnets and "high rate" subnets that can coexist, so that compatibility with ancient "b" networks need not be maintained on all nets, at great cost - just send beacons at a high rate, so that the "b" NICs can't see them.... but you need pragmatic stack implementations.
 
But the engineering is not the only challenge. The other challenge is to take the initiative and get stuff deployed.  In the case of bufferbloat, the grade currently is a "D" for deployments, maybe a "D-".  Beautiful technical work, but the economic/business/political side of things has been poor.  Look at how slow IETF has been to achieve anything (the perfect is truly the enemy of the good, and Dave Clark's "rough consensus and working code" has been replaced by technocratic malaise, and what appears to me to be a class of people who love traveling the world to a floating cocktail party without getting anything important done).
 
The problem with communications is that you can't just ship a product with a new "feature", because the innovation only works if widely adopted.  Since there is no "Linux Desktop" (and Linus hates the idea, to a large extent) Linux can't be the sole carrier of the idea.  You pretty much need iOS and Android both to buy in or to provide a path for easy third-party upgrades.  How do you do that?  Well, that's where the WHATML-type approach is necessary.
 
I don't know if this can be achieved, and there are lots of details to be worked out.  But I'll play.
 
 


On Saturday, January 31, 2015 4:05pm, "Dave Taht" <dave.taht@gmail.com> said:



I would like to have somehow assembled all the focused resources to make a go at fixing wifi, or at least having a f2f with a bunch of people in the late march timeframe. This message of mine to linux-wireless bounced for some reason and I am off to log out for 10 days, so...
see relevant netdev thread also for ore details.


---------- Forwarded message ----------
From: Dave Taht <[ dave.taht@gmail.com ]( mailto:dave.taht@gmail.com )>
Date: Sat, Jan 31, 2015 at 12:29 PM
Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
To: Arend van Spriel <[ arend@broadcom.com ]( mailto:arend@broadcom.com )>
Cc: linux-wireless <[ linux-wireless@vger.kernel.org ]( mailto:linux-wireless@vger.kernel.org )>, Michal Kazior <[ michal.kazior@tieto.com ]( mailto:michal.kazior@tieto.com )>, Eyal Perry <[ eyalpe@dev.mellanox.co.il ]( mailto:eyalpe@dev.mellanox.co.il )>, Network Development <[ netdev@vger.kernel.org ]( mailto:netdev@vger.kernel.org )>, Eric Dumazet <[ eric.dumazet@gmail.com ]( mailto:eric.dumazet@gmail.com )>




The wifi industry as a whole has vastly bigger problems than achieving 1500Mbits in a faraday cage on a single flow.
I encourage you to try tests in netperf-wrapper that explicitly test for latency under load, and in particular, the RTT_FAIR tests against 4 or more stations on a single wifi AP. You will find the results very depressing. Similarly, on your previous test series, a latency figure would have been nice to have. I just did a talk at nznog, where I tested the local wifi with less than ambits of throughput, and 3 seconds of latency, filmed here: 
[ https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt ]( https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt )
Do wish more folk were testing in the busy real world environments, like coffee shops, cities... really, anywhere outside a faraday cage!
I am not attending netconf - I was unable to raise funds to go, and the program committee wanted something "new",
instead of the preso I gave the IEEE 802.11 working group back in september. ( [ http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf ]( http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf ) )
I was very pleased with the results of that talk - the day after I gave it, the phrase "test for latency" showed up in a bunch of 802.11ax (the next generation after ac) documents. :) Still, we are stuck with the train wreck that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g, in terms of queue management, terrible uses of airtime, rate control and other stuff. Aruba and Meraki, in particular took a big interest in what I'd outlined in the preso above (we have a half dozen less well baked ideas - that's just the easy stuff that can be done to improve wifi).  I gave a followup at meraki but I don't think that's online.
Felix (nbd) is on vacation right now, as I am I. In fact I am going somewhere for a week totally lacking internet access.
Presently the plan, with what budget (none) we have and time (very little) we have is to produce a pair of proof of concept implementations for per tid queuing (see relevant commit by nbd),  leveraging the new minstrel stats, the new minstrel-blues stuff, and an aggregation aware codel with a calculated target based on the most recently active stations, and a bunch of the other stuff outlined above at IEEE.
It is my hope that this will start to provide accurate back pressure (or sufficient lack thereof for TSQ), to also improve throughput while still retaining low latency. But it is a certainty that we will run into more cross layer issues that will be a pita to resolve.
If we can put together a meet up around or during ELC in california in march? 
I am really not terribly optimistic on anything other than the 2 chipsets we can hack on (ath9k, mt76). Negotiations to get qualcomm to open up their ath10k firmware have thus far failed, nor has a ath10k-lite got anywhere. Perhaps broadcom would be willing to open up their firmware sufficiently to build in a better API?
A bit more below.

 On Jan 30, 2015 5:59 AM, "Arend van Spriel" <[ arend@broadcom.com ]( mailto:arend@broadcom.com )> wrote:
 >
 > On 01/30/15 14:19, Eric Dumazet wrote:
 >>
 >> On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
 >>
 >>> Hi Eric,
 >>>
 >>> Your suggestions are still based on the fact that you consider wireless
 >>> networking to be similar to ethernet, but as Michal indicated there are
 >>> some fundamental differences starting with CSMA/CD versus CSMA/CA. Also
 >>> the medium conditions are far from comparable. 
The analogy i now use for it is that switched ethernet is generally your classic "dumbbell"
topology. Wifi is more like a "taxi-stand" topology. If you think about how people
queue up at a taxi stand (and sometimes agree to share a ride), the inter arrival
and departure times of a taxi stand make for a better mental model. 
Admittedly, I seem to spend a lot of time, waiting for taxies, thinking about
wifi.
>> There is no shielding so
 >>> it needs to deal with interference and dynamically drops the link rate
 >>> so transmission of packets can take several milliseconds. Then with 11n
 >>> they came up with aggregation with sends up to 64 packets in a single
 >>> transmit over the air at worst case 6.5 Mbps (if I am not mistaken). The
 >>> parameter value for tcp_limit_output_bytes of 131072 means that it
 >>> allows queuing for about 1ms on a 1Gbps link, but I hope you can see
 >>> this is not realistic for dealing with all variances of the wireless
 >>> medium/standard. I suggested this as topic for the wireless workshop in
 >>> Otawa [1], but I can not attend there. Still hope that there will be
 >>> some discussions to get more awareness.

I have sometimes hoped that TSQ could be made more a function of the
number of active flows exiting an interface, but eric tells me that's impossible.
This is possibly another case where TSQ could use to be a callback function...
but frankly I care not a whit about maximizing single flow tcp throughput on wifi
in a faraday cage.

 >>
 >> Ever heard about bufferbloat ?
 >
 >
 > Sure. I am trying to get awareness about that in our wireless driver/firmware development teams. So bear with me.
 >
 >
 >> Have you read my suggestions and tried them ?
 >>
 >> You can adjust the limit per flow to pretty much you want. If you need
 >> 64 packets, just do the math. If in 2018 you need 128 packets, do the
 >> math again.
 >>
 >> I am very well aware that wireless wants aggregation, thank you.

I note that a lot of people testing this are getting it backwards. Usually it is the AP that is sending lots and lots of big packets, where the return path is predominately acks from the station. 
I am not a huge fan of stretch acks, but certainly a little bit of thinning doesn't bother me on the return path there.
Going the other way, particularly in a wifi world that insists on treating every packet as sacred (which I don't agree with at all), thinning acks can help, but single stream throughput is of interest only on benchmarks, FQing as much as possible all the flows destined the station in each aggregate masks loss and reduces the need to protect everything so much.
 >
 > Sorry if I offended you. I was just giving these as example combined with effective rate usable on the medium to say that the bandwidth is more dynamic in wireless and as such need dynamic change of queue depth. Now this can be done by making the fraction size as used in your suggestion adaptive to these conditions.

Well... see above. Maybe this technique will do more of the right thing, but... go test.

 >
 >> 131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing, and
 >> we get line rate nevertheless.
 >
 >
 > I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are moving into that direction in 11ac.
 >
 >
 >> We need this level of shallow queues (BQL, TSQ), to get very precise rtt
 >> estimations so that TCP has good entropy for its pacing, even in the 50
 >> usec rtt ranges.
 >>
 >> If we allowed 1ms of queueing, then a 40Gbit flow would queue 5 MBytes.
 >>
 >> This was terrible, because it increased cwnd and all sender queues to
 >> insane levels.
 >
 >
 > Indeed and that is what we would like to address in our wireless drivers. I will setup some experiments using the fraction sizing and post my findings. Again sorry if I offended you.


You really, really, really need to test at rates below 50mbit and with other stations, also while doing this. It's not going to be a linear curve.
 
>
 > Regards,
 > Arend
 >
 > --
 > To unsubscribe from this list: send the line "unsubscribe netdev" in
 > the body of a message to [ majordomo@vger.kernel.org ]( mailto:majordomo@vger.kernel.org )
 > More majordomo info at  [ http://vger.kernel.org/majordomo-info.html ]( http://vger.kernel.org/majordomo-info.html )

-- 

Dave Täht

thttp://[ www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks ]( http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks )

[-- Attachment #2: Type: text/html, Size: 21592 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Avery Pennarun]

I would argue that insofar as the bufferbloat project has made a
difference, it's because there was a very clear message and product:
- here's what sucks when you have bufferbloat
- here's how you can detect it
- here's how you can get rid of it
- by the way, here's which of your competitors are already beating you at it.

It turns out you don't need a standards org in order to push any of
the above things.  The IEEE exists to make sure things interop at the
MAC/PHY layer.  The IETF exists to make sure things interop at the
higher layers.  But bufferbloat isn't about interop, it's just a thing
that happens inside gateways, so it's not something you can really
write standards about.  It is something you can turn into a
competitive advantage (or disadvantage, if you're a straggler).

...but meanwhile, if we want to fix bufferbloat in wifi, nobody
actually knows how to do it, so we are still at steps 1 and 2.

This is why we're funding Dave to continue work on netperf-wrappers.
Those diagrams of latency under load are pretty convincing.  The
diagrams of page load times under different levels of latency are even
more convincing.  First, we prove there's a problem and a way to
measure the problem.  Then hopefully more people will be interested in
solving it.


On Sat, Jan 31, 2015 at 4:51 PM,  <dpreed@reed.com> wrote:
> I think we need to create an Internet focused 802.11 working group that
> would be to the "OS wireless designers and IEEE 802.11 standards groups" as
> the WHATML group was to W3C.
>
>
>
> W3C was clueless about the real world at the point WHATML was created.  And
> WHATML was a "revenge of the real" against W3C - advancing a wide variety of
> important practical innovations rather than attending endless standards
> meetings with people who were not focused on solving actually important
> problems.
>
>
>
> It took a bunch of work to get WHATML going, and it offended W3C, who became
> unhelpful.  But the approach actually worked - we now have a Web that really
> uses browser-side expressivity and that would never have happened if W3C
> were left to its own devices.
>
>
>
> The WiFi consortium was an attempt to wrest control of pragmatic direction
> from 802.11 and the proprietary-divergence folks at Qualcomm, Broadcom,
> Cisco, etc.  But it failed, because it became thieves on a raft, more
> focused on picking each others' pockets than on actually addressing the big
> issues.
>
>
>
> Jim has seen this play out in the Linux community around X.  Though there
> are lots of interests who would benefit by moving the engineering ball
> forward, everyone resists action because it means giving up the chance at
> dominance, and the central group is far too weak to do anything beyond
> adjudicating the worst battles.
>
>
>
> When I say "we" I definitely include myself (though my time is limited due
> to other commitments and the need to support my family), but I would only
> play with people who actually are committed to making stuff happen - which
> includes raising hell with the vendors if need be, but also effective
> engineering steps that can achieve quick adoption.
>
>
>
> Sadly, and I think it is manageable at the moment, there are moves out there
> being made to get the FCC to "protect" WiFi from "interference".  The
> current one was Marriott, who requested the FCC for a rule to make it legal
> to disrupt and block use of WiFi in people's rooms in their hotels, except
> with their access points.  This also needs some technical defense.  I
> believe any issues with WiFi performance in actual Marriott hotels are due
> to bufferbloat in their hotel-wide systems, just as the issues with GoGo are
> the same.  But it's possible that queueing problems in their own WiFi gear
> are bad as well.
>
>
>
> I mention this because it is related, and to the layperson, or
> non-radio-knowledgeable executive, indistinguishable.  It will take away the
> incentive to actually fix the 802.11 implementations to be better
> performing, making the problem seem to be a "management" issue that can be
> solved by making WiFi less interoperable and less flexible by rules, rather
> than by engineering.
>
>
>
> However, solving the problems of hotspot networks and hotel networks are
> definitely "real world" issues, and quite along the same lines you mention,
> Dave.  FQ is almost certainly a big deal both in WiFi and in the
> distribution networks behind WiFi. Co-existence is also a big deal
> (RTS/CTS-like mechanisms can go a long way to remediate hidden-terminal
> disruption of the basic protocols). Roaming and scaling need work as well.
>
>
>
> It would even be a good thing to invent pragmatic ways to provide "low rate"
> subnets and "high rate" subnets that can coexist, so that compatibility with
> ancient "b" networks need not be maintained on all nets, at great cost -
> just send beacons at a high rate, so that the "b" NICs can't see them....
> but you need pragmatic stack implementations.
>
>
>
> But the engineering is not the only challenge. The other challenge is to
> take the initiative and get stuff deployed.  In the case of bufferbloat, the
> grade currently is a "D" for deployments, maybe a "D-".  Beautiful technical
> work, but the economic/business/political side of things has been poor.
> Look at how slow IETF has been to achieve anything (the perfect is truly the
> enemy of the good, and Dave Clark's "rough consensus and working code" has
> been replaced by technocratic malaise, and what appears to me to be a class
> of people who love traveling the world to a floating cocktail party without
> getting anything important done).
>
>
>
> The problem with communications is that you can't just ship a product with a
> new "feature", because the innovation only works if widely adopted.  Since
> there is no "Linux Desktop" (and Linus hates the idea, to a large extent)
> Linux can't be the sole carrier of the idea.  You pretty much need iOS and
> Android both to buy in or to provide a path for easy third-party upgrades.
> How do you do that?  Well, that's where the WHATML-type approach is
> necessary.
>
>
>
> I don't know if this can be achieved, and there are lots of details to be
> worked out.  But I'll play.
>
>
>
>
>
>
>
> On Saturday, January 31, 2015 4:05pm, "Dave Taht" <dave.taht@gmail.com>
> said:
>
> I would like to have somehow assembled all the focused resources to make a
> go at fixing wifi, or at least having a f2f with a bunch of people in the
> late march timeframe. This message of mine to linux-wireless bounced for
> some reason and I am off to log out for 10 days, so...
> see relevant netdev thread also for ore details.
>
> ---------- Forwarded message ----------
> From: Dave Taht <dave.taht@gmail.com>
> Date: Sat, Jan 31, 2015 at 12:29 PM
> Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
> To: Arend van Spriel <arend@broadcom.com>
> Cc: linux-wireless <linux-wireless@vger.kernel.org>, Michal Kazior
> <michal.kazior@tieto.com>, Eyal Perry <eyalpe@dev.mellanox.co.il>, Network
> Development <netdev@vger.kernel.org>, Eric Dumazet <eric.dumazet@gmail.com>
>
>
> The wifi industry as a whole has vastly bigger problems than achieving
> 1500Mbits in a faraday cage on a single flow.
>
> I encourage you to try tests in netperf-wrapper that explicitly test for
> latency under load, and in particular, the RTT_FAIR tests against 4 or more
> stations on a single wifi AP. You will find the results very depressing.
> Similarly, on your previous test series, a latency figure would have been
> nice to have. I just did a talk at nznog, where I tested the local wifi with
> less than ambits of throughput, and 3 seconds of latency, filmed here:
>
> https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt
>
> Do wish more folk were testing in the busy real world environments, like
> coffee shops, cities... really, anywhere outside a faraday cage!
>
> I am not attending netconf - I was unable to raise funds to go, and the
> program committee wanted something "new",
>
> instead of the preso I gave the IEEE 802.11 working group back in september.
> (
> http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
> )
>
> I was very pleased with the results of that talk - the day after I gave it,
> the phrase "test for latency" showed up in a bunch of 802.11ax (the next
> generation after ac) documents. :) Still, we are stuck with the train wreck
> that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g, in
> terms of queue management, terrible uses of airtime, rate control and other
> stuff. Aruba and Meraki, in particular took a big interest in what I'd
> outlined in the preso above (we have a half dozen less well baked ideas -
> that's just the easy stuff that can be done to improve wifi).  I gave a
> followup at meraki but I don't think that's online.
>
> Felix (nbd) is on vacation right now, as I am I. In fact I am going
> somewhere for a week totally lacking internet access.
>
> Presently the plan, with what budget (none) we have and time (very little)
> we have is to produce a pair of proof of concept implementations for per tid
> queuing (see relevant commit by nbd),  leveraging the new minstrel stats,
> the new minstrel-blues stuff, and an aggregation aware codel with a
> calculated target based on the most recently active stations, and a bunch of
> the other stuff outlined above at IEEE.
>
> It is my hope that this will start to provide accurate back pressure (or
> sufficient lack thereof for TSQ), to also improve throughput while still
> retaining low latency. But it is a certainty that we will run into more
> cross layer issues that will be a pita to resolve.
>
> If we can put together a meet up around or during ELC in california in
> march?
>
> I am really not terribly optimistic on anything other than the 2 chipsets we
> can hack on (ath9k, mt76). Negotiations to get qualcomm to open up their
> ath10k firmware have thus far failed, nor has a ath10k-lite got anywhere.
> Perhaps broadcom would be willing to open up their firmware sufficiently to
> build in a better API?
>
> A bit more below.
>
>
> On Jan 30, 2015 5:59 AM, "Arend van Spriel" <arend@broadcom.com> wrote:
>>
>> On 01/30/15 14:19, Eric Dumazet wrote:
>>>
>>> On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> Your suggestions are still based on the fact that you consider wireless
>>>> networking to be similar to ethernet, but as Michal indicated there are
>>>> some fundamental differences starting with CSMA/CD versus CSMA/CA. Also
>>>> the medium conditions are far from comparable.
>
> The analogy i now use for it is that switched ethernet is generally your
> classic "dumbbell"
>
> topology. Wifi is more like a "taxi-stand" topology. If you think about how
> people
>
> queue up at a taxi stand (and sometimes agree to share a ride), the inter
> arrival
>
> and departure times of a taxi stand make for a better mental model.
>
> Admittedly, I seem to spend a lot of time, waiting for taxies, thinking
> about
>
> wifi.
>
>>> There is no shielding so
>>>> it needs to deal with interference and dynamically drops the link rate
>>>> so transmission of packets can take several milliseconds. Then with 11n
>>>> they came up with aggregation with sends up to 64 packets in a single
>>>> transmit over the air at worst case 6.5 Mbps (if I am not mistaken). The
>>>> parameter value for tcp_limit_output_bytes of 131072 means that it
>>>> allows queuing for about 1ms on a 1Gbps link, but I hope you can see
>>>> this is not realistic for dealing with all variances of the wireless
>>>> medium/standard. I suggested this as topic for the wireless workshop in
>>>> Otawa [1], but I can not attend there. Still hope that there will be
>>>> some discussions to get more awareness.
>
> I have sometimes hoped that TSQ could be made more a function of the
>
> number of active flows exiting an interface, but eric tells me that's
> impossible.
>
> This is possibly another case where TSQ could use to be a callback
> function...
>
> but frankly I care not a whit about maximizing single flow tcp throughput on
> wifi
>
> in a faraday cage.
>
>
>>>
>>> Ever heard about bufferbloat ?
>>
>>
>> Sure. I am trying to get awareness about that in our wireless
>> driver/firmware development teams. So bear with me.
>>
>>
>>> Have you read my suggestions and tried them ?
>>>
>>> You can adjust the limit per flow to pretty much you want. If you need
>>> 64 packets, just do the math. If in 2018 you need 128 packets, do the
>>> math again.
>>>
>>> I am very well aware that wireless wants aggregation, thank you.
>
> I note that a lot of people testing this are getting it backwards. Usually
> it is the AP that is sending lots and lots of big packets, where the return
> path is predominately acks from the station.
>
> I am not a huge fan of stretch acks, but certainly a little bit of thinning
> doesn't bother me on the return path there.
>
> Going the other way, particularly in a wifi world that insists on treating
> every packet as sacred (which I don't agree with at all), thinning acks can
> help, but single stream throughput is of interest only on benchmarks, FQing
> as much as possible all the flows destined the station in each aggregate
> masks loss and reduces the need to protect everything so much.
>
>>
>> Sorry if I offended you. I was just giving these as example combined with
>> effective rate usable on the medium to say that the bandwidth is more
>> dynamic in wireless and as such need dynamic change of queue depth. Now this
>> can be done by making the fraction size as used in your suggestion adaptive
>> to these conditions.
>
> Well... see above. Maybe this technique will do more of the right thing,
> but... go test.
>
>
>>
>>> 131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing, and
>>> we get line rate nevertheless.
>>
>>
>> I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are
>> moving into that direction in 11ac.
>>
>>
>>> We need this level of shallow queues (BQL, TSQ), to get very precise rtt
>>> estimations so that TCP has good entropy for its pacing, even in the 50
>>> usec rtt ranges.
>>>
>>> If we allowed 1ms of queueing, then a 40Gbit flow would queue 5 MBytes.
>>>
>>> This was terrible, because it increased cwnd and all sender queues to
>>> insane levels.
>>
>>
>> Indeed and that is what we would like to address in our wireless drivers.
>> I will setup some experiments using the fraction sizing and post my
>> findings. Again sorry if I offended you.
>
> You really, really, really need to test at rates below 50mbit and with other
> stations, also while doing this. It's not going to be a linear curve.
>
>
>
>>
>> Regards,
>> Arend
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Dave Täht
>
> thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 18488 bytes --]

I think I have a good idea how to do it, actually.  Or at least, I had a
design sketched out that was kind of plausible.

In essence, we have a good way of controlling queues in the IP stack, that
being fq_codel.  But aggregating wifi drivers have to pull all the traffic
straight through that and queue it themselves, with a different rotation
policy and (at present) no AQM.

Well, that can be fixed.  We need a transmission and aggregation scheduler
at the wifi layer that can do a per-station round-robin with a time-based
fairness policy, based on airtime.  This has to be in the wifi layer
because that's where rate selection policy is, and you can't calculate
airtime until you know the selected rate, the current success
probabilities, and how much aggregation you're going to do within your time
budget.  We need to be able to plug in per-flow and per-packet scheduling
and drop policies on top of that, to control scheduling of individual
packets into aggregates.

We could also do the trick of assigning a target drop probability and
telling a Minstrel-like rate controller what drop probability we're after;
then you get the nice property that the transmission scheduler and rate
control are work-conserving and actually get faster when they get congested.

There is also now the mathematics available to improve Minstrel, which is
about 80% of a truly optimal rate controller (interestingly, that math was
published in 2010, years after we did Minstrel; I had the intuition that it
was close to optimal but no way to do a proof).

None of this is rocket science, but it is a pretty deep and complex
refactor of something that isn't the simplest code to start with.  That
said, I'd not expect it to be more than around 1k LoC, it's just going to
be a real bear to test.

I do have the feeling that the people receiving this email ought to be
sufficient to fix the problem, if we can direct enough time to it; for
myself, I have to think quite hard as to whether this should be my 20%
project.

On Sun, Feb 1, 2015 at 2:06 PM, Avery Pennarun <apenwarr@google.com> wrote:

> I would argue that insofar as the bufferbloat project has made a
> difference, it's because there was a very clear message and product:
> - here's what sucks when you have bufferbloat
> - here's how you can detect it
> - here's how you can get rid of it
> - by the way, here's which of your competitors are already beating you at
> it.
>
> It turns out you don't need a standards org in order to push any of
> the above things.  The IEEE exists to make sure things interop at the
> MAC/PHY layer.  The IETF exists to make sure things interop at the
> higher layers.  But bufferbloat isn't about interop, it's just a thing
> that happens inside gateways, so it's not something you can really
> write standards about.  It is something you can turn into a
> competitive advantage (or disadvantage, if you're a straggler).
>
> ...but meanwhile, if we want to fix bufferbloat in wifi, nobody
> actually knows how to do it, so we are still at steps 1 and 2.
>
> This is why we're funding Dave to continue work on netperf-wrappers.
> Those diagrams of latency under load are pretty convincing.  The
> diagrams of page load times under different levels of latency are even
> more convincing.  First, we prove there's a problem and a way to
> measure the problem.  Then hopefully more people will be interested in
> solving it.
>
>
> On Sat, Jan 31, 2015 at 4:51 PM,  <dpreed@reed.com> wrote:
> > I think we need to create an Internet focused 802.11 working group that
> > would be to the "OS wireless designers and IEEE 802.11 standards groups"
> as
> > the WHATML group was to W3C.
> >
> >
> >
> > W3C was clueless about the real world at the point WHATML was created.
> And
> > WHATML was a "revenge of the real" against W3C - advancing a wide
> variety of
> > important practical innovations rather than attending endless standards
> > meetings with people who were not focused on solving actually important
> > problems.
> >
> >
> >
> > It took a bunch of work to get WHATML going, and it offended W3C, who
> became
> > unhelpful.  But the approach actually worked - we now have a Web that
> really
> > uses browser-side expressivity and that would never have happened if W3C
> > were left to its own devices.
> >
> >
> >
> > The WiFi consortium was an attempt to wrest control of pragmatic
> direction
> > from 802.11 and the proprietary-divergence folks at Qualcomm, Broadcom,
> > Cisco, etc.  But it failed, because it became thieves on a raft, more
> > focused on picking each others' pockets than on actually addressing the
> big
> > issues.
> >
> >
> >
> > Jim has seen this play out in the Linux community around X.  Though there
> > are lots of interests who would benefit by moving the engineering ball
> > forward, everyone resists action because it means giving up the chance at
> > dominance, and the central group is far too weak to do anything beyond
> > adjudicating the worst battles.
> >
> >
> >
> > When I say "we" I definitely include myself (though my time is limited
> due
> > to other commitments and the need to support my family), but I would only
> > play with people who actually are committed to making stuff happen -
> which
> > includes raising hell with the vendors if need be, but also effective
> > engineering steps that can achieve quick adoption.
> >
> >
> >
> > Sadly, and I think it is manageable at the moment, there are moves out
> there
> > being made to get the FCC to "protect" WiFi from "interference".  The
> > current one was Marriott, who requested the FCC for a rule to make it
> legal
> > to disrupt and block use of WiFi in people's rooms in their hotels,
> except
> > with their access points.  This also needs some technical defense.  I
> > believe any issues with WiFi performance in actual Marriott hotels are
> due
> > to bufferbloat in their hotel-wide systems, just as the issues with GoGo
> are
> > the same.  But it's possible that queueing problems in their own WiFi
> gear
> > are bad as well.
> >
> >
> >
> > I mention this because it is related, and to the layperson, or
> > non-radio-knowledgeable executive, indistinguishable.  It will take away
> the
> > incentive to actually fix the 802.11 implementations to be better
> > performing, making the problem seem to be a "management" issue that can
> be
> > solved by making WiFi less interoperable and less flexible by rules,
> rather
> > than by engineering.
> >
> >
> >
> > However, solving the problems of hotspot networks and hotel networks are
> > definitely "real world" issues, and quite along the same lines you
> mention,
> > Dave.  FQ is almost certainly a big deal both in WiFi and in the
> > distribution networks behind WiFi. Co-existence is also a big deal
> > (RTS/CTS-like mechanisms can go a long way to remediate hidden-terminal
> > disruption of the basic protocols). Roaming and scaling need work as
> well.
> >
> >
> >
> > It would even be a good thing to invent pragmatic ways to provide "low
> rate"
> > subnets and "high rate" subnets that can coexist, so that compatibility
> with
> > ancient "b" networks need not be maintained on all nets, at great cost -
> > just send beacons at a high rate, so that the "b" NICs can't see them....
> > but you need pragmatic stack implementations.
> >
> >
> >
> > But the engineering is not the only challenge. The other challenge is to
> > take the initiative and get stuff deployed.  In the case of bufferbloat,
> the
> > grade currently is a "D" for deployments, maybe a "D-".  Beautiful
> technical
> > work, but the economic/business/political side of things has been poor.
> > Look at how slow IETF has been to achieve anything (the perfect is truly
> the
> > enemy of the good, and Dave Clark's "rough consensus and working code"
> has
> > been replaced by technocratic malaise, and what appears to me to be a
> class
> > of people who love traveling the world to a floating cocktail party
> without
> > getting anything important done).
> >
> >
> >
> > The problem with communications is that you can't just ship a product
> with a
> > new "feature", because the innovation only works if widely adopted.
> Since
> > there is no "Linux Desktop" (and Linus hates the idea, to a large extent)
> > Linux can't be the sole carrier of the idea.  You pretty much need iOS
> and
> > Android both to buy in or to provide a path for easy third-party
> upgrades.
> > How do you do that?  Well, that's where the WHATML-type approach is
> > necessary.
> >
> >
> >
> > I don't know if this can be achieved, and there are lots of details to be
> > worked out.  But I'll play.
> >
> >
> >
> >
> >
> >
> >
> > On Saturday, January 31, 2015 4:05pm, "Dave Taht" <dave.taht@gmail.com>
> > said:
> >
> > I would like to have somehow assembled all the focused resources to make
> a
> > go at fixing wifi, or at least having a f2f with a bunch of people in the
> > late march timeframe. This message of mine to linux-wireless bounced for
> > some reason and I am off to log out for 10 days, so...
> > see relevant netdev thread also for ore details.
> >
> > ---------- Forwarded message ----------
> > From: Dave Taht <dave.taht@gmail.com>
> > Date: Sat, Jan 31, 2015 at 12:29 PM
> > Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
> > To: Arend van Spriel <arend@broadcom.com>
> > Cc: linux-wireless <linux-wireless@vger.kernel.org>, Michal Kazior
> > <michal.kazior@tieto.com>, Eyal Perry <eyalpe@dev.mellanox.co.il>,
> Network
> > Development <netdev@vger.kernel.org>, Eric Dumazet <
> eric.dumazet@gmail.com>
> >
> >
> > The wifi industry as a whole has vastly bigger problems than achieving
> > 1500Mbits in a faraday cage on a single flow.
> >
> > I encourage you to try tests in netperf-wrapper that explicitly test for
> > latency under load, and in particular, the RTT_FAIR tests against 4 or
> more
> > stations on a single wifi AP. You will find the results very depressing.
> > Similarly, on your previous test series, a latency figure would have been
> > nice to have. I just did a talk at nznog, where I tested the local wifi
> with
> > less than ambits of throughput, and 3 seconds of latency, filmed here:
> >
> > https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt
> >
> > Do wish more folk were testing in the busy real world environments, like
> > coffee shops, cities... really, anywhere outside a faraday cage!
> >
> > I am not attending netconf - I was unable to raise funds to go, and the
> > program committee wanted something "new",
> >
> > instead of the preso I gave the IEEE 802.11 working group back in
> september.
> > (
> >
> http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
> > )
> >
> > I was very pleased with the results of that talk - the day after I gave
> it,
> > the phrase "test for latency" showed up in a bunch of 802.11ax (the next
> > generation after ac) documents. :) Still, we are stuck with the train
> wreck
> > that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g, in
> > terms of queue management, terrible uses of airtime, rate control and
> other
> > stuff. Aruba and Meraki, in particular took a big interest in what I'd
> > outlined in the preso above (we have a half dozen less well baked ideas -
> > that's just the easy stuff that can be done to improve wifi).  I gave a
> > followup at meraki but I don't think that's online.
> >
> > Felix (nbd) is on vacation right now, as I am I. In fact I am going
> > somewhere for a week totally lacking internet access.
> >
> > Presently the plan, with what budget (none) we have and time (very
> little)
> > we have is to produce a pair of proof of concept implementations for per
> tid
> > queuing (see relevant commit by nbd),  leveraging the new minstrel stats,
> > the new minstrel-blues stuff, and an aggregation aware codel with a
> > calculated target based on the most recently active stations, and a
> bunch of
> > the other stuff outlined above at IEEE.
> >
> > It is my hope that this will start to provide accurate back pressure (or
> > sufficient lack thereof for TSQ), to also improve throughput while still
> > retaining low latency. But it is a certainty that we will run into more
> > cross layer issues that will be a pita to resolve.
> >
> > If we can put together a meet up around or during ELC in california in
> > march?
> >
> > I am really not terribly optimistic on anything other than the 2
> chipsets we
> > can hack on (ath9k, mt76). Negotiations to get qualcomm to open up their
> > ath10k firmware have thus far failed, nor has a ath10k-lite got anywhere.
> > Perhaps broadcom would be willing to open up their firmware sufficiently
> to
> > build in a better API?
> >
> > A bit more below.
> >
> >
> > On Jan 30, 2015 5:59 AM, "Arend van Spriel" <arend@broadcom.com> wrote:
> >>
> >> On 01/30/15 14:19, Eric Dumazet wrote:
> >>>
> >>> On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
> >>>
> >>>> Hi Eric,
> >>>>
> >>>> Your suggestions are still based on the fact that you consider
> wireless
> >>>> networking to be similar to ethernet, but as Michal indicated there
> are
> >>>> some fundamental differences starting with CSMA/CD versus CSMA/CA.
> Also
> >>>> the medium conditions are far from comparable.
> >
> > The analogy i now use for it is that switched ethernet is generally your
> > classic "dumbbell"
> >
> > topology. Wifi is more like a "taxi-stand" topology. If you think about
> how
> > people
> >
> > queue up at a taxi stand (and sometimes agree to share a ride), the inter
> > arrival
> >
> > and departure times of a taxi stand make for a better mental model.
> >
> > Admittedly, I seem to spend a lot of time, waiting for taxies, thinking
> > about
> >
> > wifi.
> >
> >>> There is no shielding so
> >>>> it needs to deal with interference and dynamically drops the link rate
> >>>> so transmission of packets can take several milliseconds. Then with
> 11n
> >>>> they came up with aggregation with sends up to 64 packets in a single
> >>>> transmit over the air at worst case 6.5 Mbps (if I am not mistaken).
> The
> >>>> parameter value for tcp_limit_output_bytes of 131072 means that it
> >>>> allows queuing for about 1ms on a 1Gbps link, but I hope you can see
> >>>> this is not realistic for dealing with all variances of the wireless
> >>>> medium/standard. I suggested this as topic for the wireless workshop
> in
> >>>> Otawa [1], but I can not attend there. Still hope that there will be
> >>>> some discussions to get more awareness.
> >
> > I have sometimes hoped that TSQ could be made more a function of the
> >
> > number of active flows exiting an interface, but eric tells me that's
> > impossible.
> >
> > This is possibly another case where TSQ could use to be a callback
> > function...
> >
> > but frankly I care not a whit about maximizing single flow tcp
> throughput on
> > wifi
> >
> > in a faraday cage.
> >
> >
> >>>
> >>> Ever heard about bufferbloat ?
> >>
> >>
> >> Sure. I am trying to get awareness about that in our wireless
> >> driver/firmware development teams. So bear with me.
> >>
> >>
> >>> Have you read my suggestions and tried them ?
> >>>
> >>> You can adjust the limit per flow to pretty much you want. If you need
> >>> 64 packets, just do the math. If in 2018 you need 128 packets, do the
> >>> math again.
> >>>
> >>> I am very well aware that wireless wants aggregation, thank you.
> >
> > I note that a lot of people testing this are getting it backwards.
> Usually
> > it is the AP that is sending lots and lots of big packets, where the
> return
> > path is predominately acks from the station.
> >
> > I am not a huge fan of stretch acks, but certainly a little bit of
> thinning
> > doesn't bother me on the return path there.
> >
> > Going the other way, particularly in a wifi world that insists on
> treating
> > every packet as sacred (which I don't agree with at all), thinning acks
> can
> > help, but single stream throughput is of interest only on benchmarks,
> FQing
> > as much as possible all the flows destined the station in each aggregate
> > masks loss and reduces the need to protect everything so much.
> >
> >>
> >> Sorry if I offended you. I was just giving these as example combined
> with
> >> effective rate usable on the medium to say that the bandwidth is more
> >> dynamic in wireless and as such need dynamic change of queue depth. Now
> this
> >> can be done by making the fraction size as used in your suggestion
> adaptive
> >> to these conditions.
> >
> > Well... see above. Maybe this technique will do more of the right thing,
> > but... go test.
> >
> >
> >>
> >>> 131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing,
> and
> >>> we get line rate nevertheless.
> >>
> >>
> >> I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are
> >> moving into that direction in 11ac.
> >>
> >>
> >>> We need this level of shallow queues (BQL, TSQ), to get very precise
> rtt
> >>> estimations so that TCP has good entropy for its pacing, even in the 50
> >>> usec rtt ranges.
> >>>
> >>> If we allowed 1ms of queueing, then a 40Gbit flow would queue 5 MBytes.
> >>>
> >>> This was terrible, because it increased cwnd and all sender queues to
> >>> insane levels.
> >>
> >>
> >> Indeed and that is what we would like to address in our wireless
> drivers.
> >> I will setup some experiments using the fraction sizing and post my
> >> findings. Again sorry if I offended you.
> >
> > You really, really, really need to test at rates below 50mbit and with
> other
> > stations, also while doing this. It's not going to be a linear curve.
> >
> >
> >
> >>
> >> Regards,
> >> Arend
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe netdev" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> > --
> > Dave Täht
> >
> > thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>

[-- Attachment #2: Type: text/html, Size: 22169 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 20483 bytes --]

This convo ended up being cut from netdev due to containing HTML.

I'd really like to get started on finally fixing WiFi starting in April.

---------- Forwarded message ----------
From: "Andrew McGregor" <andrewmcgr@gmail.com>
Date: Feb 1, 2015 12:07 AM
Subject: Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine
TSO autosizing`
To: "Avery Pennarun" <apenwarr@google.com>
Cc: "David Reed" <dpreed@reed.com>, "Dave Taht" <dave.taht@gmail.com>, "Jim
Gettys" <jg@freedesktop.org>, "Tim Shepard" <shep@alum.mit.edu>, "Matt
Mathis" <mattmathis@google.com>, "Jesper Dangaard Brouer" <
jbrouer@redhat.com>, "Jonathan Morton" <chromatix99@gmail.com>, "
cerowrt-devel@lists.bufferbloat.net" <cerowrt-devel@lists.bufferbloat.net>

> I think I have a good idea how to do it, actually.  Or at least, I had a
design sketched out that was kind of plausible.
>
> In essence, we have a good way of controlling queues in the IP stack,
that being fq_codel.  But aggregating wifi drivers have to pull all the
traffic straight through that and queue it themselves, with a different
rotation policy and (at present) no AQM.
>
> Well, that can be fixed.  We need a transmission and aggregation
scheduler at the wifi layer that can do a per-station round-robin with a
time-based fairness policy, based on airtime.  This has to be in the wifi
layer because that's where rate selection policy is, and you can't
calculate airtime until you know the selected rate, the current success
probabilities, and how much aggregation you're going to do within your time
budget.  We need to be able to plug in per-flow and per-packet scheduling
and drop policies on top of that, to control scheduling of individual
packets into aggregates.
>
> We could also do the trick of assigning a target drop probability and
telling a Minstrel-like rate controller what drop probability we're after;
then you get the nice property that the transmission scheduler and rate
control are work-conserving and actually get faster when they get congested.
>
> There is also now the mathematics available to improve Minstrel, which is
about 80% of a truly optimal rate controller (interestingly, that math was
published in 2010, years after we did Minstrel; I had the intuition that it
was close to optimal but no way to do a proof).
>
> None of this is rocket science, but it is a pretty deep and complex
refactor of something that isn't the simplest code to start with.  That
said, I'd not expect it to be more than around 1k LoC, it's just going to
be a real bear to test.

I think the best path forward is to do proof of concept on the two open
enough drivers we have - the mt76 and ath9k.

A lot of what is needed to make rational decisions is simply not available
from closed firmwares.

> I do have the feeling that the people receiving this email ought to be
sufficient to fix the problem, if we can direct enough time to it; for
myself, I have to think quite hard as to whether this should be my 20%
project.

There needs to be focus and direct funding for quite a few people. Felix
for one. It would be nice to acquire half an engineer from each of multiple
companies making wireless products and chips in a linaro like fashion, also.

> On Sun, Feb 1, 2015 at 2:06 PM, Avery Pennarun <apenwarr@google.com>
wrote:
>>
>> I would argue that insofar as the bufferbloat project has made a
>> difference, it's because there was a very clear message and product:
>> - here's what sucks when you have bufferbloat
>> - here's how you can detect it
>> - here's how you can get rid of it
>> - by the way, here's which of your competitors are already beating you
at it.
>>
>> It turns out you don't need a standards org in order to push any of
>> the above things.  The IEEE exists to make sure things interop at the
>> MAC/PHY layer.  The IETF exists to make sure things interop at the
>> higher layers.  But bufferbloat isn't about interop, it's just a thing
>> that happens inside gateways, so it's not something you can really
>> write standards about.  It is something you can turn into a
>> competitive advantage (or disadvantage, if you're a straggler).
>>
>> ...but meanwhile, if we want to fix bufferbloat in wifi, nobody
>> actually knows how to do it, so we are still at steps 1 and 2.
>>
>> This is why we're funding Dave to continue work on netperf-wrappers.

I still don't have POs for the other 75% of me this year. It is comforting
to have got such a public  confirmation that I have at least this much
floor under me this year, tho.

>> Those diagrams of latency under load are pretty convincing.  The
>> diagrams of page load times under different levels of latency are even
>> more convincing.  First, we prove there's a problem and a way to
>> measure the problem.  Then hopefully more people will be interested in
>> solving it.

Well, a wall of shame might help, I guess. I have a ton of data on a dozen
products that is miserable to see and should be quite embarrassing for the
vendors.

I'd prefer to spend time trying to fix the darn problem more directly, but
I would expect this tool will be of use in tracking improvements.

>>
>> On Sat, Jan 31, 2015 at 4:51 PM,  <dpreed@reed.com> wrote:
>> > I think we need to create an Internet focused 802.11 working group that
>> > would be to the "OS wireless designers and IEEE 802.11 standards
groups" as
>> > the WHATML group was to W3C.
>> >
>> >
>> >
>> > W3C was clueless about the real world at the point WHATML was
created.  And
>> > WHATML was a "revenge of the real" against W3C - advancing a wide
variety of
>> > important practical innovations rather than attending endless standards
>> > meetings with people who were not focused on solving actually important
>> > problems.
>> >
>> >
>> >
>> > It took a bunch of work to get WHATML going, and it offended W3C, who
became
>> > unhelpful.  But the approach actually worked - we now have a Web that
really
>> > uses browser-side expressivity and that would never have happened if
W3C
>> > were left to its own devices.
>> >
>> >
>> >
>> > The WiFi consortium was an attempt to wrest control of pragmatic
direction
>> > from 802.11 and the proprietary-divergence folks at Qualcomm, Broadcom,
>> > Cisco, etc.  But it failed, because it became thieves on a raft, more
>> > focused on picking each others' pockets than on actually addressing
the big
>> > issues.
>> >
>> >
>> >
>> > Jim has seen this play out in the Linux community around X.  Though
there
>> > are lots of interests who would benefit by moving the engineering ball
>> > forward, everyone resists action because it means giving up the chance
at
>> > dominance, and the central group is far too weak to do anything beyond
>> > adjudicating the worst battles.
>> >
>> >
>> >
>> > When I say "we" I definitely include myself (though my time is limited
due
>> > to other commitments and the need to support my family), but I would
only
>> > play with people who actually are committed to making stuff happen -
which
>> > includes raising hell with the vendors if need be, but also effective
>> > engineering steps that can achieve quick adoption.
>> >
>> >
>> >
>> > Sadly, and I think it is manageable at the moment, there are moves out
there
>> > being made to get the FCC to "protect" WiFi from "interference".  The
>> > current one was Marriott, who requested the FCC for a rule to make it
legal
>> > to disrupt and block use of WiFi in people's rooms in their hotels,
except
>> > with their access points.  This also needs some technical defense.  I
>> > believe any issues with WiFi performance in actual Marriott hotels are
due
>> > to bufferbloat in their hotel-wide systems, just as the issues with
GoGo are
>> > the same.  But it's possible that queueing problems in their own WiFi
gear
>> > are bad as well.
>> >
>> >
>> >
>> > I mention this because it is related, and to the layperson, or
>> > non-radio-knowledgeable executive, indistinguishable.  It will take
away the
>> > incentive to actually fix the 802.11 implementations to be better
>> > performing, making the problem seem to be a "management" issue that
can be
>> > solved by making WiFi less interoperable and less flexible by rules,
rather
>> > than by engineering.
>> >
>> >
>> >
>> > However, solving the problems of hotspot networks and hotel networks
are
>> > definitely "real world" issues, and quite along the same lines you
mention,
>> > Dave.  FQ is almost certainly a big deal both in WiFi and in the
>> > distribution networks behind WiFi. Co-existence is also a big deal
>> > (RTS/CTS-like mechanisms can go a long way to remediate hidden-terminal
>> > disruption of the basic protocols). Roaming and scaling need work as
well.
>> >
>> >
>> >
>> > It would even be a good thing to invent pragmatic ways to provide "low
rate"
>> > subnets and "high rate" subnets that can coexist, so that
compatibility with
>> > ancient "b" networks need not be maintained on all nets, at great cost
-
>> > just send beacons at a high rate, so that the "b" NICs can't see
them....
>> > but you need pragmatic stack implementations.
>> >
>> >
>> >
>> > But the engineering is not the only challenge. The other challenge is
to
>> > take the initiative and get stuff deployed.  In the case of
bufferbloat, the
>> > grade currently is a "D" for deployments, maybe a "D-".  Beautiful
technical
>> > work, but the economic/business/political side of things has been poor.
>> > Look at how slow IETF has been to achieve anything (the perfect is
truly the
>> > enemy of the good, and Dave Clark's "rough consensus and working code"
has
>> > been replaced by technocratic malaise, and what appears to me to be a
class
>> > of people who love traveling the world to a floating cocktail party
without
>> > getting anything important done).
>> >
>> >
>> >
>> > The problem with communications is that you can't just ship a product
with a
>> > new "feature", because the innovation only works if widely adopted.
Since
>> > there is no "Linux Desktop" (and Linus hates the idea, to a large
extent)
>> > Linux can't be the sole carrier of the idea.  You pretty much need iOS
and
>> > Android both to buy in or to provide a path for easy third-party
upgrades.
>> > How do you do that?  Well, that's where the WHATML-type approach is
>> > necessary.
>> >
>> >
>> >
>> > I don't know if this can be achieved, and there are lots of details to
be
>> > worked out.  But I'll play.

I'll play.

>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Saturday, January 31, 2015 4:05pm, "Dave Taht" <dave.taht@gmail.com>
>> > said:
>> >
>> > I would like to have somehow assembled all the focused resources to
make a
>> > go at fixing wifi, or at least having a f2f with a bunch of people in
the
>> > late march timeframe. This message of mine to linux-wireless bounced
for
>> > some reason and I am off to log out for 10 days, so...
>> > see relevant netdev thread also for ore details.
>> >
>> > ---------- Forwarded message ----------
>> > From: Dave Taht <dave.taht@gmail.com>
>> > Date: Sat, Jan 31, 2015 at 12:29 PM
>> > Subject: Re: Throughput regression with `tcp: refine TSO autosizing`
>> > To: Arend van Spriel <arend@broadcom.com>
>> > Cc: linux-wireless <linux-wireless@vger.kernel.org>, Michal Kazior
>> > <michal.kazior@tieto.com>, Eyal Perry <eyalpe@dev.mellanox.co.il>,
Network
>> > Development <netdev@vger.kernel.org>, Eric Dumazet <
eric.dumazet@gmail.com>
>> >
>> >
>> > The wifi industry as a whole has vastly bigger problems than achieving
>> > 1500Mbits in a faraday cage on a single flow.
>> >
>> > I encourage you to try tests in netperf-wrapper that explicitly test
for
>> > latency under load, and in particular, the RTT_FAIR tests against 4 or
more
>> > stations on a single wifi AP. You will find the results very
depressing.
>> > Similarly, on your previous test series, a latency figure would have
been
>> > nice to have. I just did a talk at nznog, where I tested the local
wifi with
>> > less than ambits of throughput, and 3 seconds of latency, filmed here:
>> >
>> > https://plus.google.com/u/0/107942175615993706558/posts/CY8ew8MPnMt
>> >
>> > Do wish more folk were testing in the busy real world environments,
like
>> > coffee shops, cities... really, anywhere outside a faraday cage!
>> >
>> > I am not attending netconf - I was unable to raise funds to go, and the
>> > program committee wanted something "new",
>> >
>> > instead of the preso I gave the IEEE 802.11 working group back in
september.
>> > (
>> >
http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
>> > )
>> >
>> > I was very pleased with the results of that talk - the day after I
gave it,
>> > the phrase "test for latency" showed up in a bunch of 802.11ax (the
next
>> > generation after ac) documents. :) Still, we are stuck with the train
wreck
>> > that is 802.11ac glommed on top of 802.11n, glommed on top of 802.11g,
in
>> > terms of queue management, terrible uses of airtime, rate control and
other
>> > stuff. Aruba and Meraki, in particular took a big interest in what I'd
>> > outlined in the preso above (we have a half dozen less well baked
ideas -
>> > that's just the easy stuff that can be done to improve wifi).  I gave a
>> > followup at meraki but I don't think that's online.
>> >
>> > Felix (nbd) is on vacation right now, as I am I. In fact I am going
>> > somewhere for a week totally lacking internet access.
>> >
>> > Presently the plan, with what budget (none) we have and time (very
little)
>> > we have is to produce a pair of proof of concept implementations for
per tid
>> > queuing (see relevant commit by nbd),  leveraging the new minstrel
stats,
>> > the new minstrel-blues stuff, and an aggregation aware codel with a
>> > calculated target based on the most recently active stations, and a
bunch of
>> > the other stuff outlined above at IEEE.
>> >
>> > It is my hope that this will start to provide accurate back pressure
(or
>> > sufficient lack thereof for TSQ), to also improve throughput while
still
>> > retaining low latency. But it is a certainty that we will run into more
>> > cross layer issues that will be a pita to resolve.
>> >
>> > If we can put together a meet up around or during ELC in california in
>> > march?
>> >
>> > I am really not terribly optimistic on anything other than the 2
chipsets we
>> > can hack on (ath9k, mt76). Negotiations to get qualcomm to open up
their
>> > ath10k firmware have thus far failed, nor has a ath10k-lite got
anywhere.
>> > Perhaps broadcom would be willing to open up their firmware
sufficiently to
>> > build in a better API?
>> >
>> > A bit more below.
>> >
>> >
>> > On Jan 30, 2015 5:59 AM, "Arend van Spriel" <arend@broadcom.com> wrote:
>> >>
>> >> On 01/30/15 14:19, Eric Dumazet wrote:
>> >>>
>> >>> On Fri, 2015-01-30 at 11:29 +0100, Arend van Spriel wrote:
>> >>>
>> >>>> Hi Eric,
>> >>>>
>> >>>> Your suggestions are still based on the fact that you consider
wireless
>> >>>> networking to be similar to ethernet, but as Michal indicated there
are
>> >>>> some fundamental differences starting with CSMA/CD versus CSMA/CA.
Also
>> >>>> the medium conditions are far from comparable.
>> >
>> > The analogy i now use for it is that switched ethernet is generally
your
>> > classic "dumbbell"
>> >
>> > topology. Wifi is more like a "taxi-stand" topology. If you think
about how
>> > people
>> >
>> > queue up at a taxi stand (and sometimes agree to share a ride), the
inter
>> > arrival
>> >
>> > and departure times of a taxi stand make for a better mental model.
>> >
>> > Admittedly, I seem to spend a lot of time, waiting for taxies, thinking
>> > about
>> >
>> > wifi.
>> >
>> >>> There is no shielding so
>> >>>> it needs to deal with interference and dynamically drops the link
rate
>> >>>> so transmission of packets can take several milliseconds. Then with
11n
>> >>>> they came up with aggregation with sends up to 64 packets in a
single
>> >>>> transmit over the air at worst case 6.5 Mbps (if I am not
mistaken). The
>> >>>> parameter value for tcp_limit_output_bytes of 131072 means that it
>> >>>> allows queuing for about 1ms on a 1Gbps link, but I hope you can see
>> >>>> this is not realistic for dealing with all variances of the wireless
>> >>>> medium/standard. I suggested this as topic for the wireless
workshop in
>> >>>> Otawa [1], but I can not attend there. Still hope that there will be
>> >>>> some discussions to get more awareness.
>> >
>> > I have sometimes hoped that TSQ could be made more a function of the
>> >
>> > number of active flows exiting an interface, but eric tells me that's
>> > impossible.
>> >
>> > This is possibly another case where TSQ could use to be a callback
>> > function...
>> >
>> > but frankly I care not a whit about maximizing single flow tcp
throughput on
>> > wifi
>> >
>> > in a faraday cage.
>> >
>> >
>> >>>
>> >>> Ever heard about bufferbloat ?
>> >>
>> >>
>> >> Sure. I am trying to get awareness about that in our wireless
>> >> driver/firmware development teams. So bear with me.
>> >>
>> >>
>> >>> Have you read my suggestions and tried them ?
>> >>>
>> >>> You can adjust the limit per flow to pretty much you want. If you
need
>> >>> 64 packets, just do the math. If in 2018 you need 128 packets, do the
>> >>> math again.
>> >>>
>> >>> I am very well aware that wireless wants aggregation, thank you.
>> >
>> > I note that a lot of people testing this are getting it backwards.
Usually
>> > it is the AP that is sending lots and lots of big packets, where the
return
>> > path is predominately acks from the station.
>> >
>> > I am not a huge fan of stretch acks, but certainly a little bit of
thinning
>> > doesn't bother me on the return path there.
>> >
>> > Going the other way, particularly in a wifi world that insists on
treating
>> > every packet as sacred (which I don't agree with at all), thinning
acks can
>> > help, but single stream throughput is of interest only on benchmarks,
FQing
>> > as much as possible all the flows destined the station in each
aggregate
>> > masks loss and reduces the need to protect everything so much.
>> >
>> >>
>> >> Sorry if I offended you. I was just giving these as example combined
with
>> >> effective rate usable on the medium to say that the bandwidth is more
>> >> dynamic in wireless and as such need dynamic change of queue depth.
Now this
>> >> can be done by making the fraction size as used in your suggestion
adaptive
>> >> to these conditions.
>> >
>> > Well... see above. Maybe this technique will do more of the right
thing,
>> > but... go test.
>> >
>> >
>> >>
>> >>> 131072 bytes of queue on 40Gbit is not 1ms, but 26 usec of queueing,
and
>> >>> we get line rate nevertheless.
>> >>
>> >>
>> >> I was saying it was about 1ms on *1Gbit* as the wireless TCP rates are
>> >> moving into that direction in 11ac.
>> >>
>> >>
>> >>> We need this level of shallow queues (BQL, TSQ), to get very precise
rtt
>> >>> estimations so that TCP has good entropy for its pacing, even in the
50
>> >>> usec rtt ranges.
>> >>>
>> >>> If we allowed 1ms of queueing, then a 40Gbit flow would queue 5
MBytes.
>> >>>
>> >>> This was terrible, because it increased cwnd and all sender queues to
>> >>> insane levels.
>> >>
>> >>
>> >> Indeed and that is what we would like to address in our wireless
drivers.
>> >> I will setup some experiments using the fraction sizing and post my
>> >> findings. Again sorry if I offended you.
>> >
>> > You really, really, really need to test at rates below 50mbit and with
other
>> > stations, also while doing this. It's not going to be a linear curve.
>> >
>> >
>> >
>> >>
>> >> Regards,
>> >> Arend
>> >>
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> >
>> >
>> > --
>> > Dave Täht
>> >
>> > thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>
>

[-- Attachment #2: Type: text/html, Size: 27633 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 2975 bytes --]

Since this is going to be a big job, it's worth prioritising parts of it
appropriately.

Minstrel is probably already the single best feature of the Linux Wi-Fi
stack. AFAIK it still outperforms any other rate selector we know about. So
I don't consider improving it further to be a high priority, although that
trick of using it as a sneaky random packet loss inducer is intriguing.

Much more important and urgent is getting some form of functioning SQM
closer to the hardware, where the information is. I don't think we need to
get super fancy here to do some real good, in the same way that PIE is a
major improvement over drop-tail. I'd settle for a variant of fq_codel that
gets and uses information about whether the current packet request might be
aggregated with the previous packet provided, and adjusts its choice of
packet accordingly.

At the same time, models would undoubtedly be useful to help test and
repeatably demonstrate the advantages of both simple and more sophisticated
solutions. Ns3 allows laying out a reasonably complex radio environment,
which is great for this. To counter the prevalence of one-station Faraday
cage tests in the industry, the simulated environments should represent
realistic, challenging use cases:

1) the family home, with half a dozen client devices competing with several
interference sources (Bluetooth, RC toys, microwave oven, etc). This is a
relatively easy environment, representing the expected environment for
consumer equipment.

2) the apartment block, with fewer clients per AP but lots of APs
distributed throughout a large building. Walls and floors may provide
valuable attenuation here - unless you're in Japan, where they can be
notoriously thin.

3) the railway carriage, consisting of eighty passengers in a 20x3 m space,
and roughly the same number of client devices. The uplink is 3G based and
has some inherent latency. Add some Bluetooth for flavour, stir gently.
This one is rather challenging, but there is scope to optimise AP antenna
placement, and to scale the test down slightly by reducing seat occupancy.

4) the jumbo jet, consisting of several hundred passengers crammed in like
sardines. The uplink has satellite latencies built in. Good luck.

5) the business hotel. Multiple APs will be needed to provide adequate
coverage for this environment, which should encompass the rooms as well as
lounge, conference and dining areas. Some visitors may bring their own APs,
and the system must be able to cope with this without seriously degrading
performance.

6) the trade conference. A large arena filled with thousands of people.
Multiple APs required. Good luck.

I also feel that ultimately we're going to have to get industry on board.
Not just passively letting us play around as with ath9k, but actively
taking note of our findings and implementing at least a few of our ideas
themselves. Of course, tools, models and real-world results are likely to
make that easier.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 3211 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 7456 bytes --]


Just to clarify, managing queueing in a single access point WiFi network is only a small part of the problem of fixing the rapidly degrading performance of WiFi based systems.  Similarly, mesh routing is only a small part of the problem with the scalability of cooperative meshes based on the WiFi MAC.
 
So I don't disagree with work on queue management (which is not good in any commercial product).
 
But Dave T has done some great talks on "fixing WiFi" that don't have to do with queueing very much at all.
 
For example, rate selection for various packets is terrible.  When you have nearly 1000:1 ratios of transmission rates and codes that are not backward compatible, there's a huge opportunity for improvement.  Similarly, choice of frequency bandwidth and center frequency at each station offers huge opportunities for practical scalability of systems.  Also, as we noted earlier, "handoff" from one next hop to another is a huge problem with performance in practical deployments (a factor of 10x at least, just in that).
 
Propagation information is not used at all when 802.11 systems share a channel, even in single AP deployments, yet all stations can measure propagation quite accurately in their hardware.
 
Finally, Listen-before-talk is highly wasteful for two reasons: 1) any random radio noise from other sources unnecessarily degrades communications (and in the 5.8 MHz band, the rule about "radar avoidance" requires treating very low level noise as a "signal to shut the net down by law", but there is a loophole if you can tell that it's not actually "radar" (the technique requires two or more stations to measure the same noise event, and if the power is significantly different - more than a few dB - then it can't possibly be due to a distant transmitter, and therefore can be ignored). 2) the transmitter cannot tell when the intended receiver will be perfectly able to decode the signal without interference with the station it hears (this second point is actually proven in theory in a paper by Jon Peha that argued against trivial "etiquettes" as a mechanism for sharing among uncooperative and non-interoperable stations).
 
Dave T has discussed more, as have I in other venues.
 
The reason no one is making progress on any of these particular issues is that there is no coordination at the "systems level" around creating rising tides that lift all boats in the WiFi-ish space.  It's all about ripping the competition by creating stuff that can sell better than the other guys' stuff, and avoiding cooperation at all costs.
 
I agree that, to the extent that managing queues in a single box or a single operating system doesn't require cooperation, it's much easier to get such things into the market.  That's why CeroWRT has been as effective as it has been.  But has Microsoft done anything at all about it?   Do the better ECN signals that can arise from good queue management get used by the TCP endpoints, or for that matter UDP-based protocol endpoints?
 
But the big wins in making WiFi better are going begging.  As WiFi becomes more closed, as it will as the major Internet Access Providers and Gadget builders (Google, Apple) start excluding innovators in wireless from the market by closed, proprietary solutions, the problem WILL get worse.  You won't be able to fix those problems at all.  If you have a solution you will have to convince the oligopoly to even bother trying it.
 
So, let me reiterate.  The problem is not just "getting Minstrel adopted", though I have nothing against that as a subgoal.  The problem is to find good systems-level answers, and to find a strategy to deliver those answers to a WiFi ecology that spans the planet, and where the marketing value-story focuses on things one can measure between two stations in a Faraday cage, and never on any systems-level issues.
 
 
I personally think that things like promoting semi-closed, essentially proprietary ESSID-based bridged distribution systems as "good ideas" are counterproductive to this goal.  But that's perhaps too radical for this crowd.  It reminds me of Cisco's attempt to create a proprietary Internet technology with IOS, which fortunately was not the success Cisco hoped for, or Juniper would not have existed. Maybe IOS would have been a fine standard, but it would have killed the evolution of the Internet as we know it.

On Sunday, February 1, 2015 5:47am, "Jonathan Morton" <chromatix99@gmail.com> said:



Since this is going to be a big job, it's worth prioritising parts of it appropriately.
Minstrel is probably already the single best feature of the Linux Wi-Fi stack. AFAIK it still outperforms any other rate selector we know about. So I don't consider improving it further to be a high priority, although that trick of using it as a sneaky random packet loss inducer is intriguing.
Much more important and urgent is getting some form of functioning SQM closer to the hardware, where the information is. I don't think we need to get super fancy here to do some real good, in the same way that PIE is a major improvement over drop-tail. I'd settle for a variant of fq_codel that gets and uses information about whether the current packet request might be aggregated with the previous packet provided, and adjusts its choice of packet accordingly.
At the same time, models would undoubtedly be useful to help test and repeatably demonstrate the advantages of both simple and more sophisticated solutions. Ns3 allows laying out a reasonably complex radio environment, which is great for this. To counter the prevalence of one-station Faraday cage tests in the industry, the simulated environments should represent realistic, challenging use cases:
1) the family home, with half a dozen client devices competing with several interference sources (Bluetooth, RC toys, microwave oven, etc). This is a relatively easy environment, representing the expected environment for consumer equipment.
2) the apartment block, with fewer clients per AP but lots of APs distributed throughout a large building. Walls and floors may provide valuable attenuation here - unless you're in Japan, where they can be notoriously thin.
3) the railway carriage, consisting of eighty passengers in a 20x3 m space, and roughly the same number of client devices. The uplink is 3G based and has some inherent latency. Add some Bluetooth for flavour, stir gently. This one is rather challenging, but there is scope to optimise AP antenna placement, and to scale the test down slightly by reducing seat occupancy.
4) the jumbo jet, consisting of several hundred passengers crammed in like sardines. The uplink has satellite latencies built in. Good luck.
5) the business hotel. Multiple APs will be needed to provide adequate coverage for this environment, which should encompass the rooms as well as lounge, conference and dining areas. Some visitors may bring their own APs, and the system must be able to cope with this without seriously degrading performance.
6) the trade conference. A large arena filled with thousands of people. Multiple APs required. Good luck.
I also feel that ultimately we're going to have to get industry on board. Not just passively letting us play around as with ath9k, but actively taking note of our findings and implementing at least a few of our ideas themselves. Of course, tools, models and real-world results are likely to make that easier.
- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 11383 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Andrew McGregor]

[-- Attachment #1: Type: text/plain, Size: 9023 bytes --]

So far as I'm concerned, Minstrel is as widely adopted as we could want; it
could go further, but it doesn't need any help with that.  It could be
better, however, and that's a fairly straightforward change.  One of the
problems with rate selection is that the standard itself has a bit to say
on what rates to use for what packets, and what it says is braindead.  So
if you want to be standards compliant you are also leaving quite a bit of
potential performance on the table.

I missed one item in my list of potential improvements: the most braindead
thing 802.11 has to say about rates is that broadcast and multicast packets
should be sent at 'the lowest basic rate in the current supported rate
set', which is really wasteful.  There are a couple of ways of dealing with
this: one, ignore the standard and pick the rate that is most likely to get
the frame to as many neighbours as possible (by a scan of the Minstrel
tables).  Or two, fan it out as unicast, which might well take less airtime
(due to aggregation) as well as being much more likely to be delivered,
since you get ACKs and retries by doing that.

As for the industry... sure, there are those forces, but then again there's
a non-trivial amount of influence in this audience.

On Mon, Feb 2, 2015 at 1:43 AM, <dpreed@reed.com> wrote:

> Just to clarify, managing queueing in a single access point WiFi network
> is only a small part of the problem of fixing the rapidly degrading
> performance of WiFi based systems.  Similarly, mesh routing is only a small
> part of the problem with the scalability of cooperative meshes based on the
> WiFi MAC.
>
>
>
> So I don't disagree with work on queue management (which is not good in
> any commercial product).
>
>
>
> But Dave T has done some great talks on "fixing WiFi" that don't have to
> do with queueing very much at all.
>
>
>
> For example, rate selection for various packets is terrible.  When you
> have nearly 1000:1 ratios of transmission rates and codes that are not
> backward compatible, there's a huge opportunity for improvement.
> Similarly, choice of frequency bandwidth and center frequency at each
> station offers huge opportunities for practical scalability of systems.
> Also, as we noted earlier, "handoff" from one next hop to another is a huge
> problem with performance in practical deployments (a factor of 10x at
> least, just in that).
>
>
>
> Propagation information is not used at all when 802.11 systems share a
> channel, even in single AP deployments, yet all stations can measure
> propagation quite accurately in their hardware.
>
>
>
> Finally, Listen-before-talk is highly wasteful for two reasons: 1) any
> random radio noise from other sources unnecessarily degrades communications
> (and in the 5.8 MHz band, the rule about "radar avoidance" requires
> treating very low level noise as a "signal to shut the net down by law",
> but there is a loophole if you can tell that it's not actually "radar" (the
> technique requires two or more stations to measure the same noise event,
> and if the power is significantly different - more than a few dB - then it
> can't possibly be due to a distant transmitter, and therefore can be
> ignored). 2) the transmitter cannot tell when the intended receiver will be
> perfectly able to decode the signal without interference with the station
> it hears (this second point is actually proven in theory in a paper by Jon
> Peha that argued against trivial "etiquettes" as a mechanism for sharing
> among uncooperative and non-interoperable stations).
>
>
>
> Dave T has discussed more, as have I in other venues.
>
>
>
> The reason no one is making progress on any of these particular issues is
> that there is no coordination at the "systems level" around creating rising
> tides that lift all boats in the WiFi-ish space.  It's all about ripping
> the competition by creating stuff that can sell better than the other guys'
> stuff, and avoiding cooperation at all costs.
>
>
>
> I agree that, to the extent that managing queues in a single box or a
> single operating system doesn't require cooperation, it's much easier to
> get such things into the market.  That's why CeroWRT has been as effective
> as it has been.  But has Microsoft done anything at all about it?   Do the
> better ECN signals that can arise from good queue management get used by
> the TCP endpoints, or for that matter UDP-based protocol endpoints?
>
>
>
> But the big wins in making WiFi better are going begging.  As WiFi becomes
> more closed, as it will as the major Internet Access Providers and Gadget
> builders (Google, Apple) start excluding innovators in wireless from the
> market by closed, proprietary solutions, the problem WILL get worse.  You
> won't be able to fix those problems at all.  If you have a solution you
> will have to convince the oligopoly to even bother trying it.
>
>
>
> So, let me reiterate.  The problem is not just "getting Minstrel adopted",
> though I have nothing against that as a subgoal.  The problem is to find
> good systems-level answers, and to find a strategy to deliver those answers
> to a WiFi ecology that spans the planet, and where the marketing
> value-story focuses on things one can measure between two stations in a
> Faraday cage, and never on any systems-level issues.
>
>
>
>
>
> I personally think that things like promoting semi-closed, essentially
> proprietary ESSID-based bridged distribution systems as "good ideas" are
> counterproductive to this goal.  But that's perhaps too radical for this
> crowd.  It reminds me of Cisco's attempt to create a proprietary Internet
> technology with IOS, which fortunately was not the success Cisco hoped for,
> or Juniper would not have existed. Maybe IOS would have been a fine
> standard, but it would have killed the evolution of the Internet as we know
> it.
>
>
> On Sunday, February 1, 2015 5:47am, "Jonathan Morton" <
> chromatix99@gmail.com> said:
>
>  Since this is going to be a big job, it's worth prioritising parts of it
> appropriately.
>
> Minstrel is probably already the single best feature of the Linux Wi-Fi
> stack. AFAIK it still outperforms any other rate selector we know about. So
> I don't consider improving it further to be a high priority, although that
> trick of using it as a sneaky random packet loss inducer is intriguing.
>
> Much more important and urgent is getting some form of functioning SQM
> closer to the hardware, where the information is. I don't think we need to
> get super fancy here to do some real good, in the same way that PIE is a
> major improvement over drop-tail. I'd settle for a variant of fq_codel that
> gets and uses information about whether the current packet request might be
> aggregated with the previous packet provided, and adjusts its choice of
> packet accordingly.
>
> At the same time, models would undoubtedly be useful to help test and
> repeatably demonstrate the advantages of both simple and more sophisticated
> solutions. Ns3 allows laying out a reasonably complex radio environment,
> which is great for this. To counter the prevalence of one-station Faraday
> cage tests in the industry, the simulated environments should represent
> realistic, challenging use cases:
>
> 1) the family home, with half a dozen client devices competing with
> several interference sources (Bluetooth, RC toys, microwave oven, etc).
> This is a relatively easy environment, representing the expected
> environment for consumer equipment.
>
> 2) the apartment block, with fewer clients per AP but lots of APs
> distributed throughout a large building. Walls and floors may provide
> valuable attenuation here - unless you're in Japan, where they can be
> notoriously thin.
>
> 3) the railway carriage, consisting of eighty passengers in a 20x3 m
> space, and roughly the same number of client devices. The uplink is 3G
> based and has some inherent latency. Add some Bluetooth for flavour, stir
> gently. This one is rather challenging, but there is scope to optimise AP
> antenna placement, and to scale the test down slightly by reducing seat
> occupancy.
>
> 4) the jumbo jet, consisting of several hundred passengers crammed in like
> sardines. The uplink has satellite latencies built in. Good luck.
>
> 5) the business hotel. Multiple APs will be needed to provide adequate
> coverage for this environment, which should encompass the rooms as well as
> lounge, conference and dining areas. Some visitors may bring their own APs,
> and the system must be able to cope with this without seriously degrading
> performance.
>
> 6) the trade conference. A large arena filled with thousands of people.
> Multiple APs required. Good luck.
>
> I also feel that ultimately we're going to have to get industry on board.
> Not just passively letting us play around as with ath9k, but actively
> taking note of our findings and implementing at least a few of our ideas
> themselves. Of course, tools, models and real-world results are likely to
> make that easier.
>
> - Jonathan Morton
>

[-- Attachment #2: Type: text/html, Size: 12890 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Avery Pennarun]

On Sun, Feb 1, 2015 at 6:34 PM, Andrew McGregor <andrewmcgr@gmail.com> wrote:
> I missed one item in my list of potential improvements: the most braindead
> thing 802.11 has to say about rates is that broadcast and multicast packets
> should be sent at 'the lowest basic rate in the current supported rate set',
> which is really wasteful.  There are a couple of ways of dealing with this:
> one, ignore the standard and pick the rate that is most likely to get the
> frame to as many neighbours as possible (by a scan of the Minstrel tables).
> Or two, fan it out as unicast, which might well take less airtime (due to
> aggregation) as well as being much more likely to be delivered, since you
> get ACKs and retries by doing that.

As far as I can see, the only sensible thing to do with
multicast/broadcast is some variation of the unicast fanout, unless
you've got a truly huge number of nodes.  I don't know of any
protocols (certainly not video streams) that actually work well with
the kind of packet loss you see at medium/long range with wifi if
retransmits aren't used.  I've heard that openwrt already has a patch
included that does this kind of fanout at the bridge layer.

I've also heard of a new "reliable multicast" in some newer 802.11
variant, which essentially sends out a single multicast packet and
expects an ACK from each intended recipient.  Other than adding
complexity, it seems like the best of both worlds.

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Jim Gettys]

On Sun, Feb 1, 2015 at 11:04 PM, Avery Pennarun <apenwarr@google.com> wrote:
> On Sun, Feb 1, 2015 at 6:34 PM, Andrew McGregor <andrewmcgr@gmail.com> wrote:
>> I missed one item in my list of potential improvements: the most braindead
>> thing 802.11 has to say about rates is that broadcast and multicast packets
>> should be sent at 'the lowest basic rate in the current supported rate set',
>> which is really wasteful.  There are a couple of ways of dealing with this:
>> one, ignore the standard and pick the rate that is most likely to get the
>> frame to as many neighbours as possible (by a scan of the Minstrel tables).
>> Or two, fan it out as unicast, which might well take less airtime (due to
>> aggregation) as well as being much more likely to be delivered, since you
>> get ACKs and retries by doing that.
>
> As far as I can see, the only sensible thing to do with
> multicast/broadcast is some variation of the unicast fanout, unless
> you've got a truly huge number of nodes.  I don't know of any
> protocols (certainly not video streams) that actually work well with
> the kind of packet loss you see at medium/long range with wifi if
> retransmits aren't used.  I've heard that openwrt already has a patch
> included that does this kind of fanout at the bridge layer.

I gather some Windows drivers from some vendors do this unicast fanout
(claim made by one of their engineers in an early homenet meeting).

>
> I've also heard of a new "reliable multicast" in some newer 802.11
> variant, which essentially sends out a single multicast packet and
> expects an ACK from each intended recipient.  Other than adding
> complexity, it seems like the best of both worlds.

So long as it times out in some very small, finite time.  We don't
want a repeat of the infinite retry bugs Dave found in drivers a few
years back...

"Reliable multicast" ultimately is an oxymoron, particularly on a
medium with hundreds/one bandwidth variation.  One remote low
bandwidth station cannot be allowed to drag the entire network to the
basement.
                                     - Jim

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[Avery Pennarun]

On Sun, Feb 1, 2015 at 9:43 AM,  <dpreed@reed.com> wrote:
> Just to clarify, managing queueing in a single access point WiFi network is
> only a small part of the problem of fixing the rapidly degrading performance
> of WiFi based systems.

Can you explain what you mean by "rapidly degrading?"  The performance
in odd situations is certainly not inspirational, but I haven't
noticed it getting worse over time.

> Similarly, mesh routing is only a small part of the
> problem with the scalability of cooperative meshes based on the WiFi MAC.

That's certainly true.  Not to say the mesh routing algorithms are
much good either.

>  Also, as we noted
> earlier, "handoff" from one next hop to another is a huge problem with
> performance in practical deployments (a factor of 10x at least, just in
> that).

While there is definitely some work to be done in handoff, it seems
like there are some find implementations of this already in existence.
Several brands of "enterprise access point" setups seem to do well at
this.  It would be nice if they interoperated, I guess.

The fact that there's no open source version of this kind of handoff
feature bugs me, but we are working on it here and the work is all
planned to be open source, for example: (very early version)
https://gfiber.googlesource.com/vendor/google/platform/+/master/waveguide/

> Propagation information is not used at all when 802.11 systems share a
> channel, even in single AP deployments, yet all stations can measure
> propagation quite accurately in their hardware.

802.11k seems to provide for sharing this information.  But I'm not
clear what I should use it for. :)

> Finally, Listen-before-talk is highly wasteful for two reasons: 1) any
> random radio noise from other sources unnecessarily degrades communications [...]
> 2) the transmitter cannot tell when the intended receiver will be perfectly
> able to decode the signal without interference with the station it hears
> (this second point is actually proven in theory in a paper by Jon Peha that
> argued against trivial "etiquettes" as a mechanism for sharing among
> uncooperative and non-interoperable stations).

I've thought quite a bit about your point #2 above, but I don't know
which direction to pursue.  The idea is that sometimes "just shout
over the background noise" is a globally optimal solution, right?  The
question seems to be to figure out when that is true and when it
isn't.

> I agree that, to the extent that managing queues in a single box or a single
> operating system doesn't require cooperation, it's much easier to get such
> things into the market.  That's why CeroWRT has been as effective as it has
> been.  But has Microsoft done anything at all about it?   Do the better ECN
> signals that can arise from good queue management get used by the TCP
> endpoints, or for that matter UDP-based protocol endpoints?

If we don't know the answer to the questions, then that is itself the
problem.  It's a lot easier to say, hey, ChromeOS and MacOS have good
network performance but Microsoft has bad network performance, if it's
true and we have good reproducible tests to demonstrate that.

> The reason no one is making progress on any of these particular issues is
> that there is no coordination at the "systems level" around creating rising
> tides that lift all boats in the WiFi-ish space.  It's all about ripping the
> competition by creating stuff that can sell better than the other guys'
> stuff, and avoiding cooperation at all costs.
> [...]
> But the big wins in making WiFi better are going begging.  As WiFi becomes
> more closed, as it will as the major Internet Access Providers and Gadget
> builders (Google, Apple) start excluding innovators in wireless from the
> market by closed, proprietary solutions, the problem WILL get worse.  You
> won't be able to fix those problems at all.  If you have a solution you will
> have to convince the oligopoly to even bother trying it.

As someone who works at Google Fiber (which is both a gadget maker and
an ISP) and who pushes all day long for our wifi stuff to be open
source, I'm slightly offended to be lumped in with other vendors in
your story :)  I think the ChromeOS team (which insists on only open
source wifi drivers in all chromebooks) would feel similarly.  We are
lucky to have defined our competitive advantage as something other
than short-lived slight improvements in wifi that will soon be
wastefully duplicated by everyone else.

That said, I see what you mean about the general state of the
industry.  The way to fix it is the way Linux always fixes it: make
the open source version so much better that building a proprietary
one, just to gather a small incremental advantage, is a huge waste of
time and effort.  Work on minstrel and fq_codel go really far here.

> I personally think that things like promoting semi-closed, essentially
> proprietary ESSID-based bridged distribution systems as "good ideas" are
> counterproductive to this goal.  But that's perhaps too radical for this
> crowd.

Not sure what you mean here.  ESSID-based distribution systems seem
pretty well defined to me.  The only proprietary part is the
decision-making process for assisted roaming (ie. the "inter-AP
protocol") which is only an optional performance optimization.  There
really should be an open source version of this, and I'm in fact
feebly attempting to build one, but I don't feel like the world is
falling apart through not having it.  You can build a bridged
multi-BSS ESSID today with plain out-of-the-box hostapd.

Have fun,

Avery

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[David Lang]

On Sun, 1 Feb 2015, Avery Pennarun wrote:

> On Sun, Feb 1, 2015 at 9:43 AM,  <dpreed@reed.com> wrote:
>> I personally think that things like promoting semi-closed, essentially
>> proprietary ESSID-based bridged distribution systems as "good ideas" are
>> counterproductive to this goal.  But that's perhaps too radical for this
>> crowd.
>
> Not sure what you mean here.  ESSID-based distribution systems seem
> pretty well defined to me.  The only proprietary part is the
> decision-making process for assisted roaming (ie. the "inter-AP
> protocol") which is only an optional performance optimization.  There
> really should be an open source version of this, and I'm in fact
> feebly attempting to build one, but I don't feel like the world is
> falling apart through not having it.  You can build a bridged
> multi-BSS ESSID today with plain out-of-the-box hostapd.

I will be running a fully opensource bridged ESSID system at SCaLE this month. 
last year we had ~2500 people and devices with ~50 APs deployed, and it worked 
well. The only problem was that I needed to deploy a few more APs to cover some 
of the hallway areas more reliably.

There are tricks that the commercial systems pull that I can't currently 
duplicate with opensource tools. But as Avery says, they are optimizations, not 
something required for successful operation. It would be nice to get the 
assisted roaming portion available. But it's not required.

David Lang

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]


On Sunday, February 1, 2015 11:21pm, "Avery Pennarun" <apenwarr@google.com> said:
 

> On Sun, Feb 1, 2015 at 9:43 AM, <dpreed@reed.com> wrote:
> > Just to clarify, managing queueing in a single access point WiFi network is
> > only a small part of the problem of fixing the rapidly degrading performance
> > of WiFi based systems.
> 
> Can you explain what you mean by "rapidly degrading?" The performance
> in odd situations is certainly not inspirational, but I haven't
> noticed it getting worse over time.


I was being specific about the words "WiFi-based systems", and not WiFi itself.  An obvious example is GoGo, whose degradation is probably not specifically in the WiFi portion, but in the bidirectional bufferbloat I can measure whenever I fly (frequently!).  But I also visit many enterprise sites for my day job, and hotels.  I do a few measurements when I can - almost always encountering bufferbloat (severe) on the path to the public Internet (whether wireless or not), but worse, encountering severe bad behavior on the WiFi hop, remedied by using a wired connection. And then there are hotels, and trains.
 
What's worse is that typical "sales office" connections in my past employers are often quite poor.
 
Now the causes of degradation are apparently multifaceted.   Places that seem to be fine fall off a cliff, because they used to be overprovisioned relative to traffic load, but now are not.  Of course capacity sharing should slow everyone, but the cliff adds insult to injury.
 
I have observed that deploying a lot of wifi stations that operate in basic 802.11b mode really does affect the folks with better gear in the same channel a lot.   A small packet occupies a disproportionate amount of channel time if transmitted at 1 Mb/sec - and my theory (this is hard to measure informally, so I could be wrong) is that "cheap" devices compete equally for packet time, but consume much more than their share.  My sense is that channel time should be allocated fairly by the most basic MAC access decision - LBT and backoff, not opportunities to transmit.  This would eliminate the pokey puppy problem. There are pragmatic ways to achieve this without changing the basic MAC access decision, which is the firmest-of-ware.
 
Anyway, degradation of WiFi services at a particular location over time is the norm I observe.
 

[-- Attachment #2: Type: text/html, Size: 3828 bytes --]

Re: [Cerowrt-devel] Fwd: Throughput regression with `tcp: refine TSO autosizing`

[David Lang]

On Mon, 2 Feb 2015, dpreed@reed.com wrote:

> On Sunday, February 1, 2015 11:21pm, "Avery Pennarun" <apenwarr@google.com> said:
> 
>
>> On Sun, Feb 1, 2015 at 9:43 AM, <dpreed@reed.com> wrote:
>> > Just to clarify, managing queueing in a single access point WiFi network is
>> > only a small part of the problem of fixing the rapidly degrading performance
>> > of WiFi based systems.
>> 
>> Can you explain what you mean by "rapidly degrading?" The performance
>> in odd situations is certainly not inspirational, but I haven't
>> noticed it getting worse over time.
>
>
> I was being specific about the words "WiFi-based systems", and not WiFi 
> itself.  An obvious example is GoGo, whose degradation is probably not 
> specifically in the WiFi portion, but in the bidirectional bufferbloat I can 
> measure whenever I fly (frequently!).  But I also visit many enterprise sites 
> for my day job, and hotels.  I do a few measurements when I can - almost 
> always encountering bufferbloat (severe) on the path to the public Internet 
> (whether wireless or not), but worse, encountering severe bad behavior on the 
> WiFi hop, remedied by using a wired connection. And then there are hotels, and 
> trains.
> 
> What's worse is that typical "sales office" connections in my past employers 
> are often quite poor.

Too many people deploy a couple apple Airports and think the problem is solved 
(my current employer did the same thing). Part of the problem is that people 
then think that this is the best that can be done with consumer equipment, and 
that the only think they can do after that is to go spend $50K+ to buy an 
"Enterprise" solution, that is then so complex to deploy that they may sit for 
9+ months between when they arrive and when they are deployed (two employers of 
mine have fallen into this trap). The idea that you can actually get good 
service without spending a ton of money just isn't something that is out there. 
All the 'networking vendors' out there push the proprietary "Enterprise" 
solutions.

At SCaLE, we had one of these vendors come in one year and deploy their $10K/AP 
"Enterprise" solutions, and it failed miserably (and they didn't have any info 
or explination as to why).

Some of the best conference wifi I have seen (and here I will exclude the ones 
I've setup to be fair :-) have been setup with consumer grade equipment.

I drool over the capabilities of some of the commercial equipment, but the 
expertise in setting it up is far more important than the equipment used.

> Now the causes of degradation are apparently multifaceted.  Places that seem 
> to be fine fall off a cliff, because they used to be overprovisioned relative 
> to traffic load, but now are not.  Of course capacity sharing should slow 
> everyone, but the cliff adds insult to injury.

The cliff is a real problem. It's not obvious to people how the various features 
of WiFi interact to create the cliff, and it's not something that is easy to 
duplicate for lab/performance/installation testing.

> I have observed that deploying a lot of wifi stations that operate in basic 
> 802.11b mode really does affect the folks with better gear in the same channel 
> a lot.  A small packet occupies a disproportionate amount of channel time if 
> transmitted at 1 Mb/sec - and my theory (this is hard to measure informally, 
> so I could be wrong) is that "cheap" devices compete equally for packet time, 
> but consume much more than their share.  My sense is that channel time should 
> be allocated fairly by the most basic MAC access decision - LBT and backoff, 
> not opportunities to transmit.  This would eliminate the pokey puppy problem. 
> There are pragmatic ways to achieve this without changing the basic MAC access 
> decision, which is the firmest-of-ware.

It's even worse than you think, those long (time) packets sent at the slow rates 
are far more likely to have other things interefere with them (something that 
can't hear the transmission, but who's transmission will prevent the receiver 
from hearing it) and so they are going to get retransmitted more frequently than 
faster bitrate packets.

> Anyway, degradation of WiFi services at a particular location over time is the 
> norm I observe.

well, to be fair, any service that's not updated as usage grows degrades over 
time (including things in meatspace like bus service or freeway lanes).

Some of the worst hotel network service I have seen is at higher-end hotels that 
implemented network service early and outsourced the job. Some of them are now 
updating/replacing that service and getting much better. But between updates, 
the service is always going to degrade under increased use.

David Lang