Re: [Make-wifi-fast] [Codel] Using fq_codel with a WiFi uplink to the Internet

Re: [Make-wifi-fast] [Codel] Using fq_codel with a WiFi uplink to the Internet

[Dave Taht]

On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
> I have two questions about using fq_codel on an edge router when the
> Internet uplink is through point-to-point WiFi:
>
> Question #1: Is it still effective to run fq_codel on our edge router when I
> have a WiFi uplink to the Internet, instead of a cabled connection like
> ADSL? And related to that, is a high quality point-to-point WiFi connection
> indistinguishable from a cabled connection as far as fq_codel is concerned?

It has, until recent developments, been helpful but not as effective
as we'd like.

We have two sets of code in the process of being finalized that should
work better
for a reasonably fast wifi uplink.

blog.cerowrt.org/post/fq_codel_on_ath10k/

https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html

Ideally you want to be able to run it on both sides.


> Question #2: Assuming the answer to Question #1 is an overall "yes", is it
> then better to have a guaranteed speed from the ISP, instead of having a
> variable but potentially higher speed, so that I can control the queue and
> have fq_codel and HTB prioritization work effectively?
>
> Background: I manage the network for a camp / conference center that
> supports up to about 120 users (5-10 simultaneous, at times). For Internet,
> we've been using ADSL with 5 Mbit download, 0.4 Mbit upload (remote area, so
> it’s slow). The only thing that keeps it usable is running fq_codel on a
> transparent Linux bridge that sits between the LAN and ADSL modem. On this
> bridge, I restrict the upload and download rate to about 85-90% of maximum,
> and use fq_codel, plus some HTB prioritization rules. It’s extremely
> effective at providing usable latency, so kudos to the fq_codel algorithm
> and implementation. It looks like this:
>
> LAN <=> Linux bridge with fq_codel <=> ADSL Modem 0.4 / 5.0 Mbps <=> DSLAM …
>
> But now, we have a chance to improve our throughput problem by switching to
> a point-to-point WiFi uplink that could hit speeds of 30-40 Mbit symmetric
> (more on the speeds later). We have to decide on starting a contract with
> them. At the same time, I’ll be switching the bridge to a Ubiquiti
> EdgeRouter X, which has fq_codel in its kernel, but should have the same
> effect. It would look something like:
>
> LAN <=> Ubiquiti EdgeRouter X <=> WiFi Client (Mikrotik 802.11n MIMO 2x2)
> <=> WiFi AP from ISP …
>
> We already have a test setup in place. The link rate to the ISP's AP, as
> reported by Mikrotik's admin console, is currently 86.6 Mbps transmit and
> 144.4 Mbps receive, with CCQ (connection quality) at 64% transmit / 99%
> receive. First of all, I'll try to have them get that transmit CCQ up to 99%
> like the receive, to make sure it's a stable link. But I also know that the
> actual Internet throughput will be less than the link rates, and
> speedtest.net results are currently around 30-40 Mbps symmetric.

regrettably no matter how "constant" your provider claims to hold the
connection, in case of bad weather, especially, it is unlikely they
can do so.

>
> Moreover, it's WiFi, and that led to my Question #1. I know that latency and
> throughput can vary, and that there are more queues and more things
> happening with packet aggregation, etc that I don’t understand. Can this
> aspect of the variable latency, throughput and packet transmission
> characteristics of WiFi make fq_codel less effective when used in this way
> on an intermediate bridge or edge router, or does it more depend on the
> quality of the WiFi link, where a high quality point-to-point WiFi uplink to
> a good upstream network (there’s another unknown) is indistinguishable from
> a cabled connection?

Ideally the algorithm should run very close to the wifi mac layer as
per the urls I sent earlier.

>
> My Question #2 came from the fact that I have two options from the ISP:
>
> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation,
> meaning we could get 40 Mbit, but we could also get a lot less at times (8
> Mbit I assume), depending on other network load.
>
> Option B: We can get a guaranteed bandwidth, but it costs more, so the
> maximum throughput we can pay for would be less. We would probably choose
> around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a
> seasonal business.
>
> My feeling, assuming that the answer to Question #1 is "yes" and I can
> effectively use fq_codel with WiFi at all, is to go with Option B, the
> guaranteed bandwidth. That way, I could set fq_codel to a little less than
> this bandwidth, and hopefully manage buffer bloat and do HTB prioritization
> in the same way I do now. But it depends on the answers to my two questions,
> is fq_codel still effective when using a WiFi uplink in general, and if so,
> is it better to go with a guaranteed bandwidth.

Lacking control of both sides, I would go for the garunteed bandwidth
and try to control it on the ethernet router, or with control of one
side, rate limit inbound as per what you said and let outbound float
(when the new code lands)


>
> Thanks for any thoughts on this.
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 3626 bytes --]

> On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> wrote:
>> Question #1: Is it still effective to run fq_codel on our edge router when I
>> have a WiFi uplink to the Internet, instead of a cabled connection like
>> ADSL? And related to that, is a high quality point-to-point WiFi connection
>> indistinguishable from a cabled connection as far as fq_codel is concerned?
> 
> It has, until recent developments, been helpful but not as effective
> as we'd like.
> 
> We have two sets of code in the process of being finalized that should
> work better
> for a reasonably fast wifi uplink.
> 
> blog.cerowrt.org/post/fq_codel_on_ath10k/ <http://blog.cerowrt.org/post/fq_codel_on_ath10k/>
> 
> https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html <https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html>

That looks grand. If I ever see it working, I think I'll be as emotional as the OP of the ath10k article. That is, having spent some time setting up an fq_codel bridge for our camp, and getting blank stares when I talk excitedly about what I’m doing. And yet if I turn fq_codel off, I hear pretty quickly, “what’s wrong with the Internet?”

Do I have any chance of running fq_codel in the driver on a Mikrotik 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to test it. The camp will be off-season soon until next April for the snowy Czech winter, so it’s a good time for testing, as I also test our meshed OpenWRT APs.

Q: Would it also be useful to have fq_codel running on our APs? They are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.

I could add it now using “tc", but any level lower than that would require the driver support, obviously. My feeling is that the rate limiting on my Linux bridge puts the queues “mostly” there, and not in the APs or upstream devices.

>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation,
>> meaning we could get 40 Mbit, but we could also get a lot less at times (8
>> Mbit I assume), depending on other network load.
>> 
>> Option B: We can get a guaranteed bandwidth, but it costs more, so the
>> maximum throughput we can pay for would be less. We would probably choose
>> around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a
>> seasonal business.
>> 
>> My feeling, assuming that the answer to Question #1 is "yes" and I can
>> effectively use fq_codel with WiFi at all, is to go with Option B, the
>> guaranteed bandwidth. That way, I could set fq_codel to a little less than
>> this bandwidth, and hopefully manage buffer bloat and do HTB prioritization
>> in the same way I do now. But it depends on the answers to my two questions,
>> is fq_codel still effective when using a WiFi uplink in general, and if so,
>> is it better to go with a guaranteed bandwidth.
> 
> Lacking control of both sides, I would go for the garunteed bandwidth
> and try to control it on the ethernet router, or with control of one
> side, rate limit inbound as per what you said and let outbound float
> (when the new code lands)

Thanks for the feedback. I only also wonder now if I should really have a BQL driver, or whether it makes a difference in this case. And also, whether I should rather be doing fq_codel on the ethernet adapter that’s internal to my Mac Mini, rather than the external USB adapter I’m using (more details on that also in my previous reply to Loganaden).


[-- Attachment #2: Type: text/html, Size: 15043 bytes --]

Re: [Make-wifi-fast] Using fq_codel with a WiFi uplink to the Internet

[Toke Høiland-Jørgensen]

Phineas Gage <phineas919@gmail.com> writes:

>  On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>  On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>
>  Question #1: Is it still effective to run fq_codel on our edge router when I
>  have a WiFi uplink to the Internet, instead of a cabled connection like
>  ADSL? And related to that, is a high quality point-to-point WiFi connection
>  indistinguishable from a cabled connection as far as fq_codel is concerned?
>
>  It has, until recent developments, been helpful but not as effective
>  as we'd like.
>
>  We have two sets of code in the process of being finalized that should
>  work better
>  for a reasonably fast wifi uplink.
>
>  blog.cerowrt.org/post/fq_codel_on_ath10k/
>
>  https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html
>
> That looks grand. If I ever see it working, I think I'll be as
> emotional as the OP of the ath10k article. That is, having spent some
> time setting up an fq_codel bridge for our camp, and getting blank
> stares when I talk excitedly about what I’m doing. And yet if I turn
> fq_codel off, I hear pretty quickly, “what’s wrong with the Internet?”
>
> Do I have any chance of running fq_codel in the driver on a Mikrotik
> 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
> test it. The camp will be off-season soon until next April for the
> snowy Czech winter, so it’s a good time for testing, as I also test
> our meshed OpenWRT APs.

Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
current trunk, and you'll be using the FQ-CoDel'ed driver :)

> Q: Would it also be useful to have fq_codel running on our APs? They
> are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.

Most likely, yes. You may also want to include the patches that gives
you airtime fairness on those. Keeps slow stations from slowing everyone
else down. I have a git tree with those here:
https://kau.toke.dk/git/lede/ - it's slightly behind mainline LEDE, so
you may want to use that as a base. This is the critical file, in that
case:
https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch

> I could add it now using “tc", but any level lower than that would
> require the driver support, obviously. My feeling is that the rate
> limiting on my Linux bridge puts the queues “mostly” there, and not in
> the APs or upstream devices.

Depends on your traffic patterns, of course. But yeah, if all your
clients share the same uplink and that has more bandwidth than the
AP-to-WiFi link, then that is where the bottleneck would be. But a
client with bad reception can end up with an effective rate as low as
6.5 Mbps, so not always.

-Toke

Re: [Make-wifi-fast] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 3805 bytes --]

> On Sep 23, 2016, at 6:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> writes:
> 
>> On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>> 
>> Do I have any chance of running fq_codel in the driver on a Mikrotik
>> 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
>> test it. The camp will be off-season soon until next April for the
>> snowy Czech winter, so it’s a good time for testing, as I also test
>> our meshed OpenWRT APs.
> 
> Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
> current trunk, and you'll be using the FQ-CoDel'ed driver :)

I don’t know for sure, but the specs are so close to this working board (https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd <https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd>) that I bet so. Secondly, I have to find out if the ISP will allow it. They will probably be more likely to do so if the driver could run on RouterOS 6.34.3. I’m guessing that’s not a priority right now. :)

>> Q: Would it also be useful to have fq_codel running on our APs? They
>> are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.
> 
> Most likely, yes. You may also want to include the patches that gives
> you airtime fairness on those. Keeps slow stations from slowing everyone
> else down. I have a git tree with those here:
> https://kau.toke.dk/git/lede/ <https://kau.toke.dk/git/lede/> - it's slightly behind mainline LEDE, so
> you may want to use that as a base. This is the critical file, in that
> case:
> https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch <https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch>
> 
>> I could add it now using “tc", but any level lower than that would
>> require the driver support, obviously. My feeling is that the rate
>> limiting on my Linux bridge puts the queues “mostly” there, and not in
>> the APs or upstream devices.
> 
> Depends on your traffic patterns, of course. But yeah, if all your
> clients share the same uplink and that has more bandwidth than the
> AP-to-WiFi link, then that is where the bottleneck would be. But a
> client with bad reception can end up with an effective rate as low as
> 6.5 Mbps, so not always.

Well, if our uplink goes to 30 Mbps or more, I’ve got repeater nodes that connect to their gateways at around that rate and fluctuate, so we’re likely to be moving the bottleneck around the camp sometimes if our Internet rate goes up. And in this environment, I know for sure that there are clients connecting at rates well below 30 Mbps! If there were negative MCS indexes, we would be using those.

Right now, the OpenWRT release we run on the APs comes from Open Mesh. Unless I can convince them to build a driver with this patch, I’ll have to build and flash my own OpenWRT and give up the use of their online dashboard, upgrades and support. This is possible (https://wiki.openwrt.org/toh/openmesh/om2p <https://wiki.openwrt.org/toh/openmesh/om2p>). Moreover, I’m more likely to be able to do this on our APs than our point-to-point Internet uplink devices, since those are owned by the ISP.

Thanks so much for these pointers and your efforts. The airtime fairness patch also sounds fantastic. In the main season, there can be a lot of contention in our environment at times, like when it starts raining and everyone heads to their cabins to get online. I’d love to try this out and help you test, but will see if it will be feasible for us.


[-- Attachment #2: Type: text/html, Size: 18445 bytes --]

Re: [Make-wifi-fast] Using fq_codel with a WiFi uplink to the Internet

[Toke Høiland-Jørgensen]

Phineas Gage <phineas919@gmail.com> writes:

>  On Sep 23, 2016, at 6:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Phineas Gage <phineas919@gmail.com> writes:
>
>  On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>  On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>
>  Do I have any chance of running fq_codel in the driver on a Mikrotik
>  911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
>  test it. The camp will be off-season soon until next April for the
>  snowy Czech winter, so it’s a good time for testing, as I also test
>  our meshed OpenWRT APs.
>
>  Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
>  current trunk, and you'll be using the FQ-CoDel'ed driver :)
>
> I don’t know for sure, but the specs are so close to this working
> board (https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd) that I bet
> so. Secondly, I have to find out if the ISP will allow it. They will
> probably be more likely to do so if the driver could run on RouterOS
> 6.34.3. I’m guessing that’s not a priority right now. :)

Wikipedia thinks that is based on Linux 3.3.5. Which is ancient. So no.
But in principle it could be done...

>  Q: Would it also be useful to have fq_codel running on our APs? They
>  are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.
>
>  Most likely, yes. You may also want to include the patches that gives
>  you airtime fairness on those. Keeps slow stations from slowing everyone
>  else down. I have a git tree with those here:
>  https://kau.toke.dk/git/lede/ - it's slightly behind mainline LEDE, so
>  you may want to use that as a base. This is the critical file, in that
>  case:
>  https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch
>
>  I could add it now using “tc", but any level lower than that would
>  require the driver support, obviously. My feeling is that the rate
>  limiting on my Linux bridge puts the queues “mostly” there, and not in
>  the APs or upstream devices.
>
>  Depends on your traffic patterns, of course. But yeah, if all your
>  clients share the same uplink and that has more bandwidth than the
>  AP-to-WiFi link, then that is where the bottleneck would be. But a
>  client with bad reception can end up with an effective rate as low as
>  6.5 Mbps, so not always.
>
> Well, if our uplink goes to 30 Mbps or more, I’ve got repeater nodes
> that connect to their gateways at around that rate and fluctuate, so
> we’re likely to be moving the bottleneck around the camp sometimes if
> our Internet rate goes up. And in this environment, I know for sure
> that there are clients connecting at rates well below 30 Mbps! If
> there were negative MCS indexes, we would be using those.

Indeed. We still have a few kinks to work out to get CoDel to work well
at all bandwidths. But it's not a huge showstopper; what we have now is
still tons better than before.

> Right now, the OpenWRT release we run on the APs comes from Open Mesh.
> Unless I can convince them to build a driver with this patch, I’ll
> have to build and flash my own OpenWRT and give up the use of their
> online dashboard, upgrades and support. This is possible
> (https://wiki.openwrt.org/toh/openmesh/om2p). Moreover, I’m more
> likely to be able to do this on our APs than our point-to-point
> Internet uplink devices, since those are owned by the ISP.

Yeah, I know for a fact that the patches work on those devices; had them
tested on a bunch :)

> Thanks so much for these pointers and your efforts. The airtime
> fairness patch also sounds fantastic. In the main season, there can be
> a lot of contention in our environment at times, like when it starts
> raining and everyone heads to their cabins to get online. I’d love to
> try this out and help you test, but will see if it will be feasible
> for us.

You're very welcome. And testing is always appreciated :)

-Toke

Re: [Make-wifi-fast] [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 4550 bytes --]

> On Sep 23, 2016, at 1:54 PM, Phineas Gage <phineas919@gmail.com> wrote:
> 
>>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 Mbit, but we could also get a lot less at times (8 Mbit I assume), depending on other network load.
>> 
>> The 1:5 aggregation is worth exploring a bit more deeply.  Usually this refers to the ratio between the total bandwidth provisioned across all customers (in some region and some service class) and the minimum backhaul capacity within and at the far edge of the ISP’s network.  The theory is that customers tend not to use all of their bandwidth all of the time, though they do tend to use all of it some of the time.  This theory works fairly well in practice as long as the traffic patterns are not too correlated and are distributed over a sufficient number of customers.
>> 
>> In order to be dragged down to 8Mbps, you would have to see all the other customers sharing your backhaul trying to use 8Mbps or more (on average) at the same time.  This is unlikely to occur often; consider major sportsball championship final matches, or national disasters such as one we recently had an anniversary of, as the most likely triggers.  Under such circumstances, you’ll need to step in and manually experiment to find out what works, but shaping at 8Mbps would be a reasonable default reaction.
>> 
>> Of more concern to you is how much external load is needed to pull your share of the backhaul significantly below 40Mbps - or, alternatively, below the 20Mbps you can get with the more expensive uncontended service.  This will vary depending on how many customers the 5:1 average is spread over - you can’t actually calculate it from the information given.
>> 
>> So the question is whether they have a 40/40 backhaul shared among 5 customers, or a 1G/1G backhaul which they plan to share among up to 125 customers, each with 40/40 service.  I think the latter is a perfectly reasonable possibility, and would be much more robust than the former option which would give you a reduction in throughput as soon as *any* of the other customers on the same backhaul segment did *anything*.
>> 
>> It’s a question worth asking your WISP.  Be ready to point out that you don’t plan to actually *use* 40Mbps full-time, but that your AQM solution depends on knowing how much bandwidth is available for when *your* customers randomly demand it.
> 
> Thanks a lot for this great information, I’ll ask them more about what’s behind their aggregation ratio. As I mentioned in one reply earlier, I may ask for flexibility in our contract, try the aggregated service, then switch to guaranteed service depending on the actual performance of their network. If I can rate limit to 30/30 and fq_codel, and it’s 99% effective, I’d rather take that than settle for a guaranteed 20/20 on-season and 6/6 off-season to get to the same annual contract price.

Following up on the WISP results, I tested the 40 Mbps non-guaranteed connection for a week or so, and the truth is that the speed varies much more than expected. Although I once saw a 40 Mbps download, that’s more of a black swan. Download throughput during Internet rush hour regularly dropped to 5-10 Mbps, and I once saw 1 Mbps. Upload throughput stays higher. But I can’t see how “aggregation 1:5” has any meaning for me, as I saw speeds below 8 Mbps. There is probably no way I can effectively control either the upload or download queues effectively with this type of service.

Also, they seem to prioritize traffic to speedtest.net, because while the graphs to those servers looked pretty good and consistent (I scripted a test every 30 minutes using speedtest-cli), throughput reported by using wget to various high-powered Ubuntu servers around us in Europe told a different story. During the day, 25-30 Mbps was routine, and in the evening, 5-10 Mbps was routine, with outliers on either end. This was of course with a cabled client and no other traffic from us on the connection. So I’ll be finding out more details about the “guaranteed” 10 Mbps service.

Also a couple other comments to anyone interested:

- I saw a post about recent work getting Cake to run on EdgeOS, so I expressed interest in that for the EdgeRouter X, which I installed yesterday: http://community.ubnt.com/t5/EdgeMAX/Cake-compiled-for-the-ERL/td-p/1679844

- I might make it to the OpenWrt summit (http://openwrtsummit.org/) in Berlin next Thursday, 10/13, in case I catch anyone there.


[-- Attachment #2: Type: text/html, Size: 7201 bytes --]