[Codel] Using fq_codel with a WiFi uplink to the Internet

[Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 4090 bytes --]

I have two questions about using fq_codel on an edge router when the Internet uplink is through point-to-point WiFi:

Question #1: Is it still effective to run fq_codel on our edge router when I have a WiFi uplink to the Internet, instead of a cabled connection like ADSL? And related to that, is a high quality point-to-point WiFi connection indistinguishable from a cabled connection as far as fq_codel is concerned?

Question #2: Assuming the answer to Question #1 is an overall "yes", is it then better to have a guaranteed speed from the ISP, instead of having a variable but potentially higher speed, so that I can control the queue and have fq_codel and HTB prioritization work effectively?

Background: I manage the network for a camp / conference center that supports up to about 120 users (5-10 simultaneous, at times). For Internet, we've been using ADSL with 5 Mbit download, 0.4 Mbit upload (remote area, so it’s slow). The only thing that keeps it usable is running fq_codel on a transparent Linux bridge that sits between the LAN and ADSL modem. On this bridge, I restrict the upload and download rate to about 85-90% of maximum, and use fq_codel, plus some HTB prioritization rules. It’s extremely effective at providing usable latency, so kudos to the fq_codel algorithm and implementation. It looks like this:

LAN <=> Linux bridge with fq_codel <=> ADSL Modem 0.4 / 5.0 Mbps <=> DSLAM …

But now, we have a chance to improve our throughput problem by switching to a point-to-point WiFi uplink that could hit speeds of 30-40 Mbit symmetric (more on the speeds later). We have to decide on starting a contract with them. At the same time, I’ll be switching the bridge to a Ubiquiti EdgeRouter X, which has fq_codel in its kernel, but should have the same effect. It would look something like:

LAN <=> Ubiquiti EdgeRouter X <=> WiFi Client (Mikrotik 802.11n MIMO 2x2) <=> WiFi AP from ISP …

We already have a test setup in place. The link rate to the ISP's AP, as reported by Mikrotik's admin console, is currently 86.6 Mbps transmit and 144.4 Mbps receive, with CCQ (connection quality) at 64% transmit / 99% receive. First of all, I'll try to have them get that transmit CCQ up to 99% like the receive, to make sure it's a stable link. But I also know that the actual Internet throughput will be less than the link rates, and speedtest.net results are currently around 30-40 Mbps symmetric.

Moreover, it's WiFi, and that led to my Question #1. I know that latency and throughput can vary, and that there are more queues and more things happening with packet aggregation, etc that I don’t understand. Can this aspect of the variable latency, throughput and packet transmission characteristics of WiFi make fq_codel less effective when used in this way on an intermediate bridge or edge router, or does it more depend on the quality of the WiFi link, where a high quality point-to-point WiFi uplink to a good upstream network (there’s another unknown) is indistinguishable from a cabled connection?

My Question #2 came from the fact that I have two options from the ISP:

Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 Mbit, but we could also get a lot less at times (8 Mbit I assume), depending on other network load.

Option B: We can get a guaranteed bandwidth, but it costs more, so the maximum throughput we can pay for would be less. We would probably choose around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a seasonal business.

My feeling, assuming that the answer to Question #1 is "yes" and I can effectively use fq_codel with WiFi at all, is to go with Option B, the guaranteed bandwidth. That way, I could set fq_codel to a little less than this bandwidth, and hopefully manage buffer bloat and do HTB prioritization in the same way I do now. But it depends on the answers to my two questions, is fq_codel still effective when using a WiFi uplink in general, and if so, is it better to go with a guaranteed bandwidth.

Thanks for any thoughts on this.

[-- Attachment #2: Type: text/html, Size: 5062 bytes --]

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Loganaden Velvindron]

On Wed, Sep 21, 2016 at 1:59 PM, Phineas Gage <phineas919@gmail.com> wrote:
> I have two questions about using fq_codel on an edge router when the
> Internet uplink is through point-to-point WiFi:
>
> Question #1: Is it still effective to run fq_codel on our edge router when I
> have a WiFi uplink to the Internet, instead of a cabled connection like
> ADSL? And related to that, is a high quality point-to-point WiFi connection
> indistinguishable from a cabled connection as far as fq_codel is concerned?

Yes, it is effective to a small extent. However, I would highly
recommend that you look into make-wifi-fast, and start testing the
firmware that Toke uploaded.

See this: http://blog.cerowrt.org/post/crypto_fq_bug/

(Personally, I would still prefer cables for point to point, but with
the recent efforts of make-wifi-fast, this could change by next year)

>
> Question #2: Assuming the answer to Question #1 is an overall "yes", is it
> then better to have a guaranteed speed from the ISP, instead of having a
> variable but potentially higher speed, so that I can control the queue and
> have fq_codel and HTB prioritization work effectively?
>

Guaranteed speed, or at least minimum guaranteed speed for both upload
and download is a good idea. You don't have to login and tweak each
time.


> Background: I manage the network for a camp / conference center that
> supports up to about 120 users (5-10 simultaneous, at times). For Internet,
> we've been using ADSL with 5 Mbit download, 0.4 Mbit upload (remote area, so
> it’s slow). The only thing that keeps it usable is running fq_codel on a
> transparent Linux bridge that sits between the LAN and ADSL modem. On this
> bridge, I restrict the upload and download rate to about 85-90% of maximum,
> and use fq_codel, plus some HTB prioritization rules. It’s extremely
> effective at providing usable latency, so kudos to the fq_codel algorithm
> and implementation. It looks like this:
>
> LAN <=> Linux bridge with fq_codel <=> ADSL Modem 0.4 / 5.0 Mbps <=> DSLAM …

I have a similar configuration: LAN <=> fq_codel (tp-link archer c7 v2
with pppoe) <=> FTTH modem (bridge mode)

May I ask why put the ADSL modem in bridge mode, and let the fq_codel
box handle pppoe ?


>
> But now, we have a chance to improve our throughput problem by switching to
> a point-to-point WiFi uplink that could hit speeds of 30-40 Mbit symmetric
> (more on the speeds later). We have to decide on starting a contract with
> them. At the same time, I’ll be switching the bridge to a Ubiquiti
> EdgeRouter X, which has fq_codel in its kernel, but should have the same
> effect. It would look something like:
>
Does EdgeRouter X also implement BQL for its network drivers ?

> LAN <=> Ubiquiti EdgeRouter X <=> WiFi Client (Mikrotik 802.11n MIMO 2x2)
> <=> WiFi AP from ISP …
>
> We already have a test setup in place. The link rate to the ISP's AP, as
> reported by Mikrotik's admin console, is currently 86.6 Mbps transmit and
> 144.4 Mbps receive, with CCQ (connection quality) at 64% transmit / 99%
> receive. First of all, I'll try to have them get that transmit CCQ up to 99%
> like the receive, to make sure it's a stable link. But I also know that the
> actual Internet throughput will be less than the link rates, and
> speedtest.net results are currently around 30-40 Mbps symmetric.
>
> Moreover, it's WiFi, and that led to my Question #1. I know that latency and
> throughput can vary, and that there are more queues and more things
> happening with packet aggregation, etc that I don’t understand. Can this
> aspect of the variable latency, throughput and packet transmission
> characteristics of WiFi make fq_codel less effective when used in this way
> on an intermediate bridge or edge router, or does it more depend on the
> quality of the WiFi link, where a high quality point-to-point WiFi uplink to
> a good upstream network (there’s another unknown) is indistinguishable from
> a cabled connection?
>
> My Question #2 came from the fact that I have two options from the ISP:
>
> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation,
> meaning we could get 40 Mbit, but we could also get a lot less at times (8
> Mbit I assume), depending on other network load.
>
> Option B: We can get a guaranteed bandwidth, but it costs more, so the
> maximum throughput we can pay for would be less. We would probably choose
> around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a
> seasonal business.
>
> My feeling, assuming that the answer to Question #1 is "yes" and I can
> effectively use fq_codel with WiFi at all, is to go with Option B, the
> guaranteed bandwidth. That way, I could set fq_codel to a little less than
> this bandwidth, and hopefully manage buffer bloat and do HTB prioritization
> in the same way I do now. But it depends on the answers to my two questions,
> is fq_codel still effective when using a WiFi uplink in general, and if so,
> is it better to go with a guaranteed bandwidth.
>
> Thanks for any thoughts on this.
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Dave Taht]

On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
> I have two questions about using fq_codel on an edge router when the
> Internet uplink is through point-to-point WiFi:
>
> Question #1: Is it still effective to run fq_codel on our edge router when I
> have a WiFi uplink to the Internet, instead of a cabled connection like
> ADSL? And related to that, is a high quality point-to-point WiFi connection
> indistinguishable from a cabled connection as far as fq_codel is concerned?

It has, until recent developments, been helpful but not as effective
as we'd like.

We have two sets of code in the process of being finalized that should
work better
for a reasonably fast wifi uplink.

blog.cerowrt.org/post/fq_codel_on_ath10k/

https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html

Ideally you want to be able to run it on both sides.


> Question #2: Assuming the answer to Question #1 is an overall "yes", is it
> then better to have a guaranteed speed from the ISP, instead of having a
> variable but potentially higher speed, so that I can control the queue and
> have fq_codel and HTB prioritization work effectively?
>
> Background: I manage the network for a camp / conference center that
> supports up to about 120 users (5-10 simultaneous, at times). For Internet,
> we've been using ADSL with 5 Mbit download, 0.4 Mbit upload (remote area, so
> it’s slow). The only thing that keeps it usable is running fq_codel on a
> transparent Linux bridge that sits between the LAN and ADSL modem. On this
> bridge, I restrict the upload and download rate to about 85-90% of maximum,
> and use fq_codel, plus some HTB prioritization rules. It’s extremely
> effective at providing usable latency, so kudos to the fq_codel algorithm
> and implementation. It looks like this:
>
> LAN <=> Linux bridge with fq_codel <=> ADSL Modem 0.4 / 5.0 Mbps <=> DSLAM …
>
> But now, we have a chance to improve our throughput problem by switching to
> a point-to-point WiFi uplink that could hit speeds of 30-40 Mbit symmetric
> (more on the speeds later). We have to decide on starting a contract with
> them. At the same time, I’ll be switching the bridge to a Ubiquiti
> EdgeRouter X, which has fq_codel in its kernel, but should have the same
> effect. It would look something like:
>
> LAN <=> Ubiquiti EdgeRouter X <=> WiFi Client (Mikrotik 802.11n MIMO 2x2)
> <=> WiFi AP from ISP …
>
> We already have a test setup in place. The link rate to the ISP's AP, as
> reported by Mikrotik's admin console, is currently 86.6 Mbps transmit and
> 144.4 Mbps receive, with CCQ (connection quality) at 64% transmit / 99%
> receive. First of all, I'll try to have them get that transmit CCQ up to 99%
> like the receive, to make sure it's a stable link. But I also know that the
> actual Internet throughput will be less than the link rates, and
> speedtest.net results are currently around 30-40 Mbps symmetric.

regrettably no matter how "constant" your provider claims to hold the
connection, in case of bad weather, especially, it is unlikely they
can do so.

>
> Moreover, it's WiFi, and that led to my Question #1. I know that latency and
> throughput can vary, and that there are more queues and more things
> happening with packet aggregation, etc that I don’t understand. Can this
> aspect of the variable latency, throughput and packet transmission
> characteristics of WiFi make fq_codel less effective when used in this way
> on an intermediate bridge or edge router, or does it more depend on the
> quality of the WiFi link, where a high quality point-to-point WiFi uplink to
> a good upstream network (there’s another unknown) is indistinguishable from
> a cabled connection?

Ideally the algorithm should run very close to the wifi mac layer as
per the urls I sent earlier.

>
> My Question #2 came from the fact that I have two options from the ISP:
>
> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation,
> meaning we could get 40 Mbit, but we could also get a lot less at times (8
> Mbit I assume), depending on other network load.
>
> Option B: We can get a guaranteed bandwidth, but it costs more, so the
> maximum throughput we can pay for would be less. We would probably choose
> around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a
> seasonal business.
>
> My feeling, assuming that the answer to Question #1 is "yes" and I can
> effectively use fq_codel with WiFi at all, is to go with Option B, the
> guaranteed bandwidth. That way, I could set fq_codel to a little less than
> this bandwidth, and hopefully manage buffer bloat and do HTB prioritization
> in the same way I do now. But it depends on the answers to my two questions,
> is fq_codel still effective when using a WiFi uplink in general, and if so,
> is it better to go with a guaranteed bandwidth.

Lacking control of both sides, I would go for the garunteed bandwidth
and try to control it on the ethernet router, or with control of one
side, rate limit inbound as per what you said and let outbound float
(when the new code lands)


>
> Thanks for any thoughts on this.
>
> _______________________________________________
> Codel mailing list
> Codel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Jonathan Morton]

> On 21 Sep, 2016, at 12:59, Phineas Gage <phineas919@gmail.com> wrote:
> 
> Question #1: Is it still effective to run fq_codel on our edge router when I have a WiFi uplink to the Internet, instead of a cabled connection like ADSL? And related to that, is a high quality point-to-point WiFi connection indistinguishable from a cabled connection as far as fq_codel is concerned?
> 
> Question #2: Assuming the answer to Question #1 is an overall "yes", is it then better to have a guaranteed speed from the ISP, instead of having a variable but potentially higher speed, so that I can control the queue and have fq_codel and HTB prioritization work effectively?

This is actually a pretty interesting pair of questions.  We’ve just been doing quite a lot of work related to integrating fq_codel (or something very like it) into the Linux wifi stack, where it has more information about aggregation and other things, and can therefore make smarter queuing decisions.

The very best solution would be for your WISP to integrate this work into their hardware at each end of the link.  It may take a little more time until that can happen, as the code is very very fresh and still a little raw.

Until then, I think something like what you’re already doing should work well.  Normally, fq_codel interacts badly with high-performsnce wifi because it tends to distribute packets alternately to different stations, which tends to prevent effective aggregation, but this is not a concern for a point-to-point link where all your traffic goes to and from one station to another.

> The link rate to the ISP's AP, as reported by Mikrotik's admin console, is currently 86.6 Mbps transmit and 144.4 Mbps receive…

> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 Mbit, but we could also get a lot less at times (8 Mbit I assume), depending on other network load.

The raw link rates are sufficiently above the service provisioning that, bar significant changes in geography between you and the access point, antenna faults, or really bad weather, you can probably count on getting 40/40 reliably at the link level.

The 1:5 aggregation is worth exploring a bit more deeply.  Usually this refers to the ratio between the total bandwidth provisioned across all customers (in some region and some service class) and the minimum backhaul capacity within and at the far edge of the ISP’s network.  The theory is that customers tend not to use all of their bandwidth all of the time, though they do tend to use all of it some of the time.  This theory works fairly well in practice as long as the traffic patterns are not too correlated and are distributed over a sufficient number of customers.

In order to be dragged down to 8Mbps, you would have to see all the other customers sharing your backhaul trying to use 8Mbps or more (on average) at the same time.  This is unlikely to occur often; consider major sportsball championship final matches, or national disasters such as one we recently had an anniversary of, as the most likely triggers.  Under such circumstances, you’ll need to step in and manually experiment to find out what works, but shaping at 8Mbps would be a reasonable default reaction.

Of more concern to you is how much external load is needed to pull your share of the backhaul significantly below 40Mbps - or, alternatively, below the 20Mbps you can get with the more expensive uncontended service.  This will vary depending on how many customers the 5:1 average is spread over - you can’t actually calculate it from the information given.

So the question is whether they have a 40/40 backhaul shared among 5 customers, or a 1G/1G backhaul which they plan to share among up to 125 customers, each with 40/40 service.  I think the latter is a perfectly reasonable possibility, and would be much more robust than the former option which would give you a reduction in throughput as soon as *any* of the other customers on the same backhaul segment did *anything*.

It’s a question worth asking your WISP.  Be ready to point out that you don’t plan to actually *use* 40Mbps full-time, but that your AQM solution depends on knowing how much bandwidth is available for when *your* customers randomly demand it.

 - Jonathan Morton

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 5267 bytes --]


> On Sep 21, 2016, at 12:28 PM, Loganaden Velvindron <loganaden@gmail.com> wrote:
> On Wed, Sep 21, 2016 at 1:59 PM, Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> wrote:
>> I have two questions about using fq_codel on an edge router when the
>> Internet uplink is through point-to-point WiFi:
>> 
>> Question #1: Is it still effective to run fq_codel on our edge router when I
>> have a WiFi uplink to the Internet, instead of a cabled connection like
>> ADSL? And related to that, is a high quality point-to-point WiFi connection
>> indistinguishable from a cabled connection as far as fq_codel is concerned?
> Yes, it is effective to a small extent. However, I would highly
> recommend that you look into make-wifi-fast, and start testing the
> firmware that Toke uploaded.
> 
> See this: http://blog.cerowrt.org/post/crypto_fq_bug/ <http://blog.cerowrt.org/post/crypto_fq_bug/>

Thanks, this looks great. I’ll see if my WISP is willing to experiment on our point-to-point connection, but it would require driver support for an Atheros AR9300 on a Mikrotik 911-5HnD (firmware 3.30). That’s on our side. There’s another Mikrotik on the other side, but I don’t know its specs yet.

>> Question #2: Assuming the answer to Question #1 is an overall "yes", is it
>> then better to have a guaranteed speed from the ISP, instead of having a
>> variable but potentially higher speed, so that I can control the queue and
>> have fq_codel and HTB prioritization work effectively?
> Guaranteed speed, or at least minimum guaranteed speed for both upload
> and download is a good idea. You don't have to login and tweak each
> time.

Ok, that’s along the lines of what I’m thinking also. I may ask them for flexibility in the contract and try the aggregated connection to save money, then switch to guaranteed depending on the actual conditions on their network.

>> LAN <=> Linux bridge with fq_codel <=> ADSL Modem 0.4 / 5.0 Mbps <=> DSLAM …
> I have a similar configuration: LAN <=> fq_codel (tp-link archer c7 v2
> with pppoe) <=> FTTH modem (bridge mode)
> 
> May I ask why put the ADSL modem in bridge mode, and let the fq_codel
> box handle pope ?

This way, if the bridge ever fails, I can tell someone even if I’m off-site to just “plug in the red cable”, bypassing the bridge. The modem does PPPoE, DHCP and DNS caching. The bridge provides better DNS caching, an NTP server, and HTB+fq_codel, but if it’s replaced with a cable, the network continues working without any configuration changes.

Is there a benefit to running PPPoE on the bridge, other than possibly a bit faster PPPoE encapsulation?

>> But now, we have a chance to improve our throughput problem by switching to
>> a point-to-point WiFi uplink that could hit speeds of 30-40 Mbit symmetric
>> (more on the speeds later). We have to decide on starting a contract with
>> them. At the same time, I’ll be switching the bridge to a Ubiquiti
>> EdgeRouter X, which has fq_codel in its kernel, but should have the same
>> effect. It would look something like:
>> 
> Does EdgeRouter X also implement BQL for its network drivers ?

I hadn’t heard of BQL actually, so I read this: https://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers/ <https://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers/>. I don’t think I currently have BQL in my setup, and it’s still effective, maybe because I’m doing rate limiting, but that makes me wonder what I might be missing. Also, I started this thread for an update on BQL support in the EdgeRouter X:

https://community.ubnt.com/t5/EdgeMAX/EdgeRouter-X-BQL-support-for-ethernet/m-p/1684788#U1684788 <https://community.ubnt.com/t5/EdgeMAX/EdgeRouter-X-BQL-support-for-ethernet/m-p/1684788#U1684788>

If it’s as easy to add BQL (4-8 lines of source) as this says: https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/ <https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/>, I could probably do it. I added a small ioctl for CDDA support to a Linux sound card driver some years ago, so have _very_ basic kernel development experience.

Q1: In what cases does BQL help, and is it like do be more useful or higher or lower bandwidths, or does that make a difference?

Q2: Is there a way I can tell in Linux if a given net driver supports BQL without looking at the source?

For the bridge, I’m repurposing an old Mac Mini with 1.25GHz PPC and am using its internal adapter (a Sun GEM 100Mbit adapter using the sungem driver) on the inside of our network. The outside, which has the fq_codel applied to it, is an Apple USB 100Mbit adapter.

Q3: Is it better to run fq_codel on an internal PCI based network adapter, or a USB ethernet adapter, or is there no difference?

In case it matters, the kernel has CONFIG_HZ=250. Also, I try to turn off any offloads in the interfaces file (to be honest, I don’t know if it makes a difference with the USB ethernet driver):

# eth1 interface to cable modem
auto eth1
iface eth1 inet manual
	pre-up /sbin/ethtool --offload eth1 gso off sg off gro off
	post-up /sbin/ifconfig eth1 mtu 1492


[-- Attachment #2: Type: text/html, Size: 17913 bytes --]

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 3626 bytes --]

> On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> wrote:
>> Question #1: Is it still effective to run fq_codel on our edge router when I
>> have a WiFi uplink to the Internet, instead of a cabled connection like
>> ADSL? And related to that, is a high quality point-to-point WiFi connection
>> indistinguishable from a cabled connection as far as fq_codel is concerned?
> 
> It has, until recent developments, been helpful but not as effective
> as we'd like.
> 
> We have two sets of code in the process of being finalized that should
> work better
> for a reasonably fast wifi uplink.
> 
> blog.cerowrt.org/post/fq_codel_on_ath10k/ <http://blog.cerowrt.org/post/fq_codel_on_ath10k/>
> 
> https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html <https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html>

That looks grand. If I ever see it working, I think I'll be as emotional as the OP of the ath10k article. That is, having spent some time setting up an fq_codel bridge for our camp, and getting blank stares when I talk excitedly about what I’m doing. And yet if I turn fq_codel off, I hear pretty quickly, “what’s wrong with the Internet?”

Do I have any chance of running fq_codel in the driver on a Mikrotik 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to test it. The camp will be off-season soon until next April for the snowy Czech winter, so it’s a good time for testing, as I also test our meshed OpenWRT APs.

Q: Would it also be useful to have fq_codel running on our APs? They are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.

I could add it now using “tc", but any level lower than that would require the driver support, obviously. My feeling is that the rate limiting on my Linux bridge puts the queues “mostly” there, and not in the APs or upstream devices.

>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation,
>> meaning we could get 40 Mbit, but we could also get a lot less at times (8
>> Mbit I assume), depending on other network load.
>> 
>> Option B: We can get a guaranteed bandwidth, but it costs more, so the
>> maximum throughput we can pay for would be less. We would probably choose
>> around 6/6 Mbit off-season, and 20/20 Mbit on-season, as the camp is a
>> seasonal business.
>> 
>> My feeling, assuming that the answer to Question #1 is "yes" and I can
>> effectively use fq_codel with WiFi at all, is to go with Option B, the
>> guaranteed bandwidth. That way, I could set fq_codel to a little less than
>> this bandwidth, and hopefully manage buffer bloat and do HTB prioritization
>> in the same way I do now. But it depends on the answers to my two questions,
>> is fq_codel still effective when using a WiFi uplink in general, and if so,
>> is it better to go with a guaranteed bandwidth.
> 
> Lacking control of both sides, I would go for the garunteed bandwidth
> and try to control it on the ethernet router, or with control of one
> side, rate limit inbound as per what you said and let outbound float
> (when the new code lands)

Thanks for the feedback. I only also wonder now if I should really have a BQL driver, or whether it makes a difference in this case. And also, whether I should rather be doing fq_codel on the ethernet adapter that’s internal to my Mac Mini, rather than the external USB adapter I’m using (more details on that also in my previous reply to Loganaden).


[-- Attachment #2: Type: text/html, Size: 15043 bytes --]

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

> On Sep 21, 2016, at 1:38 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> On 21 Sep, 2016, at 12:59, Phineas Gage <phineas919@gmail.com> wrote:
>> 
>> Question #1: Is it still effective to run fq_codel on our edge router when I have a WiFi uplink to the Internet, instead of a cabled connection like ADSL? And related to that, is a high quality point-to-point WiFi connection indistinguishable from a cabled connection as far as fq_codel is concerned?
>> 
>> Question #2: Assuming the answer to Question #1 is an overall "yes", is it then better to have a guaranteed speed from the ISP, instead of having a variable but potentially higher speed, so that I can control the queue and have fq_codel and HTB prioritization work effectively?
> 
> This is actually a pretty interesting pair of questions.  We’ve just been doing quite a lot of work related to integrating fq_codel (or something very like it) into the Linux wifi stack, where it has more information about aggregation and other things, and can therefore make smarter queuing decisions.
> 
> The very best solution would be for your WISP to integrate this work into their hardware at each end of the link.  It may take a little more time until that can happen, as the code is very very fresh and still a little raw.
> 
> Until then, I think something like what you’re already doing should work well.  Normally, fq_codel interacts badly with high-performsnce wifi because it tends to distribute packets alternately to different stations, which tends to prevent effective aggregation, but this is not a concern for a point-to-point link where all your traffic goes to and from one station to another.

So, for example, if one runs fq_codel on a regular AP accessed by multiple stations (using “tc”, above the driver level), this is not necessarily a good idea? And once the driver work is done and the queues are managed at a lower level, will this no longer be the case?

>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 Mbit, but we could also get a lot less at times (8 Mbit I assume), depending on other network load.
> 
> The 1:5 aggregation is worth exploring a bit more deeply.  Usually this refers to the ratio between the total bandwidth provisioned across all customers (in some region and some service class) and the minimum backhaul capacity within and at the far edge of the ISP’s network.  The theory is that customers tend not to use all of their bandwidth all of the time, though they do tend to use all of it some of the time.  This theory works fairly well in practice as long as the traffic patterns are not too correlated and are distributed over a sufficient number of customers.
> 
> In order to be dragged down to 8Mbps, you would have to see all the other customers sharing your backhaul trying to use 8Mbps or more (on average) at the same time.  This is unlikely to occur often; consider major sportsball championship final matches, or national disasters such as one we recently had an anniversary of, as the most likely triggers.  Under such circumstances, you’ll need to step in and manually experiment to find out what works, but shaping at 8Mbps would be a reasonable default reaction.
> 
> Of more concern to you is how much external load is needed to pull your share of the backhaul significantly below 40Mbps - or, alternatively, below the 20Mbps you can get with the more expensive uncontended service.  This will vary depending on how many customers the 5:1 average is spread over - you can’t actually calculate it from the information given.
> 
> So the question is whether they have a 40/40 backhaul shared among 5 customers, or a 1G/1G backhaul which they plan to share among up to 125 customers, each with 40/40 service.  I think the latter is a perfectly reasonable possibility, and would be much more robust than the former option which would give you a reduction in throughput as soon as *any* of the other customers on the same backhaul segment did *anything*.
> 
> It’s a question worth asking your WISP.  Be ready to point out that you don’t plan to actually *use* 40Mbps full-time, but that your AQM solution depends on knowing how much bandwidth is available for when *your* customers randomly demand it.

Thanks a lot for this great information, I’ll ask them more about what’s behind their aggregation ratio. As I mentioned in one reply earlier, I may ask for flexibility in our contract, try the aggregated service, then switch to guaranteed service depending on the actual performance of their network. If I can rate limit to 30/30 and fq_codel, and it’s 99% effective, I’d rather take that than settle for a guaranteed 20/20 on-season and 6/6 off-season to get to the same annual contract price.

I see that this is a great place for helpful info about fq_codel. I probably should have “aggregated” (word of the day) my replies to reduce list traffic, so will do so next time if needed.

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Toke Høiland-Jørgensen]

Phineas Gage <phineas919@gmail.com> writes:

>  On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>  On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>
>  Question #1: Is it still effective to run fq_codel on our edge router when I
>  have a WiFi uplink to the Internet, instead of a cabled connection like
>  ADSL? And related to that, is a high quality point-to-point WiFi connection
>  indistinguishable from a cabled connection as far as fq_codel is concerned?
>
>  It has, until recent developments, been helpful but not as effective
>  as we'd like.
>
>  We have two sets of code in the process of being finalized that should
>  work better
>  for a reasonably fast wifi uplink.
>
>  blog.cerowrt.org/post/fq_codel_on_ath10k/
>
>  https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-anomaly-on-ath9k.html
>
> That looks grand. If I ever see it working, I think I'll be as
> emotional as the OP of the ath10k article. That is, having spent some
> time setting up an fq_codel bridge for our camp, and getting blank
> stares when I talk excitedly about what I’m doing. And yet if I turn
> fq_codel off, I hear pretty quickly, “what’s wrong with the Internet?”
>
> Do I have any chance of running fq_codel in the driver on a Mikrotik
> 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
> test it. The camp will be off-season soon until next April for the
> snowy Czech winter, so it’s a good time for testing, as I also test
> our meshed OpenWRT APs.

Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
current trunk, and you'll be using the FQ-CoDel'ed driver :)

> Q: Would it also be useful to have fq_codel running on our APs? They
> are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.

Most likely, yes. You may also want to include the patches that gives
you airtime fairness on those. Keeps slow stations from slowing everyone
else down. I have a git tree with those here:
https://kau.toke.dk/git/lede/ - it's slightly behind mainline LEDE, so
you may want to use that as a base. This is the critical file, in that
case:
https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch

> I could add it now using “tc", but any level lower than that would
> require the driver support, obviously. My feeling is that the rate
> limiting on my Linux bridge puts the queues “mostly” there, and not in
> the APs or upstream devices.

Depends on your traffic patterns, of course. But yeah, if all your
clients share the same uplink and that has more bandwidth than the
AP-to-WiFi link, then that is where the bottleneck would be. But a
client with bad reception can end up with an effective rate as low as
6.5 Mbps, so not always.

-Toke

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 3805 bytes --]

> On Sep 23, 2016, at 6:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> writes:
> 
>> On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>> 
>> Do I have any chance of running fq_codel in the driver on a Mikrotik
>> 911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
>> test it. The camp will be off-season soon until next April for the
>> snowy Czech winter, so it’s a good time for testing, as I also test
>> our meshed OpenWRT APs.
> 
> Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
> current trunk, and you'll be using the FQ-CoDel'ed driver :)

I don’t know for sure, but the specs are so close to this working board (https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd <https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd>) that I bet so. Secondly, I have to find out if the ISP will allow it. They will probably be more likely to do so if the driver could run on RouterOS 6.34.3. I’m guessing that’s not a priority right now. :)

>> Q: Would it also be useful to have fq_codel running on our APs? They
>> are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.
> 
> Most likely, yes. You may also want to include the patches that gives
> you airtime fairness on those. Keeps slow stations from slowing everyone
> else down. I have a git tree with those here:
> https://kau.toke.dk/git/lede/ <https://kau.toke.dk/git/lede/> - it's slightly behind mainline LEDE, so
> you may want to use that as a base. This is the critical file, in that
> case:
> https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch <https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch>
> 
>> I could add it now using “tc", but any level lower than that would
>> require the driver support, obviously. My feeling is that the rate
>> limiting on my Linux bridge puts the queues “mostly” there, and not in
>> the APs or upstream devices.
> 
> Depends on your traffic patterns, of course. But yeah, if all your
> clients share the same uplink and that has more bandwidth than the
> AP-to-WiFi link, then that is where the bottleneck would be. But a
> client with bad reception can end up with an effective rate as low as
> 6.5 Mbps, so not always.

Well, if our uplink goes to 30 Mbps or more, I’ve got repeater nodes that connect to their gateways at around that rate and fluctuate, so we’re likely to be moving the bottleneck around the camp sometimes if our Internet rate goes up. And in this environment, I know for sure that there are clients connecting at rates well below 30 Mbps! If there were negative MCS indexes, we would be using those.

Right now, the OpenWRT release we run on the APs comes from Open Mesh. Unless I can convince them to build a driver with this patch, I’ll have to build and flash my own OpenWRT and give up the use of their online dashboard, upgrades and support. This is possible (https://wiki.openwrt.org/toh/openmesh/om2p <https://wiki.openwrt.org/toh/openmesh/om2p>). Moreover, I’m more likely to be able to do this on our APs than our point-to-point Internet uplink devices, since those are owned by the ISP.

Thanks so much for these pointers and your efforts. The airtime fairness patch also sounds fantastic. In the main season, there can be a lot of contention in our environment at times, like when it starts raining and everyone heads to their cabins to get online. I’d love to try this out and help you test, but will see if it will be feasible for us.


[-- Attachment #2: Type: text/html, Size: 18445 bytes --]

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Toke Høiland-Jørgensen]

Phineas Gage <phineas919@gmail.com> writes:

>  On Sep 23, 2016, at 6:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Phineas Gage <phineas919@gmail.com> writes:
>
>  On Sep 21, 2016, at 12:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>  On Wed, Sep 21, 2016 at 2:59 AM, Phineas Gage <phineas919@gmail.com> wrote:
>
>  Do I have any chance of running fq_codel in the driver on a Mikrotik
>  911-5HnD (firmware 3.30) with Atheros AR9300? If so, I may be able to
>  test it. The camp will be off-season soon until next April for the
>  snowy Czech winter, so it’s a good time for testing, as I also test
>  our meshed OpenWRT APs.
>
>  Can it run LEDE (OpenWrt)? If so, all you need to do is upgrade to
>  current trunk, and you'll be using the FQ-CoDel'ed driver :)
>
> I don’t know for sure, but the specs are so close to this working
> board (https://wiki.openwrt.org/toh/mikrotik/rb91xg_5hpnd) that I bet
> so. Secondly, I have to find out if the ISP will allow it. They will
> probably be more likely to do so if the driver could run on RouterOS
> 6.34.3. I’m guessing that’s not a priority right now. :)

Wikipedia thinks that is based on Linux 3.3.5. Which is ancient. So no.
But in principle it could be done...

>  Q: Would it also be useful to have fq_codel running on our APs? They
>  are Open Mesh OM2P HS’s with "Atheros AR9341 rev 1” chips.
>
>  Most likely, yes. You may also want to include the patches that gives
>  you airtime fairness on those. Keeps slow stations from slowing everyone
>  else down. I have a git tree with those here:
>  https://kau.toke.dk/git/lede/ - it's slightly behind mainline LEDE, so
>  you may want to use that as a base. This is the critical file, in that
>  case:
>  https://kau.toke.dk/git/lede/tree/package/kernel/mac80211/patches/347-ath9k-Add-a-per-station-airtime-deficit-scheduler.patch
>
>  I could add it now using “tc", but any level lower than that would
>  require the driver support, obviously. My feeling is that the rate
>  limiting on my Linux bridge puts the queues “mostly” there, and not in
>  the APs or upstream devices.
>
>  Depends on your traffic patterns, of course. But yeah, if all your
>  clients share the same uplink and that has more bandwidth than the
>  AP-to-WiFi link, then that is where the bottleneck would be. But a
>  client with bad reception can end up with an effective rate as low as
>  6.5 Mbps, so not always.
>
> Well, if our uplink goes to 30 Mbps or more, I’ve got repeater nodes
> that connect to their gateways at around that rate and fluctuate, so
> we’re likely to be moving the bottleneck around the camp sometimes if
> our Internet rate goes up. And in this environment, I know for sure
> that there are clients connecting at rates well below 30 Mbps! If
> there were negative MCS indexes, we would be using those.

Indeed. We still have a few kinks to work out to get CoDel to work well
at all bandwidths. But it's not a huge showstopper; what we have now is
still tons better than before.

> Right now, the OpenWRT release we run on the APs comes from Open Mesh.
> Unless I can convince them to build a driver with this patch, I’ll
> have to build and flash my own OpenWRT and give up the use of their
> online dashboard, upgrades and support. This is possible
> (https://wiki.openwrt.org/toh/openmesh/om2p). Moreover, I’m more
> likely to be able to do this on our APs than our point-to-point
> Internet uplink devices, since those are owned by the ISP.

Yeah, I know for a fact that the patches work on those devices; had them
tested on a bunch :)

> Thanks so much for these pointers and your efforts. The airtime
> fairness patch also sounds fantastic. In the main season, there can be
> a lot of contention in our environment at times, like when it starts
> raining and everyone heads to their cabins to get online. I’d love to
> try this out and help you test, but will see if it will be feasible
> for us.

You're very welcome. And testing is always appreciated :)

-Toke

Re: [Codel] Using fq_codel with a WiFi uplink to the Internet

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 4550 bytes --]

> On Sep 23, 2016, at 1:54 PM, Phineas Gage <phineas919@gmail.com> wrote:
> 
>>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 Mbit, but we could also get a lot less at times (8 Mbit I assume), depending on other network load.
>> 
>> The 1:5 aggregation is worth exploring a bit more deeply.  Usually this refers to the ratio between the total bandwidth provisioned across all customers (in some region and some service class) and the minimum backhaul capacity within and at the far edge of the ISP’s network.  The theory is that customers tend not to use all of their bandwidth all of the time, though they do tend to use all of it some of the time.  This theory works fairly well in practice as long as the traffic patterns are not too correlated and are distributed over a sufficient number of customers.
>> 
>> In order to be dragged down to 8Mbps, you would have to see all the other customers sharing your backhaul trying to use 8Mbps or more (on average) at the same time.  This is unlikely to occur often; consider major sportsball championship final matches, or national disasters such as one we recently had an anniversary of, as the most likely triggers.  Under such circumstances, you’ll need to step in and manually experiment to find out what works, but shaping at 8Mbps would be a reasonable default reaction.
>> 
>> Of more concern to you is how much external load is needed to pull your share of the backhaul significantly below 40Mbps - or, alternatively, below the 20Mbps you can get with the more expensive uncontended service.  This will vary depending on how many customers the 5:1 average is spread over - you can’t actually calculate it from the information given.
>> 
>> So the question is whether they have a 40/40 backhaul shared among 5 customers, or a 1G/1G backhaul which they plan to share among up to 125 customers, each with 40/40 service.  I think the latter is a perfectly reasonable possibility, and would be much more robust than the former option which would give you a reduction in throughput as soon as *any* of the other customers on the same backhaul segment did *anything*.
>> 
>> It’s a question worth asking your WISP.  Be ready to point out that you don’t plan to actually *use* 40Mbps full-time, but that your AQM solution depends on knowing how much bandwidth is available for when *your* customers randomly demand it.
> 
> Thanks a lot for this great information, I’ll ask them more about what’s behind their aggregation ratio. As I mentioned in one reply earlier, I may ask for flexibility in our contract, try the aggregated service, then switch to guaranteed service depending on the actual performance of their network. If I can rate limit to 30/30 and fq_codel, and it’s 99% effective, I’d rather take that than settle for a guaranteed 20/20 on-season and 6/6 off-season to get to the same annual contract price.

Following up on the WISP results, I tested the 40 Mbps non-guaranteed connection for a week or so, and the truth is that the speed varies much more than expected. Although I once saw a 40 Mbps download, that’s more of a black swan. Download throughput during Internet rush hour regularly dropped to 5-10 Mbps, and I once saw 1 Mbps. Upload throughput stays higher. But I can’t see how “aggregation 1:5” has any meaning for me, as I saw speeds below 8 Mbps. There is probably no way I can effectively control either the upload or download queues effectively with this type of service.

Also, they seem to prioritize traffic to speedtest.net, because while the graphs to those servers looked pretty good and consistent (I scripted a test every 30 minutes using speedtest-cli), throughput reported by using wget to various high-powered Ubuntu servers around us in Europe told a different story. During the day, 25-30 Mbps was routine, and in the evening, 5-10 Mbps was routine, with outliers on either end. This was of course with a cabled client and no other traffic from us on the connection. So I’ll be finding out more details about the “guaranteed” 10 Mbps service.

Also a couple other comments to anyone interested:

- I saw a post about recent work getting Cake to run on EdgeOS, so I expressed interest in that for the EdgeRouter X, which I installed yesterday: http://community.ubnt.com/t5/EdgeMAX/Cake-compiled-for-the-ERL/td-p/1679844

- I might make it to the OpenWrt summit (http://openwrtsummit.org/) in Berlin next Thursday, 10/13, in case I catch anyone there.


[-- Attachment #2: Type: text/html, Size: 7201 bytes --]