[Codel] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Codel] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 4926 bytes --]

Given the half-duplex nature of 802.11 WiFi, is it possible to use fq_codel with software rate limiting on separate hardware from the WiFi radio, while still allowing at or near the full WiFi link rate?

Note #1: This is probably more of a job for the new "Make WiFi Fast" code in LEDE, but so far I can’t run that in the environment I’m working in (more below).

Note #2: It also makes me think, I wonder how this problem is handled in the new Make WiFi Fast code, as that could be another big advantage for it, if the "half duplex rate limiting” I’m thinking of can’t be done in Linux (more below).

Note #3: This question may not be specific to fq_codel, or even WiFi, as it could apply to any half duplex physical media, with any queueing discipline, but this list seems more active than most. :)

How I got here: I arrived at this question because I started doing Flent RRUL runs over a local WiFi link and fq_codel was struggling to get latency down. I was using HTB to rate limit both input and output to 85 Mbps, which is a little less than the WiFi’s one-way throughput as reported by iperf3 (95 Mbps). Then it dawned on me that 802.11 WiFi is half duplex, so I need to limit the _combined_ rate. And indeed, simultaneous iperf3 throughput tests in both directions add up to a total combined rate of around 95 Mbps. I set the HTB input and output rate to 40 Mbps each, and average latency under load during an RRUL test dropped right away from ~30ms to ~5ms, and looked smooth, so I knew I was finally in control of the queue. 

New Problem: But now that my latency is reduced, I have a new problem. I’m rate limiting to around half my possible link rate in each direction, so if I’m only doing a download or upload on the link, I’m losing around half my potential throughput! That led to my question. Is it even possible to gain this throughput back without losing control of the queue? I would need the queueing in Linux to be aware that the physical media is half duplex, and rate limit the _total_ input and output traffic, not limit the input and output individually. Let’s call it “half duplex rate limiting.” AFAIK there’s no way to do this with IFB or Linux traffic control in general. Is there?

Background: I’ll back up a bit and add some background on what I’m doing, for those interested. I’m offering to help my community WISP try to reduce latency in their network by deploying fq_codel where possible. For those who saw my earlier WiFi related post, that was unrelated and on a different site, where fq_codel on the edge router did improve latency there. For this WISP however, since we can’t practically install and manage edge routers with fq_codel at every end user site, I’m doing some research to find out if fq_codel can at least help on the internal point-to-point backhaul links that connect their nodes. Currently, sfq is used for those. Unlike a typical edge router, which limits to “Internet speed”, these backhaul links usually operate at or near the maximum link rate of the WiFi hardware. So it’s not like a typical CPE device whose link rate might be well above the observed Internet rate, making the “half duplex problem” irrelevant, if that makes sense.

The WISP serves around 800 members, and is built with a combination of fiber and point-to-point WiFi links of various speeds and frequencies. Here’s a map:

http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=k&geolocate=98%7C114%7C111%7C117%7C109%7C111%7C118%7C115%7C107%7C97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1& <http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=k&geolocate=98|114|111|117|109|111|118|115|107|97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&>

Most of the nodes use Ubiquiti hardware for the radios and custom hardware for the routers, connected to the radios via Ethernet, and most of which run on 1 GHz AMD G-T40E processors and run a customized distribution of Voyage Linux 0.10 (Debian 8 based distribution for embedded hardware). The distro ships with kernel 3.16.7, which does have a working sch_fq_codel module. To get oriented, I started with some Flent RRUL runs on my test setup below, and that’s where I ran into this question. Next, I’ll test on some actual hardware targeted for deployment to understand its fq_codel throughput limits, but not before I understand this “half duplex problem” better.

Test setup:

PC #1  <->  WiFi (802.11n, 130 Mbps Tx Rate, RSSI -60)  <->  AirPort Express  <->  100Mbit Ethernet  <->  PC #2

PC #1: Runs a Voyage Linux VM (more on that later), with the Flent client and HTB + fq_codel, and is connected via the PC's WiFi adapter to the Airport Express.

PC #2: Runs netserver for flent to connect to, and is connected via Ethernet to the Airport Express.

Thanks for any input.

[-- Attachment #2: Type: text/html, Size: 6060 bytes --]

Re: [Codel] [Make-wifi-fast] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Jonathan Morton]

> On 9 Dec, 2016, at 12:12, Phineas Gage <phineas919@gmail.com> wrote:
> 
> Given the half-duplex nature of 802.11 WiFi, is it possible to use fq_codel with software rate limiting on separate hardware from the WiFi radio, while still allowing at or near the full WiFi link rate?

Given that you can’t reliably predict the actual wifi throughput from userspace, and that it will vary over time due to external interference and path attenuation, that would be difficult.

However, you *can* loop both the ingress and egress traffic through a common IFB interface, and shape that - using Cake, even.  That sounds like what you’re trying to experiment with.

 - Jonathan Morton

Re: [Codel] [Make-wifi-fast] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

Ok, I had not realized that, thanks. :)

I’ve not seen this done anywhere, has anyone tried it? Otherwise I’ll give it a try and write back what I find.

In this case, the throughput for the backhaul links “should” be mostly stable, and we’ll just accept any variation as “no worse than before”.

It's true, I also want to try Cake (anywhere I wrote fq_codel that could be substituted with Cake), and I see from here (https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux <https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux>) that it should work on the 3.16.7 kernel I need to target. Voyage Linux doesn’t install with kernel sources, but I should be able to get that compiled with their SDK.

> On Dec 9, 2016, at 12:39 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 9 Dec, 2016, at 12:12, Phineas Gage <phineas919@gmail.com> wrote:
>> 
>> Given the half-duplex nature of 802.11 WiFi, is it possible to use fq_codel with software rate limiting on separate hardware from the WiFi radio, while still allowing at or near the full WiFi link rate?
> 
> Given that you can’t reliably predict the actual wifi throughput from userspace, and that it will vary over time due to external interference and path attenuation, that would be difficult.
> 
> However, you *can* loop both the ingress and egress traffic through a common IFB interface, and shape that - using Cake, even.  That sounds like what you’re trying to experiment with.
> 
> - Jonathan Morton
> 


[-- Attachment #2: Type: text/html, Size: 2379 bytes --]

Re: [Codel] [Make-wifi-fast] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 6006 bytes --]

If you are able to make the settings on the outgoing side of both the ends of 
the link, the need to estimate bandwidth should pretty much go away.

The whole mess of estimating bandwidth and throttling to keep below that 
estimate is only needed when the device(s) that you have control over are not 
the ones directly adjacent to the bottleneck. If you are directly adjacent to 
the bottleneck, then you don't need to guess how much data is too much, you see 
it directly by watching the queues build up (at which point, fq_codel kicks in 
and forces things to back off)

you really want to be running fq_codel on the 'radios', that's the bottleneck 
point as you shift from ethernet to wifi.

I would guess that the airtime fairness work probably won't make a huge 
difference in your case as the backhaul networks are all operating at or near 
the same speeds, but it may be that it will help by better grouping traffic into 
transmission bursts.

David Lang

On Fri, 9 Dec 2016, Phineas Gage wrote:

> Given the half-duplex nature of 802.11 WiFi, is it possible to use fq_codel with software rate limiting on separate hardware from the WiFi radio, while still allowing at or near the full WiFi link rate?
>
> Note #1: This is probably more of a job for the new "Make WiFi Fast" code in LEDE, but so far I can’t run that in the environment I’m working in (more below).
>
> Note #2: It also makes me think, I wonder how this problem is handled in the new Make WiFi Fast code, as that could be another big advantage for it, if the "half duplex rate limiting” I’m thinking of can’t be done in Linux (more below).
>
> Note #3: This question may not be specific to fq_codel, or even WiFi, as it could apply to any half duplex physical media, with any queueing discipline, but this list seems more active than most. :)
>
> How I got here: I arrived at this question because I started doing Flent RRUL runs over a local WiFi link and fq_codel was struggling to get latency down. I was using HTB to rate limit both input and output to 85 Mbps, which is a little less than the WiFi’s one-way throughput as reported by iperf3 (95 Mbps). Then it dawned on me that 802.11 WiFi is half duplex, so I need to limit the _combined_ rate. And indeed, simultaneous iperf3 throughput tests in both directions add up to a total combined rate of around 95 Mbps. I set the HTB input and output rate to 40 Mbps each, and average latency under load during an RRUL test dropped right away from ~30ms to ~5ms, and looked smooth, so I knew I was finally in control of the queue.
>
> New Problem: But now that my latency is reduced, I have a new problem. I’m rate limiting to around half my possible link rate in each direction, so if I’m only doing a download or upload on the link, I’m losing around half my potential throughput! That led to my question. Is it even possible to gain this throughput back without losing control of the queue? I would need the queueing in Linux to be aware that the physical media is half duplex, and rate limit the _total_ input and output traffic, not limit the input and output individually. Let’s call it “half duplex rate limiting.” AFAIK there’s no way to do this with IFB or Linux traffic control in general. Is there?
>
> Background: I’ll back up a bit and add some background on what I’m doing, for 
> those interested. I’m offering to help my community WISP try to reduce latency 
> in their network by deploying fq_codel where possible. For those who saw my 
> earlier WiFi related post, that was unrelated and on a different site, where 
> fq_codel on the edge router did improve latency there. For this WISP however, 
> since we can’t practically install and manage edge routers with fq_codel at 
> every end user site, I’m doing some research to find out if fq_codel can at 
> least help on the internal point-to-point backhaul links that connect their 
> nodes. Currently, sfq is used for those. Unlike a typical edge router, which 
> limits to “Internet speed”, these backhaul links usually operate at or near 
> the maximum link rate of the WiFi hardware. So it’s not like a typical CPE 
> device whose link rate might be well above the observed Internet rate, making 
> the “half duplex problem” irrelevant, if that makes sense.
>
> The WISP serves around 800 members, and is built with a combination of fiber and point-to-point WiFi links of various speeds and frequencies. Here’s a map:
>
> http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=k&geolocate=98%7C114%7C111%7C117%7C109%7C111%7C118%7C115%7C107%7C97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1& <http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=k&geolocate=98|114|111|117|109|111|118|115|107|97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&>
>
> Most of the nodes use Ubiquiti hardware for the radios and custom hardware for 
> the routers, connected to the radios via Ethernet, and most of which run on 1 
> GHz AMD G-T40E processors and run a customized distribution of Voyage Linux 
> 0.10 (Debian 8 based distribution for embedded hardware). The distro ships 
> with kernel 3.16.7, which does have a working sch_fq_codel module. To get 
> oriented, I started with some Flent RRUL runs on my test setup below, and 
> that’s where I ran into this question. Next, I’ll test on some actual hardware 
> targeted for deployment to understand its fq_codel throughput limits, but not 
> before I understand this “half duplex problem” better.
>
> Test setup:
>
> PC #1  <->  WiFi (802.11n, 130 Mbps Tx Rate, RSSI -60)  <->  AirPort Express  <->  100Mbit Ethernet  <->  PC #2
>
> PC #1: Runs a Voyage Linux VM (more on that later), with the Flent client and HTB + fq_codel, and is connected via the PC's WiFi adapter to the Airport Express.
>
> PC #2: Runs netserver for flent to connect to, and is connected via Ethernet to the Airport Express.
>
> Thanks for any input.

Re: [Codel] [Make-wifi-fast] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 2477 bytes --]

Thanks David, I do see your point. That would be great if I could run fq_codel on the outgoing queue of the device that has the WiFi radio.

But what I’m calling “radios” in this case can be a Ubiquiti PowerBeam M5-400, for example. This is a dish with integrated CPU and WiFi radio. It runs Ubiquiti’s firmware on kernel 2.6.32.68, and doesn’t come with an fq_codel module. You’re saying that this is the hardware where I’d really want to run it, and just add fq_codel as the egress leaf queue without rate limiting, right? But I don’t think fq_codel can run on this kernel. Much of the “radio” hardware is similar to this, and I’m also not sure what throughput the small CPUS they have (like MIPS 74Kc) could even support with fq_codel.

So unless I’m missing something, the only option I see in this case is to run it on the routers that are connected to these radio devices via Ethernet and do rate limiting.

As for the Make WiFi Fast project, in addition to the airtime fairness work, part of the plan is to add AQM/FQ at the 802.11 layer, right? In that case, if I were able to run this, I suppose I wouldn’t need to even configure an fq_codel queue with tc. But I would probably need to be able to run that on Ubiquiti’s firmware. PowerBeam M5-400’s do use ath9k, which the code targets, but I think so far the WISP would be reluctant to run OpenWRT or LEDE, though I can ask...

> On Dec 9, 2016, at 2:41 PM, David Lang <david@lang.hm> wrote:
> 
> If you are able to make the settings on the outgoing side of both the ends of the link, the need to estimate bandwidth should pretty much go away.
> 
> The whole mess of estimating bandwidth and throttling to keep below that estimate is only needed when the device(s) that you have control over are not the ones directly adjacent to the bottleneck. If you are directly adjacent to the bottleneck, then you don't need to guess how much data is too much, you see it directly by watching the queues build up (at which point, fq_codel kicks in and forces things to back off)
> 
> you really want to be running fq_codel on the 'radios', that's the bottleneck point as you shift from ethernet to wifi.
> 
> I would guess that the airtime fairness work probably won't make a huge difference in your case as the backhaul networks are all operating at or near the same speeds, but it may be that it will help by better grouping traffic into transmission bursts.
> 
> David Lang


[-- Attachment #2: Type: text/html, Size: 7661 bytes --]

Re: [Codel] [Make-wifi-fast] Software rate limiting with fq_codel for point-to-point WiFi backhaul links

[Phineas Gage]

[-- Attachment #1: Type: text/plain, Size: 4088 bytes --]

Thanks for the hint Jonathon. In general, this appears to be effective, at least for the flent rrul test.

Below are flent rrul_noclassification run graphs (flent.gz files available) and the script I used to set up fq_codel. I see close to link rate with iperf3 tests, which test throughput in only one direction, yet can still control the queue reasonably well when there’s bidirectional traffic. For example, I’ve got the total rate limited to 90 Mbit, and each average flow in a flent rrul test runs around 10 Mbit. There are four in each direction, eight total, so I see around 80 Mbit of TCP throughput along with the low rate traffic at around 6 ms latency, which is not bad.

If you set up fq_codel like this on both ends of the link, it doesn’t perform quite as well- latency goes up and throughput down a bit. There are probably some "unwanted interactions" when you do this. So it’s best to set it up on only end end of the link, probably the end whose egress heads towards the Internet if applicable, but I’m still thinking that through.

Again, for those reading this later, it would be best to just run fq_codel (or Cake but I’ll get to that next) on the egress right on the hardware with the WiFi interface attached. The interface will rate limit the queue automatically when there’s bidirectional traffic, and account for variable speed, so that should be a superior configuration. But this is a solution for when that isn’t possible, and the AQM has to be run on separate hardware, connected via Ethernet.

I still want to do more testing for my WISP, both with Cake and the new LEDE builds as well, in various configurations, so I hope to do that next and will start a new thread when there are results. For now, this “half duplex rate limiting” by running AQM on a common IFB interface may help people in some situations!

https://dl.dropboxusercontent.com/u/71551878/bufferbloat/fq_codel_90mbit.png <https://dl.dropboxusercontent.com/u/71551878/bufferbloat/fq_codel_90mbit.png>

https://dl.dropboxusercontent.com/u/71551878/bufferbloat/fq_codel_90mbit_both.png <https://dl.dropboxusercontent.com/u/71551878/bufferbloat/fq_codel_90mbit_both.png>

https://dl.dropboxusercontent.com/u/71551878/bufferbloat/qos.sh <https://dl.dropboxusercontent.com/u/71551878/bufferbloat/qos.sh>

> On Dec 9, 2016, at 1:37 PM, Phineas Gage <phineas919@gmail.com> wrote:
> 
> Ok, I had not realized that, thanks. :)
> 
> I’ve not seen this done anywhere, has anyone tried it? Otherwise I’ll give it a try and write back what I find.
> 
> In this case, the throughput for the backhaul links “should” be mostly stable, and we’ll just accept any variation as “no worse than before”.
> 
> It's true, I also want to try Cake (anywhere I wrote fq_codel that could be substituted with Cake), and I see from here (https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux <https://www.bufferbloat.net/projects/codel/wiki/Cake/#installing-cake-out-of-tree-on-linux>) that it should work on the 3.16.7 kernel I need to target. Voyage Linux doesn’t install with kernel sources, but I should be able to get that compiled with their SDK.
> 
>> On Dec 9, 2016, at 12:39 PM, Jonathan Morton <chromatix99@gmail.com <mailto:chromatix99@gmail.com>> wrote:
>> 
>>> On 9 Dec, 2016, at 12:12, Phineas Gage <phineas919@gmail.com <mailto:phineas919@gmail.com>> wrote:
>>> 
>>> Given the half-duplex nature of 802.11 WiFi, is it possible to use fq_codel with software rate limiting on separate hardware from the WiFi radio, while still allowing at or near the full WiFi link rate?
>> 
>> Given that you can’t reliably predict the actual wifi throughput from userspace, and that it will vary over time due to external interference and path attenuation, that would be difficult.
>> 
>> However, you *can* loop both the ingress and egress traffic through a common IFB interface, and shape that - using Cake, even.  That sounds like what you’re trying to experiment with.
>> 
>> - Jonathan Morton
>> 
> 


[-- Attachment #2: Type: text/html, Size: 5661 bytes --]