From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wj0-x234.google.com (mail-wj0-x234.google.com [IPv6:2a00:1450:400c:c01::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 872FA3B2A2; Fri, 9 Dec 2016 05:12:37 -0500 (EST) Received: by mail-wj0-x234.google.com with SMTP id tk12so10540564wjb.3; Fri, 09 Dec 2016 02:12:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:subject:date:message-id:cc:to:mime-version; bh=U4lsOoKBc3LCBXaAuENguFSBNkoJJGnfHYpN4ZUPYKA=; b=Tj386aUM36+ScqrTU6WekkfSTs4+ZR3aUUJddwGNzrd4Uc4RMpdsYzooCsLjUP4h43 GvMfH2YUG7O3+aqPVfN2iVaEcVUsPzIQ4zethaj8NSigo4ck7prJk+AixZXgocYGgjmu IF00x5Sr78cWKW1Da6z1STfAT1e7E/gCWyCPvfs/v0DAK4k4/JBmwwZ/eM1X36LStuI8 wi8ynNhsSzLBYjAGILDZ8MubY9eCZ6y7zwDfOhZ8L5lUDXt/NMY7bkQ+bRt856y9e6cj RUFZ/uQR3VryM/BGuCm6I+z3IzGmdjzu3Id27VLW5pTi/urxo2lLn0PIyx6062m6UeS6 j06Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:subject:date:message-id:cc:to:mime-version; bh=U4lsOoKBc3LCBXaAuENguFSBNkoJJGnfHYpN4ZUPYKA=; b=VEX2xJmd3MjpsoQYYTaSzQoOCK5CxfCBD1vRmA1yfSCbsWfPYD1ahtSpuA/ITZ8qsq Rk47z0af3Jb+wMiPThDL8N2FaEaqrDcCH8jFis8T2jDQzPuFLzg9Ud7XaQMr4nx7WXq7 E5evoqBCj9N6c1KInhWB6fsbC0QPMK/ZkMt5u8jHgFdqXG4/Hp+3qvzvjHGuj4UH0Df/ b/F3onlrWak87z8yBnV+WWL9PDT4GocPZDs3lnHOECybgApntpZqxjihJVvIcw0HhSzO bYEO7ud78DlaRdVp48jM45G6qLzPo5jX9lmuwrZniwsaUjMSqkhJ1grwhMEQg6mf/xUz YicA== X-Gm-Message-State: AKaTC01vm69cZdT+tAAmIbHeBBuYlLTGjLo7yRbandzBNpyii1Lrwyu7VqFu56PkHgmuWQ== X-Received: by 10.194.23.67 with SMTP id k3mr69426615wjf.103.1481278355983; Fri, 09 Dec 2016 02:12:35 -0800 (PST) Received: from [10.72.0.34] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id w18sm19776189wme.9.2016.12.09.02.12.34 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 09 Dec 2016 02:12:35 -0800 (PST) From: Phineas Gage Content-Type: multipart/alternative; boundary="Apple-Mail=_BAEF8A9D-486B-43F9-A677-F9C93E7D2F39" Date: Fri, 9 Dec 2016 11:12:34 +0100 Message-Id: <9BA0F7D8-5A25-4033-A101-38BF001B74EB@gmail.com> Cc: make-wifi-fast@lists.bufferbloat.net To: codel@lists.bufferbloat.net Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) Subject: [Codel] Software rate limiting with fq_codel for point-to-point WiFi backhaul links X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Dec 2016 10:12:37 -0000 --Apple-Mail=_BAEF8A9D-486B-43F9-A677-F9C93E7D2F39 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Given the half-duplex nature of 802.11 WiFi, is it possible to use = fq_codel with software rate limiting on separate hardware from the WiFi = radio, while still allowing at or near the full WiFi link rate? Note #1: This is probably more of a job for the new "Make WiFi Fast" = code in LEDE, but so far I can=E2=80=99t run that in the environment = I=E2=80=99m working in (more below). Note #2: It also makes me think, I wonder how this problem is handled in = the new Make WiFi Fast code, as that could be another big advantage for = it, if the "half duplex rate limiting=E2=80=9D I=E2=80=99m thinking of = can=E2=80=99t be done in Linux (more below). Note #3: This question may not be specific to fq_codel, or even WiFi, as = it could apply to any half duplex physical media, with any queueing = discipline, but this list seems more active than most. :) How I got here: I arrived at this question because I started doing Flent = RRUL runs over a local WiFi link and fq_codel was struggling to get = latency down. I was using HTB to rate limit both input and output to 85 = Mbps, which is a little less than the WiFi=E2=80=99s one-way throughput = as reported by iperf3 (95 Mbps). Then it dawned on me that 802.11 WiFi = is half duplex, so I need to limit the _combined_ rate. And indeed, = simultaneous iperf3 throughput tests in both directions add up to a = total combined rate of around 95 Mbps. I set the HTB input and output = rate to 40 Mbps each, and average latency under load during an RRUL test = dropped right away from ~30ms to ~5ms, and looked smooth, so I knew I = was finally in control of the queue.=20 New Problem: But now that my latency is reduced, I have a new problem. = I=E2=80=99m rate limiting to around half my possible link rate in each = direction, so if I=E2=80=99m only doing a download or upload on the = link, I=E2=80=99m losing around half my potential throughput! That led = to my question. Is it even possible to gain this throughput back without = losing control of the queue? I would need the queueing in Linux to be = aware that the physical media is half duplex, and rate limit the _total_ = input and output traffic, not limit the input and output individually. = Let=E2=80=99s call it =E2=80=9Chalf duplex rate limiting.=E2=80=9D AFAIK = there=E2=80=99s no way to do this with IFB or Linux traffic control in = general. Is there? Background: I=E2=80=99ll back up a bit and add some background on what = I=E2=80=99m doing, for those interested. I=E2=80=99m offering to help my = community WISP try to reduce latency in their network by deploying = fq_codel where possible. For those who saw my earlier WiFi related post, = that was unrelated and on a different site, where fq_codel on the edge = router did improve latency there. For this WISP however, since we = can=E2=80=99t practically install and manage edge routers with fq_codel = at every end user site, I=E2=80=99m doing some research to find out if = fq_codel can at least help on the internal point-to-point backhaul links = that connect their nodes. Currently, sfq is used for those. Unlike a = typical edge router, which limits to =E2=80=9CInternet speed=E2=80=9D, = these backhaul links usually operate at or near the maximum link rate of = the WiFi hardware. So it=E2=80=99s not like a typical CPE device whose = link rate might be well above the observed Internet rate, making the = =E2=80=9Chalf duplex problem=E2=80=9D irrelevant, if that makes sense. The WISP serves around 800 members, and is built with a combination of = fiber and point-to-point WiFi links of various speeds and frequencies. = Here=E2=80=99s a map: = http://mapa.czfree.net/#lat=3D50.76816800116274&lng=3D15.066890716552734&z= oom=3D13&autofilter=3D1&type=3Dk&geolocate=3D98%7C114%7C111%7C117%7C109%7C= 111%7C118%7C115%7C107%7C97&node=3D6101&aponly=3D1&bbonly=3D1&actlink=3D1&a= ctnode=3D1& = Most of the nodes use Ubiquiti hardware for the radios and custom = hardware for the routers, connected to the radios via Ethernet, and most = of which run on 1 GHz AMD G-T40E processors and run a customized = distribution of Voyage Linux 0.10 (Debian 8 based distribution for = embedded hardware). The distro ships with kernel 3.16.7, which does have = a working sch_fq_codel module. To get oriented, I started with some = Flent RRUL runs on my test setup below, and that=E2=80=99s where I ran = into this question. Next, I=E2=80=99ll test on some actual hardware = targeted for deployment to understand its fq_codel throughput limits, = but not before I understand this =E2=80=9Chalf duplex problem=E2=80=9D = better. Test setup: PC #1 <-> WiFi (802.11n, 130 Mbps Tx Rate, RSSI -60) <-> AirPort = Express <-> 100Mbit Ethernet <-> PC #2 PC #1: Runs a Voyage Linux VM (more on that later), with the Flent = client and HTB + fq_codel, and is connected via the PC's WiFi adapter to = the Airport Express. PC #2: Runs netserver for flent to connect to, and is connected via = Ethernet to the Airport Express. Thanks for any input.= --Apple-Mail=_BAEF8A9D-486B-43F9-A677-F9C93E7D2F39 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Given the half-duplex nature of 802.11 WiFi, = is it possible to use fq_codel with software rate limiting on separate = hardware from the WiFi radio, while still allowing at or near the full = WiFi link rate?

Note #1: This is probably more of a job for the new "Make = WiFi Fast" code in LEDE, but so far I can=E2=80=99t run that in the = environment I=E2=80=99m working in (more below).

Note #2: It also makes me think, I = wonder how this problem is handled in the new Make WiFi Fast code, as = that could be another big advantage for it, if the "half duplex rate = limiting=E2=80=9D I=E2=80=99m thinking of can=E2=80=99t be done in Linux = (more below).

Note #3: This question may not be specific to fq_codel, or = even WiFi, as it could apply to any half duplex physical media, with any = queueing discipline, but this list seems more active than most. = :)

How I got = here: I arrived at this question because I started doing Flent RRUL runs = over a local WiFi link and fq_codel was struggling to get latency down. = I was using HTB to rate limit both input and output to 85 Mbps, which is = a little less than the WiFi=E2=80=99s one-way throughput as reported by = iperf3 (95 Mbps). Then it dawned on me that 802.11 WiFi is half duplex, = so I need to limit the _combined_ rate. And indeed, simultaneous iperf3 = throughput tests in both directions add up to a total combined rate of = around 95 Mbps. I set the HTB input and output rate to 40 Mbps each, and = average latency under load during an RRUL test dropped right away from = ~30ms to ~5ms, and looked smooth, so I knew I was finally in control of = the queue. 

New Problem: But now that my latency is = reduced, I have a new problem. I=E2=80=99m rate limiting to around half = my possible link rate in each direction, so if I=E2=80=99m only doing a = download or upload on the link, I=E2=80=99m losing around half my = potential throughput! That led to my question. Is it even possible to = gain this throughput back without losing control of the queue? I would = need the queueing in Linux to be aware that the physical media is half = duplex, and rate limit the _total_ input and output traffic, not limit = the input and output individually. Let=E2=80=99s call it =E2=80=9Chalf = duplex rate limiting.=E2=80=9D AFAIK there=E2=80=99s no way to do this = with IFB or Linux traffic control in general. Is there?

Background: I=E2=80=99ll = back up a bit and add some background on what I=E2=80=99m doing, for = those interested. I=E2=80=99m offering to help my community WISP try to = reduce latency in their network by deploying fq_codel where possible. = For those who saw my earlier WiFi related post, that was unrelated and = on a different site, where fq_codel on the edge router did improve = latency there. For this WISP however, since we can=E2=80=99t practically = install and manage edge routers with fq_codel at every end user site, = I=E2=80=99m doing some research to find out if fq_codel can at least = help on the internal point-to-point backhaul links that connect their = nodes. Currently, sfq is used for those. Unlike a typical edge router, = which limits to =E2=80=9CInternet speed=E2=80=9D, these backhaul links = usually operate at or near the maximum link rate of the WiFi hardware. = So it=E2=80=99s not like a typical CPE device whose link rate might be = well above the observed Internet rate, making the =E2=80=9Chalf duplex = problem=E2=80=9D irrelevant, if that makes sense.
The WISP serves around 800 members, = and is built with a combination of fiber and point-to-point WiFi links = of various speeds and frequencies. Here=E2=80=99s a map:

http://mapa.czfree.net/#lat=3D50.76816800116274&lng=3D15.06= 6890716552734&zoom=3D13&autofilter=3D1&type=3Dk&geolocate=3D= 98%7C114%7C111%7C117%7C109%7C111%7C118%7C115%7C107%7C97&node=3D6101&am= p;aponly=3D1&bbonly=3D1&actlink=3D1&actnode=3D1&
=

Most of the nodes = use Ubiquiti hardware for the radios and custom hardware for the = routers, connected to the radios via Ethernet, and most of which run on = 1 GHz AMD G-T40E processors and run a customized distribution of Voyage = Linux 0.10 (Debian 8 based distribution for embedded hardware). The = distro ships with kernel 3.16.7, which does have a working sch_fq_codel = module. To get oriented, I started with some Flent RRUL runs on my test = setup below, and that=E2=80=99s where I ran into this question. Next, = I=E2=80=99ll test on some actual hardware targeted for deployment to = understand its fq_codel throughput limits, but not before I understand = this =E2=80=9Chalf duplex problem=E2=80=9D better.

Test = setup:

PC #1 =  <->  WiFi (802.11n, 130 Mbps Tx Rate, RSSI -60) =  <->  AirPort Express  <->  100Mbit = Ethernet  <->  PC #2

PC #1: Runs a Voyage Linux VM (more on = that later), with the Flent client and HTB + fq_codel, and is connected = via the PC's WiFi adapter to the Airport Express.
PC #2: Runs netserver for flent to = connect to, and is connected via Ethernet to the Airport = Express.

Thanks for any input.
= --Apple-Mail=_BAEF8A9D-486B-43F9-A677-F9C93E7D2F39--