From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 334803BA8D; Tue, 31 Jan 2017 10:52:30 -0500 (EST) Received: by mail-wm0-x22d.google.com with SMTP id c85so266708558wmi.1; Tue, 31 Jan 2017 07:52:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=UYiFba+jxx+QF2TFTQ+eEEpCZeW7SzPEa3YpAgL3odE=; b=WCWe+VKcHfBSecKeZ1dnz8RylBxEN+2muhcSFrqFfWtngeMBs885D83v0hedVFs/vI 5IgimeT2vbIbgyhqPLrbSEc6S3TpfwLwFtzS7eQXlm5h9eV4bUpG1N1QpeYy8Zmn/Bi6 1Q1ho5vVEw6+6u7iyoN6laF8nopRR3cB/SPsWmW3frd4fLo0ed4tGYi4aQzrX3RjBEwv 0zuW4QXU5zow9P3/qszpQvZOxDa7nnlpyy2n6WwI3jIe9ZITTKK/JDNrnDFU0MViKF1q Eg81u+d2rXbVZe+a3U285TYl6o+/aW+nTsXADwXIjxegWC1CfJGvgXJ+ie9HKZ4PhihM JQ8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=UYiFba+jxx+QF2TFTQ+eEEpCZeW7SzPEa3YpAgL3odE=; b=sGSO4QUkAWdYUO2yJR2ObGHBKE432OLIIg+ZE28vkWfN5xX1zMt0M5TMN4s9IV8112 6h7TVwyHvvmmW0Nu6JlOUfd8w92yGl5o6NnRFub1FUkCwPQByk6kkipFq7hihwX0Hi7j qeVmHRvDfl+D5rUx/PPgLZfLcw4ZxFOTs+/jc86J17Z5kPSxDEdMNgkI1ejBLURej+KT ROrVmg8ndmS/KC9H7iYJAE/HxT1xI2b00zBy89RFTfJSLMs91SdrvB6+aRYXS9bXRceW doxM6LrMFpYuVVFjHPgNO3TSZRpyuhWtQ5EdIPXdGGu7XJL4qEbNRLFmwU3JSIRzOtrE c4OQ== X-Gm-Message-State: AIkVDXJNiSvBndU00sW0+NTc1hY8LWsxZebPDPfc9ZzJLBZoJuqqd2vxhBQHFogPBpVrRg== X-Received: by 10.28.166.4 with SMTP id p4mr9887302wme.87.1485877949126; Tue, 31 Jan 2017 07:52:29 -0800 (PST) Received: from [10.72.0.34] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id e72sm24534242wma.16.2017.01.31.07.52.28 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 31 Jan 2017 07:52:28 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_7F9BE222-E9AE-434A-B9D7-3B92C64848B4" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: <877f5c2pew.fsf@toke.dk> Date: Tue, 31 Jan 2017 16:52:29 +0100 Cc: make-wifi-fast@lists.bufferbloat.net, cake@lists.bufferbloat.net Message-Id: References: <32C42951-373F-4D90-8936-AA07039E5D73@gmail.com> <877f5c2pew.fsf@toke.dk> To: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: Apple Mail (2.3124) Subject: Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2017 15:52:30 -0000 --Apple-Mail=_7F9BE222-E9AE-434A-B9D7-3B92C64848B4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jan 30, 2017, at 10:44 PM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >=20 > Oh my, this is quite a lot of tests. Nice :) It=E2=80=99s also a thumbs up for the ath9k driver changes that nothing = went wrong during the testing. It takes about 15 hours for a full run = and I probably did that 4-5 times total. > Few general points on running tests: >=20 > - Yeah, as you note Flent has a batch facility. Did you not use this > simply because you couldn't find it, or was there some other reason? > Would love some feedback on what I can do to make that more useful to > people... While I have no doubt that your 'flenter.py' works, = wrapping > a wrapper in this sense makes me cringe a little bit ;) I actually didn=E2=80=99t notice it existed until I was about 85% done = and scanning the Flent man page for some other reason. I cringed, but at = that point I just stuck with what I had. I don=E2=80=99t know if Flent = can also make some basic html report with the graphs and setup output, = but that was useful to write for myself. Flent=E2=80=99s metadata = feature sounds useful and I=E2=80=99ll try that. > - I'm not sure if you're checking that applying your qdiscs actually > works? For the WiFi interfaces with 'noqueue' you *cannot* apply a > different qdisc (which also answers your question #2). Hmm. Unless I=E2=80=99m missing something, what I=E2=80=99m seeing is = that I _can_ add another qdisc, only that it=E2=80=99s ineffective = unless soft rate limiting is used. As evidence, here's my nolimit test = of fq_codel: http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html = Under the sections qos_om1 and qos_om2, you can see that fq_codel has = been added to wlan0 from the tc output. But the latency is the same 25 = ms as the default, so I=E2=80=99m not controlling the queue, and that = was true for any qdisc without rate limiting. But, here's =E2=80=9840mbit full-duplex rate limiting on the AP and = station with htb+fq_codel=E2=80=99: = http://www.drhleny.cz/bufferbloat/fq_codel_fd-wifi-both_40mbit/index.html = With this, I could reduce latency to about 9ms, so it did = =E2=80=9Csomething=E2=80=9D. And pfifo with 40mbit rate limiting did the = =E2=80=9Csomething=E2=80=9D that it usually does, bloating things up to = 200+ ms: http://www.drhleny.cz/bufferbloat/pfifo_fd-wifi-both_40mbit/index.html = So I don=E2=80=99t see any evidence that I can=E2=80=99t add a qdisc to = the Wi-Fi devices, only that I have to use soft limiting for it to be = effective. Now, HTB rate limiting seems to break down on the OM2P-HS at bitrates = above about 50mbit, when the actual throughput starts to fall away from = the set limit, but that=E2=80=99s another story. > Question 1 (and partly #13): Yeah, the version of LEDE you're running > already has the FQ-CoDel-based queueing in the ath9k driver. The > baseline you're seeing is consistent with the results we've been = getting > in testing. This is also seen by any gains you get being paired with > quite a hefty hit in throughput. So with this driver, I would say it's > not worth it. However, this is going to be different on a setup = without > the WiFi queueing fixes. Ok, that explains a lot, thanks. I was still able to see about a 50% = reduction in latency (from ~25 ms to ~12ms) with a 13% drop in = throughput (from ~92 Mbps to ~80Mbps), when doing half-duplex rate = limiting to 85Mbps and fq_codel=E2=80=99ing on the external router. See: http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth-ap_85mbit/index.html = vs the default: http://www.drhleny.cz/bufferbloat/default/index.html = I can get down to 10ms if I give up another 5 Mbps, or lower values with = more severe throughput sacrifices. But this is with a stable RSSI of around -50 and low noise. I understand = that fq-codel=E2=80=99ing in the driver must be superior in its handling = of rate changes, retries or other external factors, and that = point-to-multipoint is a different story. But maybe some of FreeNet=E2=80=99= s line-of-sight point-to-point links may also be stable enough such that = fixed software rate limiting is usable for them, I=E2=80=99m not sure = yet. It=E2=80=99s not critical, but why am I able to see this level of = reduction when there=E2=80=99s already fq-codel in the driver? 25ms is = very good, I only wonder where I=E2=80=99m getting the extra 10-15ms = from, out of interest. :) > Question 5: For TCP you can't get packet loss from user space; you'll > need packet captures for that. So no way to get it from Flent either. > You can, however, get average throughput. Look at the box plots; if = you > run multiple iterations of the same test, you can plot several data > files in a single box_combine plot, to get error bars. `flent > file.flent.gz -f summary` (which is the default if you don't specify a > plot) will get you averages per data series; or you can extract it = from > the metadata. Ok, so far, I was doing `cat file.flent.gz | grep null | wc -l`, which = is a very crude count of the nulls recorded, which seem to happen for = the udp and icmp flows with packet loss. There are always some nulls = from before the test starts and after it ends, but if the count jumps up = I speculate that there=E2=80=99s more packet loss. It=E2=80=99s pretty = weak but it=E2=80=99s a hint. I see the box plots now, that=E2=80=99s nice, thanks! --Apple-Mail=_7F9BE222-E9AE-434A-B9D7-3B92C64848B4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Jan 30, 2017, at 10:44 PM, Toke H=C3=B8iland-J=C3=B8rgensen = <toke@toke.dk> = wrote:

Oh my, this = is quite a lot of tests. Nice :)

It=E2=80= =99s also a thumbs up for the ath9k driver changes that nothing went = wrong during the testing. It takes about 15 hours for a full run and I = probably did that 4-5 times total.

Few general = points on running tests:

- Yeah, as you = note Flent has a batch facility. Did you not use this
=  simply because you couldn't find it, or was there some other = reason?
 Would love some feedback on what I can do = to make that more useful to
 people... While I have = no doubt that your 'flenter.py' works, wrapping
 a = wrapper in this sense makes me cringe a little bit ;)

I = actually didn=E2=80=99t notice it existed until I was about 85% done and = scanning the Flent man page for some other reason. I cringed, but at = that point I just stuck with what I had. I don=E2=80=99t know if Flent = can also make some basic html report with the graphs and setup output, = but that was useful to write for myself. Flent=E2=80=99s metadata = feature sounds useful and I=E2=80=99ll try that.

- I'm not sure if you're checking that applying your qdiscs = actually
 works? For the WiFi interfaces with = 'noqueue' you *cannot* apply a
 different qdisc = (which also answers your question #2).

Hmm. = Unless I=E2=80=99m missing something, what I=E2=80=99m seeing is that I = _can_ add another qdisc, only that it=E2=80=99s ineffective unless soft = rate limiting is used. As evidence, here's my nolimit test of = fq_codel:





With this, I = could reduce latency to about 9ms, so it did =E2=80=9Csomething=E2=80=9D. = And pfifo with 40mbit rate limiting did the =E2=80=9Csomething=E2=80=9D = that it usually does, bloating things up to 200+ ms:


So I don=E2=80=99t see = any evidence that I can=E2=80=99t add a qdisc to the Wi-Fi devices, only = that I have to use soft limiting for it to be effective.

Now, HTB rate limiting seems to break down on the = OM2P-HS at bitrates above about 50mbit, when the actual throughput = starts to fall away from the set limit, but that=E2=80=99s another = story.

Question 1 (and partly #13): = Yeah, the version of LEDE you're running
already has the = FQ-CoDel-based queueing in the ath9k driver. The
baseline = you're seeing is consistent with the results we've been getting
in testing. This is also seen by any gains you get being = paired with
quite a hefty hit in throughput. So with this = driver, I would say it's
not worth it. However, this is = going to be different on a setup without
the WiFi queueing = fixes.

Ok, that explains a lot, thanks. I was still able = to see about a 50% reduction in latency (from ~25 ms to ~12ms) with a = 13% drop in throughput (from ~92 Mbps to ~80Mbps), when doing = half-duplex rate limiting to 85Mbps and fq_codel=E2=80=99ing on the = external router. See:


vs the = default:

<= div>
I can get down to 10ms if I give up = another 5 Mbps, or lower values with more severe throughput = sacrifices.

But this is with a = stable RSSI of around -50 and low noise. I understand that = fq-codel=E2=80=99ing in the driver must be superior in its handling of = rate changes, retries or other external factors, and that = point-to-multipoint is a different story. But maybe some of FreeNet=E2=80=99= s line-of-sight point-to-point links may also be stable enough such that = fixed software rate limiting is usable for them, I=E2=80=99m not sure = yet.

It=E2=80=99s not critical, but = why am I able to see this level of reduction when there=E2=80=99s = already fq-codel in the driver? 25ms is very good, I only wonder where = I=E2=80=99m getting the extra 10-15ms from, out of interest. = :)

Question 5: For TCP you can't = get packet loss from user space; you'll
need packet = captures for that. So no way to get it from Flent either.
You can, however, get average throughput. Look at the box = plots; if you
run multiple iterations of the same test, = you can plot several data
files in a single box_combine = plot, to get error bars. `flent
file.flent.gz -f summary` = (which is the default if you don't specify a
plot) will = get you averages per data series; or you can extract it from
the metadata.

Ok, so far, I was doing `cat file.flent.gz | grep = null | wc -l`, which is a very crude count of the nulls recorded, which = seem to happen for the udp and icmp flows with packet loss. There are = always some nulls from before the test starts and after it ends, but if = the count jumps up I speculate that there=E2=80=99s more packet loss. = It=E2=80=99s pretty weak but it=E2=80=99s a hint.

I see the box plots now, that=E2=80=99s nice, = thanks!

= --Apple-Mail=_7F9BE222-E9AE-434A-B9D7-3B92C64848B4--