[Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

Hi, I’ve posted some Flent results and analysis for point-to-point Wi-Fi using LEDE on OM2P-HS (ath9k):

http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html <http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html>

Over 500 runs were done using different configurations, as the effects of various changes were explored. In case someone has the time to respond, there are a number of questions in red. Whatever feedback I get I’ll try to incorporate into a final document. Also, if I’ve made any false assertions I’d love to hear about it.

As described on the page, I’m doing this to try to help my cooperative WISP here in Czech, and also in case it helps the Bufferbloat project somehow. Although much of the new Wi-Fi work is occurring at the driver level, I hope it’s not misguided to still explore using the qdisc layer for Wi-Fi, particularly in cases where a new Wi-Fi driver can’t be deployed.

These first results are from LEDE on the OM2P-HS, but I hope to get some test devices from my WISP soon to test the hardware they’re using (Ubiquiti Wi-Fi devices with custom Voyage Linux routers running on a PCengines APU).

Regards,
Pete


[-- Attachment #2: Type: text/html, Size: 1706 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

> Hi, I’ve posted some Flent results and analysis for point-to-point Wi-Fi using LEDE on OM2P-HS (ath9k):
>
> http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html

Oh my, this is quite a lot of tests. Nice :)


Few general points on running tests:

- Yeah, as you note Flent has a batch facility. Did you not use this
  simply because you couldn't find it, or was there some other reason?
  Would love some feedback on what I can do to make that more useful to
  people... While I have no doubt that your 'flenter.py' works, wrapping
  a wrapper in this sense makes me cringe a little bit ;)

- Flent also has a metadata gathering feature where you can get lots of
  stats from both your qdisc-based bottlenecks, and your WiFi links.

- I'm not sure if you're checking that applying your qdiscs actually
  works? For the WiFi interfaces with 'noqueue' you *cannot* apply a
  different qdisc (which also answers your question #2).

> Over 500 runs were done using different configurations, as the effects
> of various changes were explored. In case someone has the time to
> respond, there are a number of questions in red. Whatever feedback I
> get I’ll try to incorporate into a final document. Also, if I’ve made
> any false assertions I’d love to hear about it.

I don't have time to go through your whole document now, but a few
points:

Question 1 (and partly #13): Yeah, the version of LEDE you're running
already has the FQ-CoDel-based queueing in the ath9k driver. The
baseline you're seeing is consistent with the results we've been getting
in testing. This is also seen by any gains you get being paired with
quite a hefty hit in throughput. So with this driver, I would say it's
not worth it. However, this is going to be different on a setup without
the WiFi queueing fixes.

Question 5: For TCP you can't get packet loss from user space; you'll
need packet captures for that. So no way to get it from Flent either.
You can, however, get average throughput. Look at the box plots; if you
run multiple iterations of the same test, you can plot several data
files in a single box_combine plot, to get error bars. `flent
file.flent.gz -f summary` (which is the default if you don't specify a
plot) will get you averages per data series; or you can extract it from
the metadata.

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

> - Yeah, as you note Flent has a batch facility. Did you not use this
>   simply because you couldn't find it, or was there some other reason?
>   Would love some feedback on what I can do to make that more useful to
>   people... While I have no doubt that your 'flenter.py' works, wrapping
>   a wrapper in this sense makes me cringe a little bit ;)
>

Wait, what?  It does?  (I've been using wrapper scripts as well)


> - Flent also has a metadata gathering feature where you can get lots of
>   stats from both your qdisc-based bottlenecks, and your WiFi links.
>

Again, it does?  Neat!  (I try to bury data into the tag for the run...)

Question 5: For TCP you can't get packet loss from user space; you'll
> need packet captures for that. So no way to get it from Flent either.
> You can, however, get average throughput. Look at the box plots; if you
> run multiple iterations of the same test, you can plot several data
> files in a single box_combine plot, to get error bars. `flent
> file.flent.gz -f summary` (which is the default if you don't specify a
> plot) will get you averages per data series; or you can extract it from
> the metadata.
>

You don't get packet loss, per se, but you can periodically poll the
TCP_INFO struct via getsockopt() and get the retransmission count (which
more or less gives you the packet loss rate).  (which is what iperf3 does
to gather stats like it's view of rtt, retransmits, etc).

-Aaron

[-- Attachment #2: Type: text/html, Size: 2100 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Aaron Wood <woody77@gmail.com> writes:

>  - Yeah, as you note Flent has a batch facility. Did you not use this
>    simply because you couldn't find it, or was there some other reason?
>    Would love some feedback on what I can do to make that more useful to
>    people... While I have no doubt that your 'flenter.py' works, wrapping
>    a wrapper in this sense makes me cringe a little bit ;)
>
> Wait, what?  It does?  (I've been using wrapper scripts as well)

Yes, but as you can tell it hasn't seen much usage outside of my own.
Please do try it out and let me know what you think :)

>  - Flent also has a metadata gathering feature where you can get lots of
>    stats from both your qdisc-based bottlenecks, and your WiFi links.
>
> Again, it does?  Neat!  (I try to bury data into the tag for the
> run...)

Yes. I have still not come up with a good way to use the metadata
afterwards (other than viewing it in the GUI; you can add columns to the
open files view with arbitrary metadata info). Filtering on it, and
adding it to the plots as annotations would probably be useful, but not
sure how to do the API.

Also, the --test-parameter option was originally intended to just be a
way to add arbitrary key/val pairs to the metadata. Every option you
specify that way will be saved in the data file. It has since been
co-opted by some other features, so that some keys will modify the test
behaviour; see the man page for those :)

>  Question 5: For TCP you can't get packet loss from user space; you'll
>  need packet captures for that. So no way to get it from Flent either.
>  You can, however, get average throughput. Look at the box plots; if you
>  run multiple iterations of the same test, you can plot several data
>  files in a single box_combine plot, to get error bars. `flent
>  file.flent.gz -f summary` (which is the default if you don't specify a
>  plot) will get you averages per data series; or you can extract it from
>  the metadata.
>
> You don't get packet loss, per se, but you can periodically poll the
> TCP_INFO struct via getsockopt() and get the retransmission count
> (which more or less gives you the packet loss rate). (which is what
> iperf3 does to gather stats like it's view of rtt, retransmits, etc).

Yeah, I do believe you can make netperf output that at the end of the
test as well; but Flent currently does not support collecting that.

There's some work underway, also, to poll 'ss' for socket statistics
during the test, so we can get insight into the TCP state machine...

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

Very exhaustive work, thank you.

A backstory of how I got involved in the bufferbloat effort was that I
deployed some shiny "new" and "faster" wireless-n radios (6 years
ago)... and my WISP network in Nicaragua collapsed in rain - which was
about 6 months after I'd made the cutover from g and from motorola's
radios. The signal strength was within expected parameters, the
achieved rates were half to 1/3 what they were dry, but latencies
climbed to over 30 seconds in some cases. I had *no* idea what was
going wrong at the time, and it wasn't until 6 months after I closed
the business that I ran into Jim Gettys, and the rest is history.

I never got around to writing it up, I just gave a couple talks on the
subject and moved onto fixing bufferbloat thoroughly. We got
distracted by trying to solve it on the ISP backbones before tackling
wifi in this past year.

OpenWrt Chaos calmer generally should work "pretty good" on 802.11n
p2p links with the fq_codel qdisc at the qdisc layer, and a bit of
tuning of qlen_be (I deployed 12-24 on the yurtlab campus), and
turning off 802.11e. Lacking ATF and good queue management closer to
the radio, p2mp did not work well with chaos calmer.

with the new stuff now landing in lede I still recommend turning off
802.11e and relying entirely on fq_codel down there, and I have high
hopes that basic p2mp will work well for up to X stations.

... As for the performance of openmesh being pretty good... they were
the first group to test the fq_codel intermediate queues and ATF code
way back in september, :) - it's not clear to me if that's what you
were testing or not or they shipped an update yet.

Your 20-30ms p2p latency performance on the rrul_be test is consistent
with the results we get from the fq_codel intermediate queue patches,
which keep 2 ~4ms aggregates near the hardware. Applying a qdisc on
top should make little to no difference, as all the real work gets
done there. [2]

A good test to run is to lower the mcs rates (set the rateset
manually, or add distance, and/or rain) and to see how flat latency
remains. This is also a better test of real-world conditions, if you
can get some reports back on the actual mcs rates being achieved in
the field, and use those.

It would be my hope that 802.11e is off (rrul will show this, and we
still do badly with it on)

You can probably within this deployment shape the uplinks to some
fairly low value and get good performance more the time.

I do not have any real hope for being able to make wifi better with
soft-shapers. It's a gordian knot - you need to respond rapidly to
changes in rate, both up and down, and mix up flows thoroughly,
optimize your aggregates to go to one station at a time and switch to
the next, and that's what we got with codel close to the hardware as
it is now in lede. [1]

If it helps any, this is the best later talk I've given on these subjects:

https://www.youtube.com/watch?v=Rb-UnHDw02o

UBNT's gear (commonly used by wisps) has some neat tricks to manage
things better, when I last took apart their qos system it was an
insane set of sfqs with sfqs.

[1] And as we have more radio-dense environments now, retries are
becoming an issue.

[2] http://blog.cerowrt.org/post/real_results/

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4689 bytes --]


> On Jan 31, 2017, at 12:21 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> A backstory of how I got involved in the bufferbloat effort was that I
> deployed some shiny "new" and "faster" wireless-n radios (6 years
> ago)... and my WISP network in Nicaragua collapsed in rain - which was
> about 6 months after I'd made the cutover from g and from motorola's
> radios. The signal strength was within expected parameters, the
> achieved rates were half to 1/3 what they were dry, but latencies
> climbed to over 30 seconds in some cases. I had *no* idea what was
> going wrong at the time, and it wasn't until 6 months after I closed
> the business that I ran into Jim Gettys, and the rest is history.

Quite an interesting story, could be in the project background somewhere. :)

> I never got around to writing it up, I just gave a couple talks on the
> subject and moved onto fixing bufferbloat thoroughly. We got
> distracted by trying to solve it on the ISP backbones before tackling
> wifi in this past year.

Yeah, I also don’t want to waste too much time trying to improve latency in their point-to-point Wi-Fi backhaul links unless it’s going to help. I suppose the questions are:

1) Are their backhaul links stable enough in all conditions to hold a steady enough rate that soft rate limiting is practical without sacrificing too much throughput AND
2) Is the sfq setup Ubiquiti has in their gear “good enough” in managing bufferbloat that it wouldn’t make much of a difference anyway.

As for the final connection to my residence, it's 3km NLOS through some deciduous trees close to me, so I have a better signal in winter with no leaves. While it’s pretty good all things considered (actual 20 Mbps up, 30 Mbps down when things are well), bitrate can vary maybe 20% because of the link, and more because of other network load. So even if fq_codel with soft rate limiting does help, which it does, it’s not as practical for my final connection to do it. Just thought of it now, but I wonder if I can run LEDE on my PowerBeam M5-400.

Anyway, that’s why I thought backhaul links are a better candidate for soft rate limiting (since they’re usually line-of-sight), if it even helps at all (TBD).

> ... As for the performance of openmesh being pretty good... they were
> the first group to test the fq_codel intermediate queues and ATF code
> way back in september, :) - it's not clear to me if that's what you
> were testing or not or they shipped an update yet.

Just to be clear, I was only testing LEDE on OM2P-HS. I’d like to test Chaos Calmer (or Open Mesh’s stock firmware), so I’m not mixing the testing of the ath9k changes with the qdiscs, but I’ll see if I get to this along with the Ubiquiti testing.

> A good test to run is to lower the mcs rates (set the rateset
> manually, or add distance, and/or rain) and to see how flat latency
> remains. This is also a better test of real-world conditions, if you
> can get some reports back on the actual mcs rates being achieved in
> the field, and use those.

This, I’m definitely going to try.

Although I can’t make it rain in my office :) I can use ‘iw dev wlan0 set bitrates ht-mcs-2.4 X’, where X is the MCS level. This appears to be effective, and I could even write a “rain.sh" and change it on the fly to see what happens.

> It would be my hope that 802.11e is off (rrul will show this, and we
> still do badly with it on)

802.11e is on, as it’s the default in LEDE and I didn’t change it. I can certainly turn that off and compare.

> You can probably within this deployment shape the uplinks to some
> fairly low value and get good performance more the time.
> 
> I do not have any real hope for being able to make wifi better with
> soft-shapers. It's a gordian knot - you need to respond rapidly to
> changes in rate, both up and down, and mix up flows thoroughly,
> optimize your aggregates to go to one station at a time and switch to
> the next, and that's what we got with codel close to the hardware as
> it is now in lede. [1]
> 
> If it helps any, this is the best later talk I've given on these subjects:
> 
> https://www.youtube.com/watch?v=Rb-UnHDw02o <https://www.youtube.com/watch?v=Rb-UnHDw02o>

I understand. Enjoyed that talk thoroughly, thanks!

> UBNT's gear (commonly used by wisps) has some neat tricks to manage
> things better, when I last took apart their qos system it was an
> insane set of sfqs with sf’s.

So that was one of my questions, is that setup “good enough” that external rate limiting and shaping isn’t worth it, even with a stable connection. TBD.


[-- Attachment #2: Type: text/html, Size: 6448 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

> On Jan 31, 2017, at 12:21 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> It would be my hope that 802.11e is off (rrul will show this, and we
> still do badly with it on)

Does this mean to try disabling WMM (uci set wireless.default_radio0.wmm='0’)?

That’s how I took it, as it looks like WMM is a subset of 802.11e. But if I do that, I lose 802.11n support and throughput drops from ~90 Mbps to ~22 Mbps. I could get those results out of interest, but it wouldn’t be practical in the field.

Or is there another way to turn off 802.11e without losing 802.11n support?

Pete

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Täht]

On 2/14/17 12:56 AM, Pete Heist wrote:
> 
>> On Jan 31, 2017, at 12:21 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> It would be my hope that 802.11e is off (rrul will show this, and we
>> still do badly with it on)
> 
> Does this mean to try disabling WMM (uci set wireless.default_radio0.wmm='0’)?
> 
> That’s how I took it, as it looks like WMM is a subset of 802.11e. But if I do that, I lose 802.11n support and throughput drops from ~90 Mbps to ~22 Mbps. I could get those results out of interest, but it wouldn’t be practical in the field.
> 
> Or is there another way to turn off 802.11e without losing 802.11n support?

you are correct. I misspoke. turning off wmm disables 802.11n. Which I'd
forgotten until I too ran a test with wmm disabled a week or two back.

What I'd done in cerowrt was disable the internal mac80211 QoS
classifier entirely. (-1 line of code). Another way to do it is to apply
an iptables rule to wash away all diffserv markings, but I don't care
for that answer.

try as I might I cannot remember if this was even possible in hostapd or
in the mac layer any other way. I'll look this week.

> Pete
> 
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
> 

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]


> On Feb 16, 2017, at 12:03 AM, Dave Täht <dave@taht.net> wrote:
> 
> What I'd done in cerowrt was disable the internal mac80211 QoS
> classifier entirely. (-1 line of code). Another way to do it is to apply
> an iptables rule to wash away all diffserv markings, but I don't care
> for that answer.
> 
> try as I might I cannot remember if this was even possible in hostapd or
> in the mac layer any other way. I'll look this week.

Aha, so since I’m only doing the rrul_be test for now, this won’t matter anyway right? I suppose somewhere diffserv markings are mapped to WMM access categories.

It looks like AirOS has its own mappings: https://help.ubnt.com/hc/en-us/articles/205231750-airMAX-How-is-QoS-and-prioritization-handled-by-airMAX- <https://help.ubnt.com/hc/en-us/articles/205231750-airMAX-How-is-QoS-and-prioritization-handled-by-airMAX->

and there doesn’t look like a practical way to disable it in their UI. So still, for now, disabling WMM is something I can only do in testing.

It looks like FreeNet doesn’t touch diffserv markings in their QoS config, which means they leave it up to AirOS’s mappings above. This always makes me wonder what’s to keep someone from just marking all their traffic 0x7 and stomping over everyone else.

Thanks,
Pete


[-- Attachment #2: Type: text/html, Size: 2059 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Sebastian Moeller]

> On Feb 16, 2017, at 08:57, Pete Heist <peteheist@gmail.com> wrote:
> [… discussion about DSCP to WMM classes mapping]
> This always makes me wonder what’s to keep someone from just marking all their traffic 0x7 and stomping over everyone else.

	I have a gut feeling that an AP in a untrusted/hostile environment should monitor the usage of the 4 different WMM classes and step up their class accordingly. That is in an environment where there is a lot of AC_VO or AC_VI traffic the AP should elevate its normal data packets priority to match as not too be drowned out by the other senders. Sort of a reciprocal tit-for-tat approach, with the goal that the AP will keep access to a decent share of airtime. But since I am a layman in these matters, I might be out to lunch on this…

Best Regards
	Sebastian


> 
> Thanks,
> Pete
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

> On Feb 16, 2017, at 9:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> On Feb 16, 2017, at 08:57, Pete Heist <peteheist@gmail.com> wrote:
>> [… discussion about DSCP to WMM classes mapping]
>> This always makes me wonder what’s to keep someone from just marking all their traffic 0x7 and stomping over everyone else.
> 
> 	I have a gut feeling that an AP in a untrusted/hostile environment should monitor the usage of the 4 different WMM classes and step up their class accordingly. That is in an environment where there is a lot of AC_VO or AC_VI traffic the AP should elevate its normal data packets priority to match as not too be drowned out by the other senders. Sort of a reciprocal tit-for-tat approach, with the goal that the AP will keep access to a decent share of airtime. But since I am a layman in these matters, I might be out to lunch on this…

At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.

I think the challenge might be what metric to use to know when classes are being abused. Maybe roughly if dscp_value * bytes_transferred exceeds some threshold in some given period of time, that would work. Best effort wouldn’t affect this metric so they can use this class all they want, and if someone just uses their connection for typical VoIP usage, the threshold shouldn’t be exceeded. Only when a lot of data is transferred per period of time in higher classes would they exceed the threshold.

For now, we could just measure this (with iptables?) and send an admin email when the threshold is exceeded, then automate a strategy to combat it if needed. But I bet most users in a community WISP, if notified, would just try to help fix the problem… :)

Pete

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

The approach that's in all of the Cisco documentation (FWIW) about such
things for wired networks is that the higher-priority traffic classes for
VoIP and video are also bandwidth limited to a fraction of the total (and
less than a majority, at that).  But that's in an environment where you
_can_guarantee a minimum level of service.  With the changing throughput
rates of wifi, that's a bit harder.

But I can certainly see the case being made that the VO and VI queues are
never allowed to be over X% of traffic.

-Aaron

On Thu, Feb 16, 2017 at 1:17 AM, Pete Heist <peteheist@gmail.com> wrote:

>
> > On Feb 16, 2017, at 9:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> >
> >> On Feb 16, 2017, at 08:57, Pete Heist <peteheist@gmail.com> wrote:
> >> [… discussion about DSCP to WMM classes mapping]
> >> This always makes me wonder what’s to keep someone from just marking
> all their traffic 0x7 and stomping over everyone else.
> >
> >       I have a gut feeling that an AP in a untrusted/hostile environment
> should monitor the usage of the 4 different WMM classes and step up their
> class accordingly. That is in an environment where there is a lot of AC_VO
> or AC_VI traffic the AP should elevate its normal data packets priority to
> match as not too be drowned out by the other senders. Sort of a reciprocal
> tit-for-tat approach, with the goal that the AP will keep access to a
> decent share of airtime. But since I am a layman in these matters, I might
> be out to lunch on this…
>
> At first I was thinking to just remove diffserv markings entirely, say
> with Cake’s besteffort flag, but I think that “good” and “otherwise
> unknowing” users would suffer, which I think in FreeNet is a vast majority
> of users.
>
> I think the challenge might be what metric to use to know when classes are
> being abused. Maybe roughly if dscp_value * bytes_transferred exceeds some
> threshold in some given period of time, that would work. Best effort
> wouldn’t affect this metric so they can use this class all they want, and
> if someone just uses their connection for typical VoIP usage, the threshold
> shouldn’t be exceeded. Only when a lot of data is transferred per period of
> time in higher classes would they exceed the threshold.
>
> For now, we could just measure this (with iptables?) and send an admin
> email when the threshold is exceeded, then automate a strategy to combat it
> if needed. But I bet most users in a community WISP, if notified, would
> just try to help fix the problem… :)
>
> Pete
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>

[-- Attachment #2: Type: text/html, Size: 3531 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Sebastian Moeller]

> On Feb 16, 2017, at 17:15, Aaron Wood <woody77@gmail.com> wrote:
> 
> The approach that's in all of the Cisco documentation (FWIW) about such things for wired networks is that the higher-priority traffic classes for VoIP and video are also bandwidth limited to a fraction of the total (and less than a majority, at that).  But that's in an environment where you _can_guarantee a minimum level of service.  With the changing throughput rates of wifi, that's a bit harder.
> 
> But I can certainly see the case being made that the VO and VI queues are never allowed to be over X% of traffic.

	I guess the problem is that any station can just decide by itself to just send AC_VO and in a typical home steup the AP will not get a say in that. This is why I propose the AP to escalate its own priority marking to get its packets distributed… In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 

Best Regards

> 
> -Aaron
> 
> On Thu, Feb 16, 2017 at 1:17 AM, Pete Heist <peteheist@gmail.com> wrote:
> 
> > On Feb 16, 2017, at 9:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> >
> >> On Feb 16, 2017, at 08:57, Pete Heist <peteheist@gmail.com> wrote:
> >> [… discussion about DSCP to WMM classes mapping]
> >> This always makes me wonder what’s to keep someone from just marking all their traffic 0x7 and stomping over everyone else.
> >
> >       I have a gut feeling that an AP in a untrusted/hostile environment should monitor the usage of the 4 different WMM classes and step up their class accordingly. That is in an environment where there is a lot of AC_VO or AC_VI traffic the AP should elevate its normal data packets priority to match as not too be drowned out by the other senders. Sort of a reciprocal tit-for-tat approach, with the goal that the AP will keep access to a decent share of airtime. But since I am a layman in these matters, I might be out to lunch on this…
> 
> At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.
> 
> I think the challenge might be what metric to use to know when classes are being abused. Maybe roughly if dscp_value * bytes_transferred exceeds some threshold in some given period of time, that would work. Best effort wouldn’t affect this metric so they can use this class all they want, and if someone just uses their connection for typical VoIP usage, the threshold shouldn’t be exceeded. Only when a lot of data is transferred per period of time in higher classes would they exceed the threshold.
> 
> For now, we could just measure this (with iptables?) and send an admin email when the threshold is exceeded, then automate a strategy to combat it if needed. But I bet most users in a community WISP, if notified, would just try to help fix the problem… :)
> 
> Pete
> 
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
> 

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

> On Feb 16, 2017, at 5:21 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> 
>> On Feb 16, 2017, at 17:15, Aaron Wood <woody77@gmail.com> wrote:
>> 
>> The approach that's in all of the Cisco documentation (FWIW) about such things for wired networks is that the higher-priority traffic classes for VoIP and video are also bandwidth limited to a fraction of the total (and less than a majority, at that).  But that's in an environment where you _can_guarantee a minimum level of service.  With the changing throughput rates of wifi, that's a bit harder.
>> 
>> But I can certainly see the case being made that the VO and VI queues are never allowed to be over X% of traffic.
> 
> 	I guess the problem is that any station can just decide by itself to just send AC_VO and in a typical home steup the AP will not get a say in that. This is why I propose the AP to escalate its own priority marking to get its packets distributed… In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 

I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.

Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.

Sebastian, I wasn’t sure what you meant by the AP’s “own packets”, ones that originate from the AP? One thing with might happen if you raise some priorities is there may be an arms race, and also once packets are forwarded to an upstream router with an already elevated priority they might unfairly compete there, but maybe you meant only to treat some packets differently in the AP, not to actually change the value in the DSCP field...

Pete

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Jonathan Morton]

> On 16 Feb, 2017, at 18:51, Pete Heist <peteheist@gmail.com> wrote:
> 
> At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.

That’s not what the “besteffort” flag does.  It ignores DSCPs and puts all traffic into a single tin, but doesn’t remove the DSCP marking.

>> In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
> 
> I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
> 
> Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.

This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the bandwidth left spare by other tins’ traffic, but no more than that.

Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.

However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if at all, but it does have implications for the effect of other stations sharing the frequency.

 - Jonathan Morton

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4000 bytes --]


> On Feb 16, 2017, at 6:19 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 16 Feb, 2017, at 18:51, Pete Heist <peteheist@gmail.com> wrote:
>> 
>> At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.
> 
> That’s not what the “besteffort” flag does.  It ignores DSCPs and puts all traffic into a single tin, but doesn’t remove the DSCP marking.

Thanks, I had mixed this with “squash”.

>>> In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
>> 
>> I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
>> 
>> Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.
> 
> This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the bandwidth left spare by other tins’ traffic, but no more than that.
> 
> Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.
> 
> However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if at all, but it does have implications for the effect of other stations sharing the frequency.

Aha, that sounds like just what we need as far as diffserv is concerned! I’ll punt on trying to understand what that means for Wi-Fi just yet, but will come back to it.

Even though I’m sticking to point-to-point Wi-Fi, I would like to use Cake if possible since we’re shaping on external routers, so I’m testing it more extensively (especially vs sfq since that’s what’s in use now) and will add to my tests:

- diffserv markings (‘rrul’, where so far I’ve done only ‘rrul_be’ for simplicity to get my test setup verified)
- flow isolation (triple-isolate), by simulating P2P-like flows, maybe with Flent’s rrul_torrent somehow, also on multiple IPs? Will figure it out.
- VoIP (if I can get d-ITG working finally)

This reminds me, I found this location for the latest Cake man page and finally read it in detail, as I should have earlier:

http://static.lochnair.net/bufferbloat/tc-cake.8.html <http://static.lochnair.net/bufferbloat/tc-cake.8.html>

1) Should I be using the Ethernet keyword when running Cake on routers connected to the AP (or station) via Ethernet? And overheads for Wi-Fi also, is that even possible to get right, or if not, what’s closest?

2) Is there a more official link for the man page? The one installed with the source from here (git://kau.toke.dk/cake/iproute2/ <https://github.com/dtaht/sch_cake>) seems older.

Thanks for taking the time to explain!


[-- Attachment #2: Type: text/html, Size: 5369 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4896 bytes --]


> On Feb 16, 2017, at 8:05 PM, Pete Heist <peteheist@gmail.com> wrote:
> 
>> 
>> On Feb 16, 2017, at 6:19 PM, Jonathan Morton <chromatix99@gmail.com <mailto:chromatix99@gmail.com>> wrote:
>> 
>>> On 16 Feb, 2017, at 18:51, Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> wrote:
>>> 
>>> At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.
>> 
>> That’s not what the “besteffort” flag does.  It ignores DSCPs and puts all traffic into a single tin, but doesn’t remove the DSCP marking.
> 
> Thanks, I had mixed this with “squash”.
> 
>>>> In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
>>> 
>>> I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
>>> 
>>> Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.
>> 
>> This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the bandwidth left spare by other tins’ traffic, but no more than that.
>> 
>> Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.
>> 
>> However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if at all, but it does have implications for the effect of other stations sharing the frequency.
> 
> Aha, that sounds like just what we need as far as diffserv is concerned! I’ll punt on trying to understand what that means for Wi-Fi just yet, but will come back to it.
> 
> Even though I’m sticking to point-to-point Wi-Fi, I would like to use Cake if possible since we’re shaping on external routers, so I’m testing it more extensively (especially vs sfq since that’s what’s in use now) and will add to my tests:
> 
> - diffserv markings (‘rrul’, where so far I’ve done only ‘rrul_be’ for simplicity to get my test setup verified)
> - flow isolation (triple-isolate), by simulating P2P-like flows, maybe with Flent’s rrul_torrent somehow, also on multiple IPs? Will figure it out.
> - VoIP (if I can get d-ITG working finally)
> 
> This reminds me, I found this location for the latest Cake man page and finally read it in detail, as I should have earlier:
> 
> http://static.lochnair.net/bufferbloat/tc-cake.8.html <http://static.lochnair.net/bufferbloat/tc-cake.8.html>
> 
> 1) Should I be using the Ethernet keyword when running Cake on routers connected to the AP (or station) via Ethernet? And overheads for Wi-Fi also, is that even possible to get right, or if not, what’s closest?
> 
> 2) Is there a more official link for the man page? The one installed with the source from here (git://kau.toke.dk/cake/iproute2/ <https://github.com/dtaht/sch_cake>) seems older.
> 
> Thanks for taking the time to explain!


3) And a followup question from this statement on the man page:

"CAKE can divide traffic into "tins" based on the Diffserv field. Each tin has its own independent set of flow-isolation queues, and is serviced based on a WRR algorithm. To avoid perverse Diffserv marking incentives, tin weights have a "priority sharing" value when bandwidth used by that tin is below a threshold, and a lower "bandwidth sharing" value when above. “

If each diffserv tin has its own flow isolation queues, doesn’t that mean (if this is run on a backhaul router) that if one host is abusing the VoIP diffserv marking, for example, that they’ll impact other users by using up the bandwidth in the tin?

[-- Attachment #2: Type: text/html, Size: 12281 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Sebastian Moeller]

> On Feb 16, 2017, at 18:19, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> 
>> On 16 Feb, 2017, at 18:51, Pete Heist <peteheist@gmail.com> wrote:
>> 
>> At first I was thinking to just remove diffserv markings entirely, say with Cake’s besteffort flag, but I think that “good” and “otherwise unknowing” users would suffer, which I think in FreeNet is a vast majority of users.
> 
> That’s not what the “besteffort” flag does.  It ignores DSCPs and puts all traffic into a single tin, but doesn’t remove the DSCP marking.
> 
>>> In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
>> 
>> I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
>> 
>> Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.
> 
> This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the bandwidth left spare by other tins’ traffic, but no more than that.
> 
> Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.
> 
> However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if at all, but it does have implications for the effect of other stations sharing the frequency.

	Here is part of the problem: the more aggressive airtime access of the VI and VO classes will massively cut down the actual achievable bandwidth over all classes. And I might add this effect is not restricted to the AP and station under one’s control, but all other stations and APs using the same frequency that are in the close RF neighborhood will affect the available airtime and hence achievable bandwidth. If you look how wifi achieves its higher bandwidth it is by using longer periods of airtime to make up for the rather fixed time “preamble” that wastes time without transmission of user data. VI users in the vicinity will drastically (IIRC) reduce the ability to send those aggregates. In other words link saturation is partly a function of which AC classes are used and not a nice and fixed entity as for typical wired connections. Now if you can control both sides of your transfer _and_ all other users of the same frequency that are RF-visible to your endpoints, it might be possible to thnk of a wifi link in similar terms as a wired one, but I would be cautious…

Best Regards
       Sebastian





> 
> - Jonathan Morton
> 

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4450 bytes --]


> On Feb 16, 2017, at 10:03 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> On Feb 16, 2017, at 18:19, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>>> In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
>>> 
>>> I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
>>> 
>>> Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.
>> 
>> This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the bandwidth left spare by other tins’ traffic, but no more than that.
>> 
>> Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.
>> 
>> However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if at all, but it does have implications for the effect of other stations sharing the frequency.
> 
> 	Here is part of the problem: the more aggressive airtime access of the VI and VO classes will massively cut down the actual achievable bandwidth over all classes. And I might add this effect is not restricted to the AP and station under one’s control, but all other stations and APs using the same frequency that are in the close RF neighborhood will affect the available airtime and hence achievable bandwidth. If you look how wifi achieves its higher bandwidth it is by using longer periods of airtime to make up for the rather fixed time “preamble” that wastes time without transmission of user data. VI users in the vicinity will drastically (IIRC) reduce the ability to send those aggregates. In other words link saturation is partly a function of which AC classes are used and not a nice and fixed entity as for typical wired connections. Now if you can control both sides of your transfer _and_ all other users of the same frequency that are RF-visible to your endpoints, it might be possible to thnk of a wifi link in similar terms as a wired one, but I would be cautious…

Thanks for that info. In my testing I’m focusing on point-to-point Wi-Fi, but I see the complexity that WMM presents, especially when there's more than one station.

It's perplexing that at least two concerns- packet retransmission and prioritization, occur at multiple layers in the stack. 802.11 ack frames are sent in response to every data frame (aggregation aside), and retransmission occurs at this layer, but also higher up in the TCP layer. Prioritization can occur at the IP layer, but then again in the media layer with WMM. This seems to violate separation of concerns, and makes it difficult to ascertain and control what’s going on in the system as a whole.

It feels like WMM went a step too far. There may have been (may still be) valid performance reasons for Wi-Fi to take on such concerns, but as the data rates get higher and processing power increases, it feels like it would be better to have a wireless technology that just delivers frames, and to push reliability, prioritization and aggregation back up into the higher layers so that long-term innovation can take place in software. The 802.11 spec is on my reading list so I might learn if and where this goes off the tracks.

[-- Attachment #2: Type: text/html, Size: 6964 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Feb 16, 2017, at 10:03 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>
>  On Feb 16, 2017, at 18:19, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>  In a sense if there are thresholds for permissible VO/VI traffic fractions below which the AP will not escalate its own priority this will come close to throttling the high priority senders, no? 
>
>  I thought Aaron’s suggestion sounds both sensible and not difficult to implement. That way we wouldn’t even have to regularly monitor it, and anyone who is marking all their packets thinking they’re doing themselves a favor is just limiting their max throughput.
>
>  Could there be another keyword in Cake to do this automatically, say “fairdiffserv", or would this just be feature bloat for what is already a sophisticated shaper? I don’t know if there are sensible mappings from dscp value to max percentage throughput that would work most of
>  the time, or if there could also be an adjustable curve parameter that controls the percentage backoff as you go up dscp levels.
>
>  This is actually what Cake already does by default (the “diffserv3” mode).  If you look at the detailed statistics (tc -s qdisc), you’ll see that each tin has a “threshold” bandwidth.  If there’s more traffic than that threshold in that tin, the tin will be deprioritised - it can still use all of the
>  bandwidth left spare by other tins’ traffic, but no more than that.
>
>  Additionally, diffserv3 mode uses more aggressive AQM settings on the “voice” tin than the “best effort” tin, on the grounds that the former is a request for minimum latency.  This should also discourage bulk traffic from using unnecessarily high DSCPs.
>
>  However, in both the “besteffort” and “diffserv3” cases, the DSCP may be interpreted independently by the NIC as well as Cake.  In the case of wifi, this affects the medium grant latency and priority.  If the link isn’t saturated, this shouldn’t affect Cake’s prioritisation strategy much if
>  at all, but it does have implications for the effect of other stations sharing the frequency.
>
>  Here is part of the problem: the more aggressive airtime access of the VI and VO classes will massively cut down the actual achievable bandwidth over all classes. And I might add this effect is not restricted to the AP and station under one’s control, but all other stations and APs using
>  the same frequency that are in the close RF neighborhood will affect the available airtime and hence achievable bandwidth. If you look how wifi achieves its higher bandwidth it is by using longer periods of airtime to make up for the rather fixed time “preamble” that wastes time without
>  transmission of user data. VI users in the vicinity will drastically (IIRC) reduce the ability to send those aggregates. In other words link saturation is partly a function of which AC classes are used and not a nice and fixed entity as for typical wired connections. Now if you can control both
>  sides of your transfer _and_ all other users of the same frequency that are RF-visible to your endpoints, it might be possible to thnk of a wifi link in similar terms as a wired one, but I would be cautious…
>
> Thanks for that info. In my testing I’m focusing on point-to-point Wi-Fi, but I see the complexity that WMM presents, especially when there's more than one station.
>
> It's perplexing that at least two concerns- packet retransmission and prioritization, occur at multiple layers in the stack. 802.11 ack frames are sent in response to every data frame (aggregation aside), and retransmission occurs at this layer, but also higher up in the TCP layer. Prioritization
> can occur at the IP layer, but then again in the media layer with WMM. This seems to violate separation of concerns, and makes it difficult to ascertain and control what’s going on in the system as a whole.
>
> It feels like WMM went a step too far. There may have been (may still be) valid performance reasons for Wi-Fi to take on such concerns, but as the data rates get higher and processing power increases, it feels like it would be better to have a wireless technology that just delivers frames,
> and to push reliability, prioritization and aggregation back up into the higher layers so that long-term innovation can take place in software. The 802.11 spec is on my reading list so I might learn if and where this goes off the tracks.

Note that WMM also affects max aggregation sizes; the VO queue doesn't
do aggregation at all, for instance; and the max aggregate size for VI
is smaller than for BE. This *should* be an incentive to not use the
higher queues for bulk traffic.

That being said, I do believe there are issues with how the QoS levels
are currently handled in the Linux WiFi stack, and looking into that in
more detail is on my list somewhere... :)


-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

In terms of disabling classification and always using best effort,
always passing NULL here to the classifier will do the trick.

http://lxr.free-electrons.com/source/net/mac80211/wme.c#L218

At first glance it does not appear to have *any* default, which
implies that rrul and rrul_be will be equivalent. Which they aren't.

The qos_map facility is exposed to userspace via hostapd via the
qos_map parameter, but my eyes crossed on how to set it properly.

https://w1.fi/cgit/hostap/plain/hostapd/hostapd.conf

How it gets set for clients appears to be in wpa_supplicant but not iw.

http://w1.fi/cgit/hostap/tree/wpa_supplicant/events.c

I leave further exploration of the right string of bits to do the
right thing to y'all...

(for example, I actually might try to move voice and videoconferencing
(AF41 is what google is using for "goog" congestion control in webrtc)
traffic to the VI queue, but disable VO and BK entirely)...

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 5510 bytes --]


> On Jan 30, 2017, at 10:44 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Oh my, this is quite a lot of tests. Nice :)

It’s also a thumbs up for the ath9k driver changes that nothing went wrong during the testing. It takes about 15 hours for a full run and I probably did that 4-5 times total.

> Few general points on running tests:
> 
> - Yeah, as you note Flent has a batch facility. Did you not use this
>  simply because you couldn't find it, or was there some other reason?
>  Would love some feedback on what I can do to make that more useful to
>  people... While I have no doubt that your 'flenter.py' works, wrapping
>  a wrapper in this sense makes me cringe a little bit ;)

I actually didn’t notice it existed until I was about 85% done and scanning the Flent man page for some other reason. I cringed, but at that point I just stuck with what I had. I don’t know if Flent can also make some basic html report with the graphs and setup output, but that was useful to write for myself. Flent’s metadata feature sounds useful and I’ll try that.

> - I'm not sure if you're checking that applying your qdiscs actually
>  works? For the WiFi interfaces with 'noqueue' you *cannot* apply a
>  different qdisc (which also answers your question #2).

Hmm. Unless I’m missing something, what I’m seeing is that I _can_ add another qdisc, only that it’s ineffective unless soft rate limiting is used. As evidence, here's my nolimit test of fq_codel:

http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html <http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html>

Under the sections qos_om1 and qos_om2, you can see that fq_codel has been added to wlan0 from the tc output. But the latency is the same 25 ms as the default, so I’m not controlling the queue, and that was true for any qdisc without rate limiting.

But, here's ‘40mbit full-duplex rate limiting on the AP and station with htb+fq_codel’:

http://www.drhleny.cz/bufferbloat/fq_codel_fd-wifi-both_40mbit/index.html <http://www.drhleny.cz/bufferbloat/fq_codel_fd-wifi-both_40mbit/index.html>

With this, I could reduce latency to about 9ms, so it did “something”. And pfifo with 40mbit rate limiting did the “something” that it usually does, bloating things up to 200+ ms:

http://www.drhleny.cz/bufferbloat/pfifo_fd-wifi-both_40mbit/index.html <http://www.drhleny.cz/bufferbloat/pfifo_fd-wifi-both_40mbit/index.html>

So I don’t see any evidence that I can’t add a qdisc to the Wi-Fi devices, only that I have to use soft limiting for it to be effective.

Now, HTB rate limiting seems to break down on the OM2P-HS at bitrates above about 50mbit, when the actual throughput starts to fall away from the set limit, but that’s another story.

> Question 1 (and partly #13): Yeah, the version of LEDE you're running
> already has the FQ-CoDel-based queueing in the ath9k driver. The
> baseline you're seeing is consistent with the results we've been getting
> in testing. This is also seen by any gains you get being paired with
> quite a hefty hit in throughput. So with this driver, I would say it's
> not worth it. However, this is going to be different on a setup without
> the WiFi queueing fixes.

Ok, that explains a lot, thanks. I was still able to see about a 50% reduction in latency (from ~25 ms to ~12ms) with a 13% drop in throughput (from ~92 Mbps to ~80Mbps), when doing half-duplex rate limiting to 85Mbps and fq_codel’ing on the external router. See:

http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth-ap_85mbit/index.html <http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth-ap_85mbit/index.html>

vs the default:

http://www.drhleny.cz/bufferbloat/default/index.html <http://www.drhleny.cz/bufferbloat/default/index.html>

I can get down to 10ms if I give up another 5 Mbps, or lower values with more severe throughput sacrifices.

But this is with a stable RSSI of around -50 and low noise. I understand that fq-codel’ing in the driver must be superior in its handling of rate changes, retries or other external factors, and that point-to-multipoint is a different story. But maybe some of FreeNet’s line-of-sight point-to-point links may also be stable enough such that fixed software rate limiting is usable for them, I’m not sure yet.

It’s not critical, but why am I able to see this level of reduction when there’s already fq-codel in the driver? 25ms is very good, I only wonder where I’m getting the extra 10-15ms from, out of interest. :)

> Question 5: For TCP you can't get packet loss from user space; you'll
> need packet captures for that. So no way to get it from Flent either.
> You can, however, get average throughput. Look at the box plots; if you
> run multiple iterations of the same test, you can plot several data
> files in a single box_combine plot, to get error bars. `flent
> file.flent.gz -f summary` (which is the default if you don't specify a
> plot) will get you averages per data series; or you can extract it from
> the metadata.

Ok, so far, I was doing `cat file.flent.gz | grep null | wc -l`, which is a very crude count of the nulls recorded, which seem to happen for the udp and icmp flows with packet loss. There are always some nulls from before the test starts and after it ends, but if the count jumps up I speculate that there’s more packet loss. It’s pretty weak but it’s a hint.

I see the box plots now, that’s nice, thanks!


[-- Attachment #2: Type: text/html, Size: 7318 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Jan 30, 2017, at 10:44 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Oh my, this is quite a lot of tests. Nice :)
>
> It’s also a thumbs up for the ath9k driver changes that nothing went
> wrong during the testing. It takes about 15 hours for a full run and I
> probably did that 4-5 times total.

Cool, thanks for confirming that :)
>
>  Few general points on running tests:
>
>  - Yeah, as you note Flent has a batch facility. Did you not use this
>   simply because you couldn't find it, or was there some other reason?
>   Would love some feedback on what I can do to make that more useful to
>   people... While I have no doubt that your 'flenter.py' works, wrapping
>   a wrapper in this sense makes me cringe a little bit ;)
>
> I actually didn’t notice it existed until I was about 85% done and
> scanning the Flent man page for some other reason. I cringed, but at
> that point I just stuck with what I had. I don’t know if Flent can
> also make some basic html report with the graphs and setup output, but
> that was useful to write for myself. Flent’s metadata feature sounds
> useful and I’ll try that.

Right, so first thing is advertising it better. I'll look into that;
really do need to freshen up the web site some...

I will look over your automation script in more detail and see if
there's anything that might be worth adding to Flent's capabilities. Not
sure if HTML report generation can be generalised sufficiently to be
useful outside your specific scenario, but I'll think it over :)

>  - I'm not sure if you're checking that applying your qdiscs actually
>   works? For the WiFi interfaces with 'noqueue' you *cannot* apply a
>   different qdisc (which also answers your question #2).
>
> Hmm. Unless I’m missing something, what I’m seeing is that I _can_ add
> another qdisc, only that it’s ineffective unless soft rate limiting is
> used. As evidence, here's my nolimit test of fq_codel:
>
> http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html

Ah, totally missed that the qdisc information was available on those
pages as well; guess my scroll bar must have been broken. ;)

And yeah, just verified that you can indeed install a qdisc on a noqueue
device; how odd.

>  Question 1 (and partly #13): Yeah, the version of LEDE you're running
>  already has the FQ-CoDel-based queueing in the ath9k driver. The
>  baseline you're seeing is consistent with the results we've been getting
>  in testing. This is also seen by any gains you get being paired with
>  quite a hefty hit in throughput. So with this driver, I would say it's
>  not worth it. However, this is going to be different on a setup without
>  the WiFi queueing fixes.
>
> Ok, that explains a lot, thanks. I was still able to see about a 50%
> reduction in latency (from ~25 ms to ~12ms) with a 13% drop in
> throughput (from ~92 Mbps to ~80Mbps), when doing half-duplex rate
> limiting to 85Mbps and fq_codel’ing on the external router. See:
>
> http://www.drhleny.cz/bufferbloat/fq_codel_hd-eth-ap_85mbit/index.html
>
> vs the default:
>
> http://www.drhleny.cz/bufferbloat/default/index.html
>
> I can get down to 10ms if I give up another 5 Mbps, or lower values
> with more severe throughput sacrifices.
>
> But this is with a stable RSSI of around -50 and low noise. I
> understand that fq-codel’ing in the driver must be superior in its
> handling of rate changes, retries or other external factors, and that
> point-to-multipoint is a different story. But maybe some of FreeNet’s
> line-of-sight point-to-point links may also be stable enough such that
> fixed software rate limiting is usable for them, I’m not sure yet.

Yeah, whether that is worth it depends on your requirements of course,
and how stable your link is. That's very much a YMMV issue, I think.

> It’s not critical, but why am I able to see this level of reduction
> when there’s already fq-codel in the driver? 25ms is very good, I only
> wonder where I’m getting the extra 10-15ms from, out of interest. :)

The driver queues up two aggregates beneath the queue to keep the
hardware busy. It may be possible to improve slightly upon this, but we
have not gotten around to trying yet.

>  Question 5: For TCP you can't get packet loss from user space; you'll
>  need packet captures for that. So no way to get it from Flent either.
>  You can, however, get average throughput. Look at the box plots; if you
>  run multiple iterations of the same test, you can plot several data
>  files in a single box_combine plot, to get error bars. `flent
>  file.flent.gz -f summary` (which is the default if you don't specify a
>  plot) will get you averages per data series; or you can extract it from
>  the metadata.
>
> Ok, so far, I was doing `cat file.flent.gz | grep null | wc -l`, which
> is a very crude count of the nulls recorded, which seem to happen for
> the udp and icmp flows with packet loss. There are always some nulls
> from before the test starts and after it ends, but if the count jumps
> up I speculate that there’s more packet loss. It’s pretty weak but
> it’s a hint.

This is very unlikely to get you anything resembling a right answer. The
null values recorded by Flent are simply *sampling periods* without any
output from netperf. There will be a bunch of those at the start or end
of each test, and there can be other reasons apart from packet loss that
will give null values.

You can get packet loss for ICMP by looking at the sequence numbers in
the raw_values object in the data file (you can get that as a Python
object by doing 'from flent import resultset; r =
resultset.load(filename)' and poking around in that object). Don't think
there's currently a way to export that as a loss measure...

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 5212 bytes --]


> On Feb 1, 2017, at 3:48 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> writes:
> 
>> Few general points on running tests:
>> 
>> - Yeah, as you note Flent has a batch facility. Did you not use this
>>  simply because you couldn't find it, or was there some other reason?
>>  Would love some feedback on what I can do to make that more useful to
>>  people... While I have no doubt that your 'flenter.py' works, wrapping
>>  a wrapper in this sense makes me cringe a little bit ;)
>> 
>> I actually didn’t notice it existed until I was about 85% done and
>> scanning the Flent man page for some other reason. I cringed, but at
>> that point I just stuck with what I had. I don’t know if Flent can
>> also make some basic html report with the graphs and setup output, but
>> that was useful to write for myself. Flent’s metadata feature sounds
>> useful and I’ll try that.
> 
> Right, so first thing is advertising it better. I'll look into that;
> really do need to freshen up the web site some...
> 
> I will look over your automation script in more detail and see if
> there's anything that might be worth adding to Flent's capabilities. Not
> sure if HTML report generation can be generalised sufficiently to be
> useful outside your specific scenario, but I'll think it over :)

I’m not particularly fond of anything in flenter.py, as it’s not very generic. If there’s ever HTML generation support in flent, it could be by processing a user supplied template and offering tags that are replaced with various things. I used templates in flenter.py, but I was also hardcoding HTML in my script, so it was just a “get it done” piece of work that doesn’t appeal to the engineer in me.

I’m more familiar with Go these days, which has a robust template package. Earlier I wondered if a combined netperf / flent tool could be written in golang (not volunteering though at this point :). It could be nice to have native binaries for different platforms, and goroutines might come in handy for the actually testing of multiple flows, but if there are low-level calls happening in netperf, you’d end up using cgo, which of course makes portability harder.

>> Hmm. Unless I’m missing something, what I’m seeing is that I _can_ add
>> another qdisc, only that it’s ineffective unless soft rate limiting is
>> used. As evidence, here's my nolimit test of fq_codel:
>> 
>> http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html <http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html>
> 
> Ah, totally missed that the qdisc information was available on those
> pages as well; guess my scroll bar must have been broken. ;)
> 
> And yeah, just verified that you can indeed install a qdisc on a noqueue
> device; how odd.

I should test Chaos Calmer and make clearer in the paper the separation of when I’m testing the new ath9k driver changes, and when I’m testing the qdisc layer. It will need quite a re-org.

>> It’s not critical, but why am I able to see this level of reduction
>> when there’s already fq-codel in the driver? 25ms is very good, I only
>> wonder where I’m getting the extra 10-15ms from, out of interest. :)
> 
> The driver queues up two aggregates beneath the queue to keep the
> hardware busy. It may be possible to improve slightly upon this, but we
> have not gotten around to trying yet.

Ok, if rtt were about half of 25ms there would be almost no argument for external rate limiting. Even as it is now, I question what difference the user sees between 12ms and 25ms latency for Internet traffic. It also makes me more interested to see results for Chaos Calmer with fq_codel applied on the Wi-Fi device without limiting.

>> Ok, so far, I was doing `cat file.flent.gz | grep null | wc -l`, which
>> is a very crude count of the nulls recorded, which seem to happen for
>> the udp and icmp flows with packet loss. There are always some nulls
>> from before the test starts and after it ends, but if the count jumps
>> up I speculate that there’s more packet loss. It’s pretty weak but
>> it’s a hint.
> 
> This is very unlikely to get you anything resembling a right answer. The
> null values recorded by Flent are simply *sampling periods* without any
> output from netperf. There will be a bunch of those at the start or end
> of each test, and there can be other reasons apart from packet loss that
> will give null values.
> 
> You can get packet loss for ICMP by looking at the sequence numbers in
> the raw_values object in the data file (you can get that as a Python
> object by doing 'from flent import resultset; r =
> resultset.load(filename)' and poking around in that object). Don't think
> there's currently a way to export that as a loss measure...

Understood. :) Somehow though the losses look higher for the UDP flows than ICMP, if I take this test as an example (pfifo limit 32):

http://www.drhleny.cz/bufferbloat/pfifo_hd-wifi-ap_limit_32_80mbit/index.html

The UDP lines end sooner than the ICMP line. I take that to mean there wasn’t as much loss of ICMP?


[-- Attachment #2: Type: text/html, Size: 26923 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Feb 1, 2017, at 3:48 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Pete Heist <peteheist@gmail.com> writes:
>
>  Few general points on running tests:
>
>  - Yeah, as you note Flent has a batch facility. Did you not use this
>   simply because you couldn't find it, or was there some other reason?
>   Would love some feedback on what I can do to make that more useful to
>   people... While I have no doubt that your 'flenter.py' works, wrapping
>   a wrapper in this sense makes me cringe a little bit ;)
>
>  I actually didn’t notice it existed until I was about 85% done and
>  scanning the Flent man page for some other reason. I cringed, but at
>  that point I just stuck with what I had. I don’t know if Flent can
>  also make some basic html report with the graphs and setup output, but
>  that was useful to write for myself. Flent’s metadata feature sounds
>  useful and I’ll try that.
>
>  Right, so first thing is advertising it better. I'll look into that;
>  really do need to freshen up the web site some...
>
>  I will look over your automation script in more detail and see if
>  there's anything that might be worth adding to Flent's capabilities. Not
>  sure if HTML report generation can be generalised sufficiently to be
>  useful outside your specific scenario, but I'll think it over :)
>
> I’m not particularly fond of anything in flenter.py, as it’s not very
> generic. If there’s ever HTML generation support in flent, it could be
> by processing a user supplied template and offering tags that are
> replaced with various things. I used templates in flenter.py, but I
> was also hardcoding HTML in my script, so it was just a “get it done”
> piece of work that doesn’t appeal to the engineer in me.

Right, a template might work. But not sure how generic the use case is,
or whether another mechanism might be better (such as a separate tool
that could keep track of data files; maybe as a web-based database or
something).

> I’m more familiar with Go these days, which has a robust template
> package. Earlier I wondered if a combined netperf / flent tool could
> be written in golang (not volunteering though at this point :). It
> could be nice to have native binaries for different platforms, and
> goroutines might come in handy for the actually testing of multiple
> flows, but if there are low-level calls happening in netperf, you’d
> end up using cgo, which of course makes portability harder.

Yeah, I have recently begun learning Go myself, and like it too. Apart
from the fact that it produces these huge statically linked binaries,
and requires glibc, so you can't run it on embedded systems (such as
LEDE).

If I were to integrate code that actually shipped packets into Flent, I
would probably use Cython...

>
>  Hmm. Unless I’m missing something, what I’m seeing is that I _can_ add
>  another qdisc, only that it’s ineffective unless soft rate limiting is
>  used. As evidence, here's my nolimit test of fq_codel:
>
>  http://www.drhleny.cz/bufferbloat/fq_codel_nolimit/index.html
>
>  Ah, totally missed that the qdisc information was available on those
>  pages as well; guess my scroll bar must have been broken. ;)
>
>  And yeah, just verified that you can indeed install a qdisc on a noqueue
>  device; how odd.
>
> I should test Chaos Calmer and make clearer in the paper the
> separation of when I’m testing the new ath9k driver changes, and when
> I’m testing the qdisc layer. It will need quite a re-org.

Yeah, that would probably be a good idea :)

>  It’s not critical, but why am I able to see this level of reduction
>  when there’s already fq-codel in the driver? 25ms is very good, I only
>  wonder where I’m getting the extra 10-15ms from, out of interest. :)
>
>  The driver queues up two aggregates beneath the queue to keep the
>  hardware busy. It may be possible to improve slightly upon this, but we
>  have not gotten around to trying yet.
>
> Ok, if rtt were about half of 25ms there would be almost no argument
> for external rate limiting. Even as it is now, I question what
> difference the user sees between 12ms and 25ms latency for Internet
> traffic. It also makes me more interested to see results for Chaos
> Calmer with fq_codel applied on the Wi-Fi device without limiting.

Yup, exactly. We want to get to the point where you'll have no reason to
do any rate limiting.

>  Ok, so far, I was doing `cat file.flent.gz | grep null | wc -l`, which
>  is a very crude count of the nulls recorded, which seem to happen for
>  the udp and icmp flows with packet loss. There are always some nulls
>  from before the test starts and after it ends, but if the count jumps
>  up I speculate that there’s more packet loss. It’s pretty weak but
>  it’s a hint.
>
>  This is very unlikely to get you anything resembling a right answer. The
>  null values recorded by Flent are simply *sampling periods* without any
>  output from netperf. There will be a bunch of those at the start or end
>  of each test, and there can be other reasons apart from packet loss that
>  will give null values.
>
>  You can get packet loss for ICMP by looking at the sequence numbers in
>  the raw_values object in the data file (you can get that as a Python
>  object by doing 'from flent import resultset; r =
>  resultset.load(filename)' and poking around in that object). Don't think
>  there's currently a way to export that as a loss measure...
>
> Understood. :) Somehow though the losses look higher for the UDP flows
> than ICMP, if I take this test as an example (pfifo limit 32):
>
> http://www.drhleny.cz/bufferbloat/pfifo_hd-wifi-ap_limit_32_80mbit/index.html
>
> The UDP lines end sooner than the ICMP line. I take that to mean there
> wasn’t as much loss of ICMP?

Netperf has this tendency to stop the UDP flows if there's packet loss.
If you're on 2.7+ of netperf it should restart after a while, but if it
happens a lot you can still lose quite a few samples...

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]


> Yeah, I have recently begun learning Go myself, and like it too. Apart
> from the fact that it produces these huge statically linked binaries,
> and requires glibc, so you can't run it on embedded systems (such as
> LEDE).
> 
> If I were to integrate code that actually shipped packets into Flent, I
> would probably use Python…

Even after the new SSA back end, they can still be large. I hadn’t thought to run Flent on embedded hardware, so there isn’t a performance impact from running the test code itself on the hardware you’re testing. But that’s true, if it needs to sometimes, then Go doesn’t work.

>> It’s not critical, but why am I able to see this level of reduction
>> when there’s already fq-codel in the driver? 25ms is very good, I only
>> wonder where I’m getting the extra 10-15ms from, out of interest. :)
>> 
>> The driver queues up two aggregates beneath the queue to keep the
>> hardware busy. It may be possible to improve slightly upon this, but we
>> have not gotten around to trying yet.
>> 
>> Ok, if rtt were about half of 25ms there would be almost no argument
>> for external rate limiting. Even as it is now, I question what
>> difference the user sees between 12ms and 25ms latency for Internet
>> traffic. It also makes me more interested to see results for Chaos
>> Calmer with fq_codel applied on the Wi-Fi device without limiting.
> 
> Yup, exactly. We want to get to the point where you'll have no reason to
> do any rate limiting.

That reminds me, is there any way to disable fq-codel in the ath9k driver, and revert to being able to use the qdisc layer without limiting? Then I could do this testing without having to install Chaos Calmer, and it could avoid some re-flashing in case I need to re-test something in the new driver code again.

I picked up 2x NanoStation M5 and 2x of FreeNet’s older APUs (https://pcengines.ch/alix2d2.htm <https://pcengines.ch/alix2d2.htm>), and should add more results soon. I also hope to try some of their more modern hardware, as the APUs in particular are a bit ancient, but I'll see if it becomes available.

[-- Attachment #2: Type: text/html, Size: 8783 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  Yeah, I have recently begun learning Go myself, and like it too. Apart
>  from the fact that it produces these huge statically linked binaries,
>  and requires glibc, so you can't run it on embedded systems (such as
>  LEDE).
>
>  If I were to integrate code that actually shipped packets into Flent, I
>  would probably use Python…
>
> Even after the new SSA back end, they can still be large. I hadn’t
> thought to run Flent on embedded hardware, so there isn’t a
> performance impact from running the test code itself on the hardware
> you’re testing. But that’s true, if it needs to sometimes, then Go
> doesn’t work.

It's not so much running the test *from* the router, as it is having the
server component (netserver) run on a router to test. To do that it'll
have to be C, basically...

>  It’s not critical, but why am I able to see this level of reduction
>  when there’s already fq-codel in the driver? 25ms is very good, I only
>  wonder where I’m getting the extra 10-15ms from, out of interest. :)
>
>  The driver queues up two aggregates beneath the queue to keep the
>  hardware busy. It may be possible to improve slightly upon this, but we
>  have not gotten around to trying yet.
>
>  Ok, if rtt were about half of 25ms there would be almost no argument
>  for external rate limiting. Even as it is now, I question what
>  difference the user sees between 12ms and 25ms latency for Internet
>  traffic. It also makes me more interested to see results for Chaos
>  Calmer with fq_codel applied on the Wi-Fi device without limiting.
>
>  Yup, exactly. We want to get to the point where you'll have no reason to
>  do any rate limiting.
>
> That reminds me, is there any way to disable fq-codel in the ath9k
> driver, and revert to being able to use the qdisc layer without
> limiting? Then I could do this testing without having to install Chaos
> Calmer, and it could avoid some re-flashing in case I need to re-test
> something in the new driver code again.

Nope, no way to turn it off.

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

On Wed, Feb 8, 2017 at 8:11 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Pete Heist <peteheist@gmail.com> writes:
>
>>  Yeah, I have recently begun learning Go myself, and like it too. Apart
>>  from the fact that it produces these huge statically linked binaries,
>>  and requires glibc, so you can't run it on embedded systems (such as
>>  LEDE).
>>
>>  If I were to integrate code that actually shipped packets into Flent, I
>>  would probably use Python…
>>
>> Even after the new SSA back end, they can still be large. I hadn’t
>> thought to run Flent on embedded hardware, so there isn’t a
>> performance impact from running the test code itself on the hardware
>> you’re testing. But that’s true, if it needs to sometimes, then Go
>> doesn’t work.
>
> It's not so much running the test *from* the router, as it is having the
> server component (netserver) run on a router to test. To do that it'll
> have to be C, basically...

I note that I usually run tests *through* the router rather than *to*
the router, and most of my targets for this have evolved to be arm
(odroid c2), and x86 platforms, so I could get past a gbit.

Long ago I started writing a spec for the things that we couldn't
quite do sanely using netperf as a substrate (twd in my github repo),
but I gave up as getting to netperf equivalence was too hard.

Of note I've always regarded exposing d-itg and some sort of monotonic
voip-like test to the internet as too dangerous without a solid 3 way
handshake, and leveraging existing voip and webrtc/videoconferencing
servers (like freeswitch) way too hard to setup and measure. (there
are plenty of tools to generate rtp-like packets).

I am unfond of the current ping and udp tests in the rrul component
because they don't look like these traffic types, and drawing
conclusions from their behavior as if they were is wrong. Also, with
very low RTTs, the measurement flows tend to skew the tests - you'll
consistently see "lower bandwidth" measured when you cut an RTT from
100 to 1ms via active queue management, when what is really happening
is 100x more measurement traffic.

Of the choices today for writing network servers, go appears to be the
leading candidate, and certain things that might affect timings (for
example, stop the world garbage collection) can be turned off during a
test.

Being lazy about writing protocols and wanting to (eventually) also
have a C version, I'd looked over both protobufs, and
https://capnproto.org/ as means to abstract enough of that out to
handle both test negotiation and simulation...

and then I went back to just using trusty old netperf 'cause when you
are getting orders of magnitude improvements throughout the stack it
really didn't matter.

We're now at the point with both cake and fq_codel at the wifi mac
layer, where getting precision timings with variances below a ms is
becoming needed. (I basically ignore anything less than 3ms as noise
presently)...

and for example, I'm anal enough (now) to see this tiny latency spikes
on tests like this on 60 second intervals and wonder where they are
coming from...

http://www.taht.net/~d/worldwide_fairness.png


>>  It’s not critical, but why am I able to see this level of reduction
>>  when there’s already fq-codel in the driver? 25ms is very good, I only
>>  wonder where I’m getting the extra 10-15ms from, out of interest. :)
>>
>>  The driver queues up two aggregates beneath the queue to keep the
>>  hardware busy. It may be possible to improve slightly upon this, but we
>>  have not gotten around to trying yet.
>>
>>  Ok, if rtt were about half of 25ms there would be almost no argument
>>  for external rate limiting. Even as it is now, I question what
>>  difference the user sees between 12ms and 25ms latency for Internet
>>  traffic. It also makes me more interested to see results for Chaos
>>  Calmer with fq_codel applied on the Wi-Fi device without limiting.
>>
>>  Yup, exactly. We want to get to the point where you'll have no reason to
>>  do any rate limiting.
>>
>> That reminds me, is there any way to disable fq-codel in the ath9k
>> driver, and revert to being able to use the qdisc layer without
>> limiting? Then I could do this testing without having to install Chaos
>> Calmer, and it could avoid some re-flashing in case I need to re-test
>> something in the new driver code again.
>
> Nope, no way to turn it off.
>
> -Toke
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

In  terms of getting to a decent one way delay measurements - with
sane authentication - the only tool I found and have played with was

http://software.internet2.edu/owamp/

which has an associated internet draft.

I don't trust it - don't think they use good realtime capabilities and
things like fdtimers - and was unsure if the authentication/crypto was
any good - but it seemed like it could be improved. In order to get
useful data out of it, I ended up parsing the raw output of the
tool...

and you just tempted me to go set it up again....

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

On Wed, Feb 8, 2017 at 9:10 AM, Dave Taht <dave.taht@gmail.com> wrote:
> In  terms of getting to a decent one way delay measurements - with
> sane authentication - the only tool I found and have played with was
>
> http://software.internet2.edu/owamp/
>
> which has an associated internet draft.
>
> I don't trust it - don't think they use good realtime capabilities and
> things like fdtimers - and was unsure if the authentication/crypto was
> any good - but it seemed like it could be improved. In order to get
> useful data out of it, I ended up parsing the raw output of the
> tool...
>
> and you just tempted me to go set it up again...

ah, yea, this is what stopped me.

root@dancer:~/git/iproute2# owampd
owampd[23973]: WARNING: No limits specified.
owampd[23973]: Running owampd as root is folly!
owampd[23973]: Use the -U option! (or allow root with the -f option)


-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4853 bytes --]


> On Feb 8, 2017, at 5:35 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Wed, Feb 8, 2017 at 8:11 AM, Toke Høiland-Jørgensen <toke@toke.dk <mailto:toke@toke.dk>> wrote:
>> 
>> It's not so much running the test *from* the router, as it is having the
>> server component (netserver) run on a router to test. To do that it'll
>> have to be C, basically...
> 
> I note that I usually run tests *through* the router rather than *to*
> the router, and most of my targets for this have evolved to be arm
> (odroid c2), and x86 platforms, so I could get past a gbit.

Me too so far. The OM2P-HS (520 MHz MIPS 74Kc, similar to the NSM5 I’m about to test), was unable to max out the link rate even with iperf3, so I knew I had to run flent on separate hardware. Right now I have a 4 node config:

flent-client/router — station — AP — router/flent-server

When I test the Alix router, which is pretty low end, I’ll go to a 6 node config because I don’t think it will bear running flent while routing and managing the queue:

flent-client — router — station — AP — router — flent-server

It seems that flent should be run on sufficiently overpowered hardware (basically any even semi-modern Intel hardware) relative to what’s being tested. Maybe there are cases where it needs to run on embedded devices that I’m not thinking of.

> Of note I've always regarded exposing d-itg and some sort of monotonic
> voip-like test to the internet as too dangerous without a solid 3 way
> handshake, and leveraging existing voip and webrtc/videoconferencing
> servers (like freeswitch) way too hard to setup and measure. (there
> are plenty of tools to generate rtp-like packets).

I haven’t gotten d-itg to run so far. I haven’t looked into it much, but for some reason it seems like ITGRecv only wants to listen for signaling (port 9000) on tcp6, even with an explicit bind address:

sysadmin@mbp:~$ sudo ITGRecv -a 10.0.1.2
ITGRecv version 2.8.1 (r1023)
Compile-time options: sctp dccp bursty multiport
Press Ctrl-C to terminate
...
sysadmin@mbp:~$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 localhost:domain        0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN     
tcp        0      0 mbp:ssh                 10.72.0.34:60739        ESTABLISHED
tcp        0    172 mbp:ssh                 10.72.0.34:60768        ESTABLISHED
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
tcp6       0      0 [::]:12865              [::]:*                  LISTEN     
tcp6       0      0 [::]:9000               [::]:*                  LISTEN     
udp        0      0 localhost:domain        0.0.0.0:*                          
udp        0      0 0.0.0.0:bootpc          0.0.0.0:*                          

> I am unfond of the current ping and udp tests in the rrul component
> because they don't look like these traffic types, and drawing
> conclusions from their behavior as if they were is wrong. Also, with
> very low RTTs, the measurement flows tend to skew the tests - you'll
> consistently see "lower bandwidth" measured when you cut an RTT from
> 100 to 1ms via active queue management, when what is really happening
> is 100x more measurement traffic.

I’ve been wondering about this- how seriously to take the rtt from rrul tests in general. When I’m making config changes that change the rtt by a few milliseconds here or there, I’m not convinced I’m not missing other externalities, which are so far are probably only accessible by reading the tea leaves. So for example, when sfq and fq_codel seem quite close in my half duplex rate limiting results, at least in terms of average rtt, are they really? Aren't there test cases where I can better demonstrate that fq_codel is “better” than sfq?

Ultimately it will come down to how well fq_codel (or Cake) behaves in the real world, and I’m not even sure how to measure that. In FreeNet, I can get regular ICMP ping results between their routers and look at them after a day’s time, for example, but will I really have everything I need to know I’ve made a difference? And furthermore, if Ubiquiti is prioritizing ICMP (I intend to find that out), are those ICMP results even useful? I may have to measure RTT with regular best-effort UDP packets between the routers, and even then, I want to know when I see latency spikes whether I’m looking at bufferbloat or radio related issues, so I’ll need other stats as well. They have Ubiquiti’s airControl deployed, but my sense is that it’s going to be easy to deploy fq_codel, and not very easy to demonstrate if and how it’s helped!

[-- Attachment #2: Type: text/html, Size: 19349 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

> On Feb 8, 2017, at 5:11 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> That reminds me, is there any way to disable fq-codel in the ath9k
>> driver, and revert to being able to use the qdisc layer without
>> limiting? Then I could do this testing without having to install Chaos
>> Calmer, and it could avoid some re-flashing in case I need to re-test
>> something in the new driver code again.
> 
> Nope, no way to turn it off.


Ok, it would be convenient for testing.

Also, could the queue management code be abstracted into a separate module, so it can be replaced, like a qdisc? I don’t know if the disparity between hardware makes this too difficult or not…

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>> On Feb 8, 2017, at 5:11 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> That reminds me, is there any way to disable fq-codel in the ath9k
>>> driver, and revert to being able to use the qdisc layer without
>>> limiting? Then I could do this testing without having to install Chaos
>>> Calmer, and it could avoid some re-flashing in case I need to re-test
>>> something in the new driver code again.
>> 
>> Nope, no way to turn it off.
>
> Ok, it would be convenient for testing.

Yeah, but "it should be easy to test how crappy the old code was" is not
a very good argument for putting code into the kernel ;)

What you can do is have two different versions of mac80211.ko and
ath9k.ko and reload them between your test runs. There's even code in
Flent to capture the module version which you can use to distinguish
between the cases.

> Also, could the queue management code be abstracted into a separate
> module, so it can be replaced, like a qdisc? I don’t know if the
> disparity between hardware makes this too difficult or not…

Well it's abstracted into mac80211 so all wireless drivers can use it
(theoretically; right now only ath9k, ath10k and the not-yet-merged mt76
do). Making it more generic than that is not possible, since it's tied
to the mac80211 data structures. Which is kinda the point; the whole
problem was that a "generic layer" (the qdisc) didn't work well
enough...

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]


> On Feb 9, 2017, at 2:51 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
>> Also, could the queue management code be abstracted into a separate
>> module, so it can be replaced, like a qdisc? I don’t know if the
>> disparity between hardware makes this too difficult or not…
> 
> Well it's abstracted into mac80211 so all wireless drivers can use it
> (theoretically; right now only ath9k, ath10k and the not-yet-merged mt76
> do). Making it more generic than that is not possible, since it's tied
> to the mac80211 data structures. Which is kinda the point; the whole
> problem was that a "generic layer" (the qdisc) didn't work well
> enough...

Nice. :)

I’ll mention one example I noticed of the kind of adjustment that probably can’t be done from the qdisc layer today. On page 3 from http://pollere.net/Pdfdocs/noteburstymacs.pdf <http://pollere.net/Pdfdocs/noteburstymacs.pdf>:

"In addition to increasing the interval by the waiting delay s, another adjustment might be useful for certain kinds of bursty MACs. If the MAC is a request-and-grant type, as wifi in infrastructure mode, cable modems and some satellite modems, the allocation of bytes or packets that can be sent during each transmission slot is generally known at the beginning of transmission and may vary for each transmission slot. In that case, it MAY be useful to use that value instead of the MTU value to reset first_above_time_."

[-- Attachment #2: Type: text/html, Size: 2202 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Feb 9, 2017, at 2:51 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Also, could the queue management code be abstracted into a separate
>  module, so it can be replaced, like a qdisc? I don’t know if the
>  disparity between hardware makes this too difficult or not…
>
>  Well it's abstracted into mac80211 so all wireless drivers can use it
>  (theoretically; right now only ath9k, ath10k and the not-yet-merged mt76
>  do). Making it more generic than that is not possible, since it's tied
>  to the mac80211 data structures. Which is kinda the point; the whole
>  problem was that a "generic layer" (the qdisc) didn't work well
>  enough...
>
> Nice. :)
>
> I’ll mention one example I noticed of the kind of adjustment that
> probably can’t be done from the qdisc layer today. On page 3 from
> http://pollere.net/Pdfdocs/noteburstymacs.pdf:
>
> "In addition to increasing the interval by the waiting delay s,
> another adjustment might be useful for certain kinds of bursty MACs.
> If the MAC is a request-and-grant type, as wifi in infrastructure
> mode, cable modems and some satellite modems, the allocation of bytes
> or packets that can be sent during each transmission slot is generally
> known at the beginning of transmission and may vary for each
> transmission slot. In that case, it MAY be useful to use that value
> instead of the MTU value to reset first_above_time_."

Yeah, we've had this issue with CoDel when the per-station WiFi rate
drops too low (this is a function of both signal quality and number of
stations). As a temporary fix, I just fiddled with the target until
CoDel stopped being too aggressive, but that is not a proper solution,
and so has not been upstreamed. I'm planning to get back around to
fixing that properly at some point... Problem is that it is not always
the case that "the allocation of bytes or packets that can be sent
during each transmission slot is generally known". For ath9k we could
get at this information at dequeue time, but for ath10k, only the
firmware knows ahead of time...

-Toke

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2303 bytes --]


> On Feb 9, 2017, at 3:44 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <peteheist@gmail.com> writes:
> 
>> I’ll mention one example I noticed of the kind of adjustment that
>> probably can’t be done from the qdisc layer today. On page 3 from
>> http://pollere.net/Pdfdocs/noteburstymacs.pdf:
>> 
>> "In addition to increasing the interval by the waiting delay s,
>> another adjustment might be useful for certain kinds of bursty MACs.
>> If the MAC is a request-and-grant type, as wifi in infrastructure
>> mode, cable modems and some satellite modems, the allocation of bytes
>> or packets that can be sent during each transmission slot is generally
>> known at the beginning of transmission and may vary for each
>> transmission slot. In that case, it MAY be useful to use that value
>> instead of the MTU value to reset first_above_time_."
> 
> Yeah, we've had this issue with CoDel when the per-station WiFi rate
> drops too low (this is a function of both signal quality and number of
> stations). As a temporary fix, I just fiddled with the target until
> CoDel stopped being too aggressive, but that is not a proper solution,
> and so has not been upstreamed. I'm planning to get back around to
> fixing that properly at some point... Problem is that it is not always
> the case that "the allocation of bytes or packets that can be sent
> during each transmission slot is generally known". For ath9k we could
> get at this information at dequeue time, but for ath10k, only the
> firmware knows ahead of time...
> 
> -Toke

Interesting, I hope there’s a solution. Here are the fixed MCS level results I saw yesterday:

http://www.drhleny.cz/bufferbloat/mcstmp/mcs_latency.png <http://www.drhleny.cz/bufferbloat/mcstmp/mcs_latency.png>

http://www.drhleny.cz/bufferbloat/mcstmp/mcs_throughput.png <http://www.drhleny.cz/bufferbloat/mcstmp/mcs_throughput.png>

in case anything can be seen in it related to this. I noticed that for the ath9k driver CoDel latency maxed out at 45 ms at around MCS 10, and didn’t get any higher down to MCS 8.

Overall, the CoDel / Cake algorithms seem to keep rrul latency much lower than sfq at lower bitrates, but the differences are smaller at higher bitrates. More results later...

Pete


[-- Attachment #2: Type: text/html, Size: 3490 bytes --]

Re: [Make-wifi-fast] [Cake] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Wed, Feb 8, 2017 at 10:26 AM, Pete Heist <peteheist@gmail.com> wrote:

> Even after the new SSA back end, they can still be large.
>

​Take if from a long time compiler write who has nearly 30 years experience
working with SSA:  SSA is no panacea.  It is a representation that makes
some optimizations easier to express and sometimes more effective.  Many of
the optimizations that SSA facilitates tend to raise register pressure.  I
can be a very delicate balancing act.

Optimizing for size is a dark art.  Most compiler projects invest
relatively little effort in that direction.​

​/john​

[-- Attachment #2: Type: text/html, Size: 1536 bytes --]

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Dave Taht]

* Question #16: Is there any other testing anyone would like to see
while I have this rig up?

Please!

1) ECN on on both sides.
2) A voip test
3) P2MP (3 or more stations, rtt_fair_var* tests)
4) Lowered MCS rates or distance or rain

Of these, the last will improve the accuracy and relevance of your
testbench results the most.

* Question #15: FreeNet's link to the Internet is 1 Gbps, and AFAIK
does not experience saturation. The e1000 ethernet driver that the
Internet router uses supports BQL. Is there any sense in running
fq_codel or similar on the router, if it does not become saturated?

You don't need queue management until you need queue management. A
basic flaw of mrtg's design in your graph here is that it takes
samples in 5 minute windows. If you are totally nuking your link for 1
sec out of 5 and the rest nearly idle, you won't see it. A flent test
in their offices to somewhere nearby over that link will be revealing.

In general, applying fq_codel on a BQL enabled system is always a
goodness, it costs next to nothing in CPU to do it that way.
(outbound). Depending on your measurement of what happens on inbound,
you might want to do inbound rate shaping....

* Question #14: Not directly related to this testing, how might
Google's new BBR Congestion Control change the approach to managing
bufferbloat in general?

I don't know. I've shown that BBR interacts badly with single queued
AQMs, but fq_codel and cake deal with it without issues.

http://blog.cerowrt.org/flent/bbr-comprehensive/ has some early tests
in it, but I wouldn't call that dataset definitive. In particular on
some tests I was only running BBR in one direction and not the other.

* Question #13: The gains found here are not as high as they typically
are for ADSL, where I have seen a 10x improvement in rtt under load.
The main question- is it worth the extra effort to do rate limiting
and queue management for point-to-point Wi-Fi? Will the gains be
greater when I test with Ubiquiti Wi-Fi hardware, which may or may not
have queue management built in to the driver? Hopefully I'll have more
information on that last question soon.

The gains are relative to the bandwidth and the amount of fixed
buffering in the radio. For example, I can get 320Mbits/sec out of the
archer c7v2's ath10k, with 10ms latency, at 8 feet. 20 feet, and
through a wall, it's 200Mbit/100ms latency. I don't like the initial
shape of that curve! (what I typically see happen on 802.11ac is it
hitting a wall and not working at all)

I happen to like ubnt's gear, although their default firmware only
lasts 5 minutes for me these days. They have very good throughput and
latency characteristics with decent signal strength. I look forward to
a benchmark of what you can get from them. (I am still looking for a
good upgrade from the nanostation M5s)

Question #12: What's the reason for Cake's occasional sudden
throughput shifts, and why do its latencies tend to be higher than
fq_codel?

We are still debugging cake. Recently the API got a bit wacked. The
AQM is tuned for DSL speeds. More data is needed.

* Question #10: I expected the latency to get at least slightly lower
with a lower target/interval. What's the reason it doesn't?

Queue mostly controlled in the radio on your tests, I think

* Question #11: In a multi-hop Wi-Fi link, should target and interval
be tuned only for the current link, or for the nature of the traffic
being carried? In other words, 5ms / 100ms is the default, and
typically recommended for Internet traffic. Should that be kept for a
single, lower-latency intermediate link in an ISPs backhaul because
it's carrying Internet traffic, or should lower values theoretically
be used?

Very good question. For a p2p link terminating two offices, I can see
trying to tune it lower. Don't know. Keep it where it is for "to the
internet". Also multi-hop wifi has only been somewhat explored in a
couple papers so far, my hope was to tackle it again after everything
stablized....

* On the pfifo analysis - I really enjoyed the rush song and it was
very appropriate for how pfifo misbehaves!

* Question #9: Does my assertion make sense, that it's "better" to do
half-duplex queueing on only one end of the link? The side towards the
Internet?

Usually the stuff coming from the internet dominates the traffic by a
8-20 figure, so coming from might be a worthwhile place to shape.

I ran out of time before tackling 8-1....

On Mon, Jan 30, 2017 at 1:21 PM, Pete Heist <peteheist@gmail.com> wrote:
> Hi, I’ve posted some Flent results and analysis for point-to-point Wi-Fi
> using LEDE on OM2P-HS (ath9k):
>
> http://www.drhleny.cz/bufferbloat/wifi_bufferbloat.html
>
> Over 500 runs were done using different configurations, as the effects of
> various changes were explored. In case someone has the time to respond,
> there are a number of questions in red. Whatever feedback I get I’ll try to
> incorporate into a final document. Also, if I’ve made any false assertions
> I’d love to hear about it.
>
> As described on the page, I’m doing this to try to help my cooperative WISP
> here in Czech, and also in case it helps the Bufferbloat project somehow.
> Although much of the new Wi-Fi work is occurring at the driver level, I hope
> it’s not misguided to still explore using the qdisc layer for Wi-Fi,
> particularly in cases where a new Wi-Fi driver can’t be deployed.
>
> These first results are from LEDE on the OM2P-HS, but I hope to get some
> test devices from my WISP soon to test the hardware they’re using (Ubiquiti
> Wi-Fi devices with custom Voyage Linux routers running on a PCengines APU).
>
> Regards,
> Pete
>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] Flent results for point-to-point Wi-Fi on LEDE/OM2P-HS available

[Pete Heist]

> On Jan 31, 2017, at 12:55 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> * Question #16: Is there any other testing anyone would like to see
> while I have this rig up?
> 
> 1) ECN on on both sides.
> 2) A voip test
> 3) P2MP (3 or more stations, rtt_fair_var* tests)
> 4) Lowered MCS rates or distance or rain

Ok, I’ll see what I can do on 1, 2 and 4.

#3 might be for another day, another site (Open Mesh at an outdoor camp, another story), but I’d like to get to it.

> * Question #15: FreeNet's link to the Internet is 1 Gbps, and AFAIK
> does not experience saturation. The e1000 ethernet driver that the
> Internet router uses supports BQL. Is there any sense in running
> fq_codel or similar on the router, if it does not become saturated?
> 
> You don't need queue management until you need queue management. A
> basic flaw of mrtg's design in your graph here is that it takes
> samples in 5 minute windows. If you are totally nuking your link for 1
> sec out of 5 and the rest nearly idle, you won't see it. A flent test
> in their offices to somewhere nearby over that link will be revealing.
> 
> In general, applying fq_codel on a BQL enabled system is always a
> goodness, it costs next to nothing in CPU to do it that way.
> (outbound). Depending on your measurement of what happens on inbound,
> you might want to do inbound rate shaping….

Ok, I’ll see if they’re game to try that. That would be great if it helped.

> The gains are relative to the bandwidth and the amount of fixed
> buffering in the radio. For example, I can get 320Mbits/sec out of the
> archer c7v2's ath10k, with 10ms latency, at 8 feet. 20 feet, and
> through a wall, it's 200Mbit/100ms latency. I don't like the initial
> shape of that curve! (what I typically see happen on 802.11ac is it
> hitting a wall and not working at all)

My testing was actually through one wall also (drywall). I didn’t have the radios right next to one another. But I’m at 2.4 Ghz and I think you’re at 5 GHz, so walls are going to affect that setup more, of course. I’m surprised how much of a latency hit you see through one wall.

> I happen to like ubnt's gear, although their default firmware only
> lasts 5 minutes for me these days. They have very good throughput and
> latency characteristics with decent signal strength. I look forward to
> a benchmark of what you can get from them. (I am still looking for a
> good upgrade from the nanostation M5s)

My PowerBeam M5-400 has been very stable. I think I’ve only rebooted it for config changes or on general principle.

> Question #12: What's the reason for Cake's occasional sudden
> throughput shifts, and why do its latencies tend to be higher than
> fq_codel?
> 
> We are still debugging cake. Recently the API got a bit wacked. The
> AQM is tuned for DSL speeds. More data is needed.

Ok, there’s plenty of Cake data in my results to mine, or if Jonathan asks I’ll try something else.

> * On the pfifo analysis - I really enjoyed the rush song and it was
> very appropriate for how pfifo misbehaves!
> 
> * Question #9: Does my assertion make sense, that it's "better" to do
> half-duplex queueing on only one end of the link? The side towards the
> Internet?
> 
> Usually the stuff coming from the internet dominates the traffic by a
> 8-20 figure, so coming from might be a worthwhile place to shape.
> 
> I ran out of time before tackling 8-1....

Thanks for all the help. After I test more from your suggestions, I’ll re-post with results and any remaining questions, but a lot of it’s clearing up for me!