<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 16, 2014 at 5:56 PM, Sebastian Moeller <span dir="ltr"><<a href="mailto:moeller0@gmx.de" target="_blank">moeller0@gmx.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Dave,<div><br></div><div>many thanks for all the information & elucidation, as always.</div>
</div></blockquote><div><br></div><div>I enjoy trying to find the words to explain.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">
<div><br><div><div class="im"><div>On Jan 16, 2014, at 23:30 , Dave Taht <<a href="mailto:dave.taht@gmail.com" target="_blank">dave.taht@gmail.com</a>> wrote:</div><br><blockquote type="cite">On Thu, Jan 16, 2014 at 10:29 AM, Sebastian Moeller <<a href="mailto:moeller0@gmx.de" target="_blank">moeller0@gmx.de</a>> wrote:<br>
<blockquote type="cite">Hi Aaron,<br><br>On Jan 16, 2014, at 16:03 , Aaron Wood <<a href="mailto:woody77@gmail.com" target="_blank">woody77@gmail.com</a>> wrote:<br><br><blockquote type="cite">All,<br><br>I'm noting this here in case anyone is interested. After I write this up, I'm going to start from scratch on the configuration, and factory-reset the router.<br>
<br>=====<br><br>The 5GHz radio on my 3800 seems to be in a very odd state. I'm not quire sure what state it's in, but it seems to be only doing HT20 1x1. And in a fairly broken manner at that.<br><br>Running the rrul test (over wifi directly to the router as the netserver), tcp uploads were 25Mbps or so, but download was 5Mbps.<br>
</blockquote><br> This is with your mac? Try rrul_noclassification, macosx (at least 10.8) will not do RRUL fair to a fast host. Why I do not know… it always prioritizes the upload, as if it did not see/trust the downstream markings (heck maybe it is busy using all bandwidth for upstream so that it literally never sees the markings on the downstream packets..)<br>
</blockquote><br>rrul with classification blows up 802.11e on all devices, everywhere.<br>The VO and VI queues generally get all the bandwidth.<br>Been saying that a while. VO and VI should be strictly admission<br>controlled and are not, anywhere. All the queues fill<br>
and bad things happen. What should happen in a 802.11n world is that a<br>set of packets should wind up in the best queue for the TXOP, and VO<br>used not at all.<br><br>rrul_noclassification better looks like the intent for classification<br>
was for 802.11e and thus works better. There are a couple<br>other tests in the netperf-wrapper suite that don't use classification<br>at all, that might be saner to use.<br></blockquote><div><br></div></div><div><span style="white-space:pre-wrap"> </span>Ah, so in rrul_noclassification, the UDP flows still are tos marked (at least that is reported in the plots and visible in the plots), but even using tcp_bidirectional I see a crazy imbalance 80:1, so this laptop's Broadcom BCM43xx (apple is not as informative as I would like about the components, but the firmware marker points at broadcom I would say) isn't better than the intel wifi in your's I would say…</div>
</div></div></div></blockquote><div><br></div><div>the iwl is a nightmare. the 802.11ac stuff is looking bad too.<br><br>Another issue with the current implementation of rrul is my intent with the specification was to test voip-like streams, an<br>
isochronous 10ms packet in each direction. <br><br></div><div>The implementation currently sends measurement flows based on the RTT, just like ping. As the RTT declines in length, <br>the amount of "space" used up by the measurement flow gets bigger and bigger. At a 3ms RTT, just the EF measurement<br>
flow eats ~2/3s of the available txops as it runs through the VO queue, which is limited to a single packet per txop. The other measurement flows like the CS5 flow, eat the VI queue, and the BE and BK queues get starved for txops.<br>
<br></div><div>I can barely explain to myself how the queues are supposed to get airtime scheduled, see the 802.11e page on wikipedia. I thought 802.11e was a bad idea in the first place... but what rrul does is try to get txops on all 4 queues, which means it<br>
needs 4x as much airtime (this is not accurate), and grabs airtime for it's VO queue first most of the time, followed by<br>VI, BE, and bk.<br></div><div><br></div><div>I think for wifi testing with the current rrul test there needs to be a new test that does everything in BE. (toke?)<br>
</div><div>Classification is very rarely used in the real world anyway.<br></div><div><br></div><div>Most of the usage of rrul to date has been over longer RTTs over ethernet... (again, I'm delighted y'all are doing this,<br>
</div><div>and I do hope to get a more voip-like test) <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>
<div><div><br></div><div><img src="cid:2C98E1DB-B6AF-481E-B541-9725BC6AB41C@home.lan" height="240" width="320"></div><div class="im"><br><blockquote type="cite"><br>lastly, if you are doing a test over the internet, many providers pee<br>
on the tos bits. Unless you've done a packet capture, you can't trust<br>that you are actually seeing classified packets coming back from the<br>internet.<br></blockquote><div><br></div></div><div><span style="white-space:pre-wrap"> </span>Good point, comparing just the local rrul plots with the ones to demo, I see what you mean, there is a tiny bit of the priority classes visible in the uplink (bur barely) and none at all in the downlink, so my ISP does not think too much of the toe bits (I guess the tos effect on the uplink is from what cero is doing and since cero controls the bottleneck some "imprint" remains to be seen at packet reception time at demo, or so I think...).</div>
</div></div></div></blockquote><div><br></div><div>simple.qos respects 3 of the 4 tiers that wifi does.<br><br>simplest does not.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><div><div class="im"><br><blockquote type="cite"><br>One of the things I hope to fix with the twd effort is to detect tos<br>bit preservation and note it in the test.</blockquote><blockquote type="cite">
<br>I'm delighted you'all are seeing these results for yourselves. Getting<br>dinged on bandwidth after aiming for low latency by the public is not<br>something I'd wanted to happen with a "stable" release. Regrettably<br>
fixing the drivers to work better only has<br>felix working on it in his spare time, and I've been trying to clear<br>my plate for months to help do the delicate rework<br>required. (or recruit others to help)<br></blockquote>
<div><br></div></div><div><span style="white-space:pre-wrap"> </span>I would love to help, but this is far out of my league and area of expertise…</div><div><br></div></div></div></div></blockquote><div><br></div><div>yer helping plenty, and the more people that "get this", the sooner people will work<br>
on fixing it. I have enjoyed trying to explain these behaviors today. Someday<br>once we have words that match the concepts they will make sense to a CTO.<br> <br></div><div>I have been very pleased by googling for bufferbloat of late. Almost everyone that<br>
has talked about it on the web for the past month seems to get it.<br><br>So if we start now, and make this the year of "make-wifi-fast", in a couple years<br>maybe the world will get it...<br><br></div><div>... sadly long after 802.11ac is fully deployed and messing up everything for<br>
everybody.<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><div></div><div>best</div><span class="HOEnZb"><font color="#888888"><div>
<span style="white-space:pre-wrap"> </span>Sebastian</div></font></span><div><div class="h5"><br><blockquote type="cite"><br><br><blockquote type="cite">About the other issue I do not know anything…<br><br>Best Regards<br>
Sebastian<br><br><blockquote type="cite">This is me 1-2 meters from the router. Load was never more than 0.33. (I can share the results of people are interested).<br><br>After a full power cycle, wifi isn't coming up at all.<br>
<br>=====<br><br>How I got here:<br><br><br>I'm in France, and had dutifully set my unit with the FR country code when setting up CeroWRT. I had noticed some odd latencies (periodic 100-200ms latency every 10-20 seconds over wifi) on the 5GHz network. The router was on channel 36, and I wanted to move it up to the far-upper ranges, so I tried to specify a "custom" channel to do so (140). This was the channel I thought I had been using with stock (Netgear) firmware.<br>
<br>Wifi didn't come back up after applying the changes, and the luci interface seemed to be tripping up over stuff that it was reading out of the configuration files.<br><br>I ssh'd in via ethernet, and fixed up the configurations by hand.<br>
<br>Except the driver is still reporting that the 5GHz network won't kick into 802.11n modes, and won't use HT40. It seems to be sure it's configured for it, but isn't using it.<br><br>Further, digging into the rc_stats files with the minstrel speeds, I found some very odd data (not what I was expecting to see:<br>
<br>(laptop, which can do 2x2 HT40)<br>rate throughput ewma prob this prob this succ/attempt success attempts<br> D 6 6.0 99.9 100.0 2( 2) 65 65<br> 9 0.0 0.0 0.0 0( 0) 0 0<br>
12 2.9 25.0 100.0 0( 0) 1 1<br> 18 4.3 25.0 100.0 0( 0) 1 1<br> 24 5.6 25.0 100.0 0( 0) 1 1<br>
A P 36 32.4 99.9 100.0 0( 0) 51 51<br> C 48 10.4 25.0 100.0 0( 0) 1 1<br> B 54 11.5 25.0 100.0 0( 0) 1 1<br>
<br>Total packet count:: ideal 53 lookaround 7<br><br>(AppleTV, 1x1 HT20)<br>root@cerowrt:/sys/kernel/debug/ieee80211/phy1/netdev:sw10# cat stations/58\:55\:ca\:51\:b5\:4b/rc_stats<br>rate throughput ewma prob this prob this succ/attempt success attempts<br>
6 3.5 57.8 100.0 0( 0) 6 6<br> 9 3.9 43.7 100.0 0( 0) 2 2<br> 12 5.1 43.7 100.0 0( 0) 2 2<br>
18 10.0 57.8 100.0 0( 0) 3 3<br> D 24 13.1 57.8 100.0 0( 0) 3 3<br> C 36 14.2 43.7 100.0 0( 0) 2 2<br>
B 48 18.2 43.7 100.0 0( 0) 2 2<br>A P 54 46.2 99.9 100.0 1( 1) 348 367<br><br></blockquote></blockquote><br>No AMPDUs. Hmm. Might be a bug.<br>
<br><blockquote type="cite"><blockquote type="cite">Total packet count:: ideal 331 lookaround 37<br></blockquote></blockquote><br>Hmm. The radios are set for HT20 for the 2.4ghz and HT40+ for the<br>5ghz. I note that<br>
HT40 in wireless-n the 8 channels used up need to be congruent.<br><br>HT40+ is 36+40, and 44+48 for example. You can't do 40+44.<br><br>Availability of HTXX is dependent upon your regulatory domain.<br><br><blockquote type="cite">
<blockquote type="cite">Whereas what I'm seeing for the 2.4GHz radio is:<br><br>root@cerowrt:/sys/kernel/debug/ieee80211/phy0/netdev:sw00/stations# cat 10\:9a\:dd\:30\:96\:34/rc_stats<br>type rate throughput ewma prob this prob retry this succ/attempt success attempts<br>
CCK/LP 1.0M 0.7 100.0 100.0 0 0( 0) 2 2<br>CCK/SP 2.0M 0.0 0.0 0.0 0 0( 0) 0 0<br>CCK/SP 5.5M 0.0 0.0 0.0 0 0( 0) 0 0<br>
CCK/SP 11.0M 0.0 0.0 0.0 0 0( 0) 0 0<br>HT20/LGI MCS0 5.6 100.0 100.0 1 0( 0) 2 2<br>HT20/LGI MCS1 0.0 0.0 0.0 0 0( 0) 0 0<br>
HT20/LGI MCS2 0.0 0.0 0.0 0 0( 0) 0 0<br>HT20/LGI MCS3 0.0 0.0 0.0 0 0( 0) 0 0<br>HT20/LGI MCS4 0.0 0.0 0.0 0 0( 0) 0 0<br>
HT20/LGI MCS5 30.3 100.0 100.0 5 0( 0) 1 1<br>HT20/LGI t MCS6 32.5 100.0 100.0 5 0( 0) 11 11<br>HT20/LGI T P MCS7 35.0 100.0 100.0 5 6( 6) 34 34<br>
<br>Total packet count:: ideal 45 lookaround 3<br>Average A-MPDU length: 1.3<br></blockquote></blockquote><br>You are doing good at the highest possible rate. However packet<br>aggregation is pretty terrible.<br><br>
<blockquote type="cite"><blockquote type="cite"><br>And here are radio blocks from the current /etc/config/wireless:<br><br>config wifi-device 'radio1'<br> option type 'mac80211'<br> option macaddr '28:c6:8e:bb:9a:49'<br>
list ht_capab 'SHORT-GI-40'<br> list ht_capab 'TX-STBC'<br> list ht_capab 'RX-STBC1'<br> list ht_capab 'DSSS_CCK-40'<br> option txpower '17'<br> option distance '25'<br>
option channel '48'<br> option country 'US'<br><br>config wifi-device 'radio0'<br> option type 'mac80211'<br> option hwmode '11ng'<br> option macaddr '28:c6:8e:bb:9a:47'<br>
option htmode 'HT20'<br> list ht_capab 'SHORT-GI-40'<br> list ht_capab 'TX-STBC'<br> list ht_capab 'RX-STBC1'<br> list ht_capab 'DSSS_CCK-40'<br> option txpower '26'<br>
option country 'FR'<br> option distance '15'<br> option channel 'auto'<br></blockquote></blockquote><br>I don't know anyone that has fiddled with distance to such an extent.<br>
your country codes need to be the same and you should look at what<br>is allowed in FR.<br><br><blockquote type="cite"><blockquote type="cite">======<br><br>Some notes after having repaired the situation:<br><br>- The pci paths to the radios was missing from /etc/config/wireless, that's the only thing that I saw that seemed grossly out of place.<br>
<br>- Back up and running, and yes, it's much happier, now. Over wifi I get 60-70Mbps upload and ~40Mbps download (running rrul). Latency sucks. Wifi has some ugly bufferbloat. (although these results are somewhat in question when the router has a 1m load average over 5.0...)<br>
</blockquote></blockquote><br>Trying to measure the one way delay here is important (and hard. The<br>only tool I've found for it so far was owamp, so I'm trying to write<br>that test in twd). A TON of your delay is coming from your client. A<br>
network connection is like a fountain, or a toilet, both sides of the<br>flow count...<br><br><blockquote type="cite"><blockquote type="cite"><br>- Enabling all the SQM features I was having previously also considerably cleaned up wifi performance. It's more balanced, but still not nearly as balanced as I see on gigabit ethernet.<br>
<br><br><br>-Aaron<br>_______________________________________________<br>Cerowrt-devel mailing list<br><a href="mailto:Cerowrt-devel@lists.bufferbloat.net" target="_blank">Cerowrt-devel@lists.bufferbloat.net</a><br><a href="https://lists.bufferbloat.net/listinfo/cerowrt-devel" target="_blank">https://lists.bufferbloat.net/listinfo/cerowrt-devel</a><br>
</blockquote><br>_______________________________________________<br>Cerowrt-devel mailing list<br><a href="mailto:Cerowrt-devel@lists.bufferbloat.net" target="_blank">Cerowrt-devel@lists.bufferbloat.net</a><br><a href="https://lists.bufferbloat.net/listinfo/cerowrt-devel" target="_blank">https://lists.bufferbloat.net/listinfo/cerowrt-devel</a><br>
</blockquote><br><br><br>-- <br>Dave Täht<br><br>Fixing bufferbloat with cerowrt: <a href="http://www.teklibre.com/cerowrt/subscribe.html" target="_blank">http://www.teklibre.com/cerowrt/subscribe.html</a><br></blockquote>
</div></div></div><br></div></div></blockquote></div><br><br clear="all"><br>-- <br>Dave Täht<br><br>Fixing bufferbloat with cerowrt: <a href="http://www.teklibre.com/cerowrt/subscribe.html" target="_blank">http://www.teklibre.com/cerowrt/subscribe.html</a>
</div></div>