[Cerowrt-devel] Turris Omnia

[Cerowrt-devel] Turris Omnia

[James Cloos]

I can't find any mailing list specifically about the Omnia, I hope
someone here may have some tips...

I've put my omnia into production, but the wireless is extremely
sub-optimal.

Things like the droid netinfo widget show 433 Mbps for the 11ac and 72
Mbps on the 2.4, but throughput sucks.

The thoughput seems backwards:  upload speeds eclipse download speeds.

I'm using it only for my 802.11 wlan; its "wan" interface is on my
primary switch, and I've set it for routing rather than bridging, with a
/27 for each of the 5.0 and 2.4 (and a /24 its wired lan ports, which
are not currently in use).

As an example, rsync/ssh/tcp/ip reports net throughput of around 6 Mbps
down on the 11ac, and around ten times that for upload.

I tried 40Mhz and 20Mhz as well as just 11n on the 5.0 and there was no
improvement over the 80Mhz 11ac.

I'm doing a (Gentoo) emerge sync right now on the laptop, which only
does 11n on 2.4.  That is almost OK for a change.  On the omnia,
luci/admin/status/overview reports 135 Mbps for both up and down
from/to the laptop (during said emerge sync).  And the rsync(1)
output looks like it may be updating about that fast.

That is vastly better than I previously saw.

But even so, the 5.0 radio still never shows more than a 6Mbps download
speed and spikes of up to 300 Mbps upload.

I can't find anything online about backwards throughput for  802.11.
The only search results talk about wan links rather than wlan links.

Does anyone have any ideas of how to diagnose or fix this?

Thanks,

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[David Lang]

On Sun, 6 Nov 2016, James Cloos wrote:

> I can't find any mailing list specifically about the Omnia, I hope
> someone here may have some tips...
>
> I've put my omnia into production, but the wireless is extremely
> sub-optimal.
>
> Things like the droid netinfo widget show 433 Mbps for the 11ac and 72
> Mbps on the 2.4, but throughput sucks.
>
> The thoughput seems backwards:  upload speeds eclipse download speeds.
>
> I'm using it only for my 802.11 wlan; its "wan" interface is on my
> primary switch, and I've set it for routing rather than bridging, with a
> /27 for each of the 5.0 and 2.4 (and a /24 its wired lan ports, which
> are not currently in use).
>
> As an example, rsync/ssh/tcp/ip reports net throughput of around 6 Mbps
> down on the 11ac, and around ten times that for upload.
>
> I tried 40Mhz and 20Mhz as well as just 11n on the 5.0 and there was no
> improvement over the 80Mhz 11ac.
>
> I'm doing a (Gentoo) emerge sync right now on the laptop, which only
> does 11n on 2.4.  That is almost OK for a change.  On the omnia,
> luci/admin/status/overview reports 135 Mbps for both up and down
> from/to the laptop (during said emerge sync).  And the rsync(1)
> output looks like it may be updating about that fast.
>
> That is vastly better than I previously saw.
>
> But even so, the 5.0 radio still never shows more than a 6Mbps download
> speed and spikes of up to 300 Mbps upload.
>
> I can't find anything online about backwards throughput for  802.11.
> The only search results talk about wan links rather than wlan links.
>
> Does anyone have any ideas of how to diagnose or fix this?

My first question is if you have checked that you don't have other 5GHz users on 
the same channel in your area.

Also, check that you have a country code set so that you can use the full 
power/frequency range for your location.

upload faster than download is unusual, but interference can cause strange 
issues.

Get the RF right before you worry about other things.

David Lang

Re: [Cerowrt-devel] Turris Omnia

[James Cloos]

>>>>> "DL" == David Lang <david@lang.hm> writes:

DL> My first question is if you have checked that you don't have other
DL> 5GHz users on the same channel in your area.

Yes,  The Wifi Analyzer droid app (run on my fairly high end phablet)
shows only one other 5.0 ap in range;  I chose the start of the 5.0
channel range since there was nothing else.

I don't know that I have anything else to check for non-802 competition
in that band, but I'm not stepping on any other 802 toes there.

DL> Also, check that you have a country code set so that you can use the
DL> full power/frequency range for your location.

Yes, that is set to US.

If it help, I'm currently using (from /etc/config/wireless):

config wifi-device 'radio0'
	option type 'mac80211'
	option country 'US'
	option hwmode '11a'
	option path 'platform/soc/soc:pcie-controller/pci0000:00/0000:00:02.0/0000:02:00.0'
	option disabled '0'
	option channel '36'
	option htmode 'VHT80'

config wifi-iface
	option device 'radio0'
	option network 'wifi50'
	option mode 'ap'
	option disabled '0'
	option hidden '0'
	option encryption 'psk2+tkip+aes'

((ssid and key elided.))

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[David Lang]

On Sun, 6 Nov 2016, James Cloos wrote:

>>>>>> "DL" == David Lang <david@lang.hm> writes:
>
> DL> My first question is if you have checked that you don't have other
> DL> 5GHz users on the same channel in your area.
>
> Yes,  The Wifi Analyzer droid app (run on my fairly high end phablet)
> shows only one other 5.0 ap in range;  I chose the start of the 5.0
> channel range since there was nothing else.
>
> I don't know that I have anything else to check for non-802 competition
> in that band, but I'm not stepping on any other 802 toes there.
>
> DL> Also, check that you have a country code set so that you can use the
> DL> full power/frequency range for your location.
>
> Yes, that is set to US.
>
> If it help, I'm currently using (from /etc/config/wireless):
>
> config wifi-device 'radio0'
> 	option type 'mac80211'
> 	option country 'US'
> 	option hwmode '11a'
> 	option path 'platform/soc/soc:pcie-controller/pci0000:00/0000:00:02.0/0000:02:00.0'
> 	option disabled '0'
> 	option channel '36'
> 	option htmode 'VHT80'
>
> config wifi-iface
> 	option device 'radio0'
> 	option network 'wifi50'
> 	option mode 'ap'
> 	option disabled '0'
> 	option hidden '0'
> 	option encryption 'psk2+tkip+aes'
>
> ((ssid and key elided.))

I wonder if you are running into the problem with encryption and packet 
re-ordering that was solved a couple months ago, try disabling fq_codel or test 
without encryption to see if that's the problem.

David Lang

Re: [Cerowrt-devel] Turris Omnia

[James Cloos]

>>>>> "DL" == David Lang <david@lang.hm> writes:

DL> I wonder if you are running into the problem with encryption and
DL> packet re-ordering that was solved a couple months ago, try disabling
DL> fq_codel or test without encryption to see if that's the problem.

THanks for the idea, but tc fails to delete the qdiscs:

Running commands like:

 :; tc qdisc del dev wlan0 root

returns ENOENT.

Running:

 :; wshaper.htb stop

doesn't work either.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[David Lang]

On Mon, 7 Nov 2016, James Cloos wrote:

>>>>>> "DL" == David Lang <david@lang.hm> writes:
>
> DL> I wonder if you are running into the problem with encryption and
> DL> packet re-ordering that was solved a couple months ago, try disabling
> DL> fq_codel or test without encryption to see if that's the problem.
>
> THanks for the idea, but tc fails to delete the qdiscs:

what does throughput look like without encryption?

David Lang

> Running commands like:
>
> :; tc qdisc del dev wlan0 root
>
> returns ENOENT.
>
> Running:
>
> :; wshaper.htb stop
>
> doesn't work either.
>
> -JimC
>

Re: [Cerowrt-devel] Turris Omnia

[Sebastian Moeller]

Hi James, hi David,

> On Nov 7, 2016, at 06:29, James Cloos <cloos@jhcloos.com> wrote:
> 
>>>>>> "DL" == David Lang <david@lang.hm> writes:
> 
> DL> I wonder if you are running into the problem with encryption and
> DL> packet re-ordering that was solved a couple months ago, try disabling
> DL> fq_codel or test without encryption to see if that's the problem.

	I believe this problem only occurred with the fq code built into the wifi driver or ieee80211 subsystem, so running fq_codel on top of a wifi interface should not cause re-ordering that could confuse the driver.


> 
> THanks for the idea, but tc fails to delete the qdiscs:
> 
> Running commands like:
> 
> :; tc qdisc del dev wlan0 root
> 
> returns ENOENT.
> 
> Running:
> 
> :; wshaper.htb stop
> 
> doesn't work either.

What does “tc -d qdisc” show?

Best Regards
	Sebastian


> 
> -JimC
> -- 
> James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Turris Omnia

[James Cloos]

I don't know why I didn't think of this last night, but I prevented
wshaper from starting at boot and rebooted, and now throughput is
reasonable.

luci/admin/status/overview still reports inaccurate values, but someone
on linux-wireless explained that the ath10 hw chooses the tx internally
and apparently doesn't report it, so that is expected.

But I was able to transfer to the client device at around 300 Mbps now,
with wshaper not started.

tc q still reports that fq_codel is in use:

qdisc mq 0: dev wlan1 root 
qdisc fq_codel 0: dev wlan1 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc mq 0: dev wlan0 root 
qdisc fq_codel 0: dev wlan0 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 

but with wshaper there were 8 rather than 4 lines matching parent for
each interface.  There still are 8 for the 3 ethernet devices, but that
doesn't seem to be a problem....

So this seems solved.

Thanks.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[James Cloos]

>>>>> "SM" == Sebastian Moeller <moeller0@gmx.de> writes:

DL> I wonder if you are running into the problem with encryption and
DL> packet re-ordering that was solved a couple months ago, try disabling
DL> fq_codel or test without encryption to see if that's the problem.

SM> 	I believe this problem only occurred with the fq code built
SM> into the wifi driver or ieee80211 subsystem, so running fq_codel on
SM> top of a wifi interface should not cause re-ordering that could
SM> confuse the driver.

As I noted in my last reply to David, rebooting w/ /etc/init.d/wshaper
disabled fixed things.

In the working scenario tc -d q reports:

qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth0 parent :8 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc mq 0: dev eth2 root 
qdisc fq_codel 0: dev eth2 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :5 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :6 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :7 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev eth2 parent :8 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc mq 0: dev wlan1 root 
qdisc fq_codel 0: dev wlan1 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan1 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc mq 0: dev wlan0 root 
qdisc fq_codel 0: dev wlan0 parent :1 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :3 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
qdisc fq_codel 0: dev wlan0 parent :4 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 

Starting wshaper changes that with this diff:

@@ -29 +29,5 @@
-qdisc noqueue 0: dev br-lan root refcnt 2 
+qdisc htb 1: dev br-lan root refcnt 2 r2q 10 default 20 direct_packets_stat 0 ver 3.17 direct_qlen 1000
+qdisc sfq 10: dev br-lan parent 1:10 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 
+qdisc sfq 20: dev br-lan parent 1:20 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 
+qdisc sfq 30: dev br-lan parent 1:30 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 
+qdisc ingress ffff: dev br-lan parent ffff:fff1 ---------------- 

br-lan only has eth0 and eth2, which are not in use, so the difference
must be somewhere else in what it does.

Thanks.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[Toke Høiland-Jørgensen]

James Cloos <cloos@jhcloos.com> writes:

> I don't know why I didn't think of this last night, but I prevented
> wshaper from starting at boot and rebooted, and now throughput is
> reasonable.

Not too surprising that wshaper is to blame:
https://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die/

Also, https://github.com/CZ-NIC/turris-os/issues/9

-Toke

Re: [Cerowrt-devel] Turris Omnia

[James Cloos]

>>>>> "TH" == Toke Høiland-Jørgensen <toke@toke.dk> writes:

TH> Not too surprising that wshaper is to blame:
TH> https://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die/

TH> Also, https://github.com/CZ-NIC/turris-os/issues/9

Thanks for those; I'd forgotten to check CZ-NIC/turris-os/issues and it
has been so long since I read Wondershaper_Must_Die that I forgot about
that, too. [SIGH]

I've now removed wshaper completely.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [Cerowrt-devel] Turris Omnia

[Dave Täht]

On 11/7/16 6:30 AM, James Cloos wrote:

> but with wshaper there were 8 rather than 4 lines matching parent for
> each interface.  There still are 8 for the 3 ethernet devices, but that
> doesn't seem to be a problem....

The armada chipset in the omnia has 8 hardware queues on the ethernet
devices, so you end up with 8 fq_codel instances on them.

They did not support BQL, thus fq_codel was ineffective, until recently
- a patch was submitted upstream to address this, I don't know if it
made it into openwrt yet. I believed it shaved 10-20ms of latency off a
saturated gigE network.

Second problem is that BQL gives out an eventually even share of queue
to the right hw queues, but that's "eventually".

A third problem, even with BQL, is that you end up with the birthday
problem with so few queues - two flows can easily hash into the same hw
queue. I think the designer's original intent was not to have 8 equal
weight queues, but to have them apply to levels of QoS. A secondary
intent was to map queues to cores for processing. Not in the intent was
to provide consistently low latency or network fairness. :/

Last problem - and why cake may be important to some - was that they had
*aggressively* enabled (64k!) GRO offloads in the driver, which is
helpful on benchmarks, but really hurts sqm-scripts w/fq_codel in some
cases.

> 
> So this seems solved.

Yea!

I note that tc -s qdisc show tends to be more revealing. If the BQL and
GRO problem still exist, you generally will never see fq_codel mark or
drop packets on the ethernet devices, even if you drive two ports into one.

> Thanks.
> 
> -JimC
>