[Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Mikael Abrahamsson]

Hi,

I've been running openwrt 18.06.2 for a long time on my WRT1200AC, with 
CAKE, and everything has been great. I always got A+ in dslreports for 
bufferbloat. I have a 250/100 ETTH connection (IP over ethernet native, no 
PPPoE, no PON).

I noticed some intermittent network stuttering issues I wanted to diagnose 
last week, so I started looking into this. When doing the test I get 
+100ms spikes in the downstream direction when I run CAKE. If I instead 
switch to FQ_CODEL I do not see these spikes. As a matter of fact, if I 
set my downstream to 300 when running cake and let my ISP handle the 
buffering then I get an average of 5-10ms worth of buffering with no 
spikes. My ISP isn't bloated in that direction.

# tc qdisc show dev eth1.2
qdisc cake 801f: root refcnt 2 bandwidth 100Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 22
qdisc ingress ffff: parent ffff:fff1 ----------------

tc qdisc show dev ifb4eth1.2
qdisc cake 8020: root refcnt 2 bandwidth 240Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 22

http://www.dslreports.com/speedtest/52542158 is the result I get with 
above settings.

http://www.dslreports.com/speedtest/52542263 is the result with FQ_CODEL:

# tc qdisc show dev eth1.2
qdisc htb 1: root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 110: parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc ingress ffff: parent ffff:fff1 ----------------

# tc qdisc show dev ifb4eth1.2
qdisc htb 1: root refcnt 2 r2q 10 default 18 direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 110: parent 1:11 limit 1001p flows 1024 quantum 500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn

How do I diagnose this further?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> Hi,
>
> I've been running openwrt 18.06.2 for a long time on my WRT1200AC, with 
> CAKE, and everything has been great. I always got A+ in dslreports for 
> bufferbloat. I have a 250/100 ETTH connection (IP over ethernet native, no 
> PPPoE, no PON).
>
> I noticed some intermittent network stuttering issues I wanted to diagnose 
> last week, so I started looking into this. When doing the test I get 
> +100ms spikes in the downstream direction when I run CAKE. If I instead 
> switch to FQ_CODEL I do not see these spikes. As a matter of fact, if I 
> set my downstream to 300 when running cake and let my ISP handle the 
> buffering then I get an average of 5-10ms worth of buffering with no 
> spikes. My ISP isn't bloated in that direction.

Hmm, this seems like it's an issue with the timers CAKE uses for
shaping. Not really sure why that would happen...

> # tc qdisc show dev eth1.2
> qdisc cake 801f: root refcnt 2 bandwidth 100Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 22
> qdisc ingress ffff: parent ffff:fff1 ----------------
>
> tc qdisc show dev ifb4eth1.2
> qdisc cake 8020: root refcnt 2 bandwidth 240Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 22

This looks like you're running sqm-scripts on a VLAN interface? Does it
make any difference if you run it on the physical eth1?

-Toke

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Thu, 8 Aug 2019, Toke Høiland-Jørgensen wrote:

> This looks like you're running sqm-scripts on a VLAN interface? Does it 
> make any difference if you run it on the physical eth1?

Due to how the WRT1200AC is set up internally, this is the only way to do 
it. It's how "wan" is set up.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

on most current devices, you don't have any directly exposed ports on the cpu, 
you have ports that connect to a switch and that switch then routes particular 
vlans out the back of the case. There's no longer any difference between LAN and 
WAN except what VLAN they are on and what color the jack is.

David Lang

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Thu, 8 Aug 2019, Toke Høiland-Jørgensen wrote:
>
>> This looks like you're running sqm-scripts on a VLAN interface? Does it 
>> make any difference if you run it on the physical eth1?
>
> Due to how the WRT1200AC is set up internally, this is the only way to do 
> it. It's how "wan" is set up.

Ah right, I see. My Turris Omnia does something similar, but also has a
separate hardwired WAN port. I just assumed the WRT1200AC was similar
(it's been a while since I used mine, since I replaced it with the Omnia).

Another thing you could try, to confirm that the culprit is the shaper,
is to use HTB as a shaper and run CAKE in unlimited mode as the leaf
qdisc. This *should* work, and if it doesn't give you the latency
spikes, then its definitely a wink in the direction of the shaper...

-Toke

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Rosen Penev]

On Thu, Aug 8, 2019 at 9:21 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Thu, 8 Aug 2019, Toke Høiland-Jørgensen wrote:
>
> > This looks like you're running sqm-scripts on a VLAN interface? Does it
> > make any difference if you run it on the physical eth1?
>
> Due to how the WRT1200AC is set up internally, this is the only way to do
> it. It's how "wan" is set up.
Note that OpenWrt uses its NIH swconfig to configure the switch
instead of DSA. DSA might give you different results.
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Sebastian Moeller]

> On Aug 9, 2019, at 19:44, Rosen Penev <rosenp@gmail.com> wrote:
> 
> On Thu, Aug 8, 2019 at 9:21 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> On Thu, 8 Aug 2019, Toke Høiland-Jørgensen wrote:
>> 
>>> This looks like you're running sqm-scripts on a VLAN interface? Does it
>>> make any difference if you run it on the physical eth1?
>> 
>> Due to how the WRT1200AC is set up internally, this is the only way to do
>> it. It's how "wan" is set up.
> Note that OpenWrt uses its NIH swconfig to configure the switch

	Who knew it was the national institute of health that invented swconfig ;)

Best Regards & only kidding
	Sebastian


> instead of DSA. DSA might give you different results.

	I believe Toke's omnia might run DSA already, at least the turris OS 4 betas use DSA instead of swconfig, not that this has any bearing on the wrt1200ac...

>> 
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Jonathan Morton]

> On 9 Aug, 2019, at 8:49 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> Note that OpenWrt uses its NIH swconfig to configure the switch
> 
> 	Who knew it was the national institute of health that invented swconfig ;)

Perhaps it's "Not Invented Here".

 - Jonathan Morton

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Sebastian Moeller]

> On Aug 9, 2019, at 19:53, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 9 Aug, 2019, at 8:49 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>> Note that OpenWrt uses its NIH swconfig to configure the switch
>> 
>> 	Who knew it was the national institute of health that invented swconfig ;)
> 
> Perhaps it's "Not Invented Here".

	Does not make much sense, since swconfig was invented by members of the OpenWrt project, so it really was "invented here" at least in relation to the thread topic. But, I admit, that is how I expanded NIH as well just after the expansion I posted, I guess professional blinders guided my acronym-fu here.... Anyway, just wanted to lighten up the day...

Best Regards
	Sebastian

> 
> - Jonathan Morton

Re: [Bloat] weird problem WRT1200 + CAKE on OpenWrt 18.06.2

[Rosen Penev]

On Fri, Aug 9, 2019 at 10:49 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
>
> > On Aug 9, 2019, at 19:44, Rosen Penev <rosenp@gmail.com> wrote:
> >
> > On Thu, Aug 8, 2019 at 9:21 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >>
> >> On Thu, 8 Aug 2019, Toke Høiland-Jørgensen wrote:
> >>
> >>> This looks like you're running sqm-scripts on a VLAN interface? Does it
> >>> make any difference if you run it on the physical eth1?
> >>
> >> Due to how the WRT1200AC is set up internally, this is the only way to do
> >> it. It's how "wan" is set up.
> > Note that OpenWrt uses its NIH swconfig to configure the switch
>
>         Who knew it was the national institute of health that invented swconfig ;)
>
> Best Regards & only kidding
>         Sebastian
>
>
> > instead of DSA. DSA might give you different results.
>
>         I believe Toke's omnia might run DSA already, at least the turris OS 4 betas use DSA instead of swconfig, not that this has any bearing on the wrt1200ac...
Correct. On the Omnia, the WAN port is not connected to the switch.
It's not too relevant here.
>
> >>
> >> --
> >> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> >> Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>