[Bloat] CAKE in openwrt high CPU

[Bloat] CAKE in openwrt high CPU

[Mikael Abrahamsson]

Hi,

I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
router, from my previous WRT1200AC (marvell armada 385).

I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
on the APU2.

Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
and tried it again, and even the APU2 can only do CAKE up to ~300 
megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
OpenWrt).

Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
see when CPU based forwarding is maxed out. From my recollection of 
running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
using more CPU than FQ_CODEL.

Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
bufferbloat but... I thought CAKE supposedly should use less CPU, not 
more?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] CAKE in openwrt high CPU

[Dave Taht]

cake reschedules too much compared to the tweaks we have to keep htb
fed, at these rates.

It was kind of my hope to gain a hw assist in future versions of the
apu series. a programmable completion interrupt is available in some
versions of that chipset,

On Sun, Aug 30, 2020 at 10:27 AM Mikael Abrahamsson via Bloat
<bloat@lists.bufferbloat.net> wrote:
>
>
> Hi,
>
> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential
> router, from my previous WRT1200AC (marvell armada 385).
>
> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3
> on the APU2.
>
> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much
> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000
> and tried it again, and even the APU2 can only do CAKE up to ~300
> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in
> OpenWrt).
>
> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I
> see when CPU based forwarding is maxed out. From my recollection of
> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE
> using more CPU than FQ_CODEL.
>
> Anyone know what's up? I'm fine running FQ_CODEL, it solves any
> bufferbloat but... I thought CAKE supposedly should use less CPU, not
> more?
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> writes:

> Hi,
>
> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
> router, from my previous WRT1200AC (marvell armada 385).
>
> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
> on the APU2.
>
> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
> and tried it again, and even the APU2 can only do CAKE up to ~300 
> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
> OpenWrt).
>
> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
> see when CPU based forwarding is maxed out. From my recollection of 
> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
> using more CPU than FQ_CODEL.
>
> Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
> bufferbloat but... I thought CAKE supposedly should use less CPU, not 
> more?

Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
would be FQ-CoDel+HTB)? An exact config might be useful (or just the
output of tc -s qdisc).

If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?

https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 2782 bytes --]

On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:

> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
> output of tc -s qdisc).

Yeah, I guess I'm also using HTB to get the 900 megabit/s SQM is looking 
for.

If I only use FQ_CODEL to get interface speeds my performance is fine.

> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
>
> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n

I enabled it just now to get the config.

qdisc cake 8030: dev eth0 root refcnt 9 bandwidth 900Mbit besteffort 
triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw 
overhead 0
  Sent 4346128 bytes 11681 pkt (dropped 0, overlimits 1004 requeues 17)
  backlog 0b 0p requeues 17
  memory used: 33328b of 15140Kb
  capacity estimate: 900Mbit
  min/max network layer size:           42 /    1514
  min/max overhead-adjusted size:       42 /    1514
  average network hdr offset:           14

                   Tin 0
   thresh        900Mbit
   target          5.0ms
   interval      100.0ms
   pk_delay         18us
   av_delay          6us
   sp_delay          4us
   backlog            0b
   pkts            11681
   bytes         4346128
   way_inds           30
   way_miss          735
   way_cols            0
   drops               0
   marks               0
   ack_drop            0
   sp_flows            3
   bk_flows            1
   un_flows            0
   max_len         22710
   quantum          1514

qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
  Sent 4716199 bytes 10592 pkt (dropped 0, overlimits 0 requeues 0)
  backlog 0b 0p requeues 0

...

qdisc cake 8031: dev ifb4eth0 root refcnt 2 bandwidth 900Mbit besteffort 
triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw 
overhead 0
  Sent 4946683 bytes 10592 pkt (dropped 0, overlimits 492 requeues 0)
  backlog 0b 0p requeues 0
  memory used: 35Kb of 15140Kb
  capacity estimate: 900Mbit
  min/max network layer size:           60 /    1514
  min/max overhead-adjusted size:       60 /    1514
  average network hdr offset:           14

                   Tin 0
   thresh        900Mbit
   target          5.0ms
   interval      100.0ms
   pk_delay         19us
   av_delay          6us
   sp_delay          4us
   backlog            0b
   pkts            10592
   bytes         4946683
   way_inds           33
   way_miss          969
   way_cols            0
   drops               0
   marks               0
   ack_drop            0
   sp_flows            2
   bk_flows            1
   un_flows            0
   max_len         21196
   quantum          1514


-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:
>
>> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
>> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
>> output of tc -s qdisc).
>
> Yeah, I guess I'm also using HTB to get the 900 megabit/s SQM is looking 
> for.

Ah, right, makes more sense :)

> If I only use FQ_CODEL to get interface speeds my performance is fine.

And what about when you're running CAKE in 'unlimited' mode?

>> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
>>
>> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n
>
> I enabled it just now to get the config.
>
> qdisc cake 8030: dev eth0 root refcnt 9 bandwidth 900Mbit besteffort 
> triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw 
> overhead 0

Hmm, right, you could try no-split-gso as an option as well; you're
pretty close to the point where we turn it off by default, and you're
getting pretty large packets (max_len), so your performance may be
suffering from the splitting...

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Foulkes]

Toke, that link returns a 404 for me.

For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.

That said, I’d love to find options to increase throughput for single-tin configs.

Cheers,

Jonathan

> On Aug 31, 2020, at 7:35 AM, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> writes:
> 
>> Hi,
>> 
>> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
>> router, from my previous WRT1200AC (marvell armada 385).
>> 
>> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
>> on the APU2.
>> 
>> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
>> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
>> and tried it again, and even the APU2 can only do CAKE up to ~300 
>> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
>> OpenWrt).
>> 
>> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
>> see when CPU based forwarding is maxed out. From my recollection of 
>> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
>> using more CPU than FQ_CODEL.
>> 
>> Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
>> bufferbloat but... I thought CAKE supposedly should use less CPU, not 
>> more?
> 
> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
> output of tc -s qdisc).
> 
> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
> 
> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n
> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Jonathan Foulkes <jf@jonathanfoulkes.com> writes:

> Toke, that link returns a 404 for me.

Ah, seems an extra character snuck in at the end - try this:

https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6

> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
>
> That said, I’d love to find options to increase throughput for
> single-tin configs.

Yeah, doing something about this is on my list, one way or another. Not
sure how much more we can do in terms of overhead, so we may have to go
for multi-q (and multi-CPU) support. How many CPU cores does the
IQrouter have?

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

HI Jonathan,

> On Sep 1, 2020, at 17:41, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
> Toke, that link returns a 404 for me.
> 
> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 

	Are you talking about the fact that with competing connections, you only see the current isolation quantum's equivalent f the actual rate? In that case maybe parse the "tc -s qdisc" output to get an idea how much data/packets cake managed to push through in total in each direction instead of relaying on the measured goodput? I am probably barking up the wrong tree here...

> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.

	That sounds like solid advice ;) Especially in the light of the exceedingly useful "ingress" keyword, which under-load-will drop depending on a flow's "unresponsiveness" such that more responsive flows end up getting a somewhat bigger share of the post-cake throughput...

> 
> That said, I’d love to find options to increase throughput for single-tin configs.

	With or without isolation options?

Best Regards
	Sebastian

> 
> Cheers,
> 
> Jonathan
> 
>> On Aug 31, 2020, at 7:35 AM, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> writes:
>> 
>>> Hi,
>>> 
>>> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
>>> router, from my previous WRT1200AC (marvell armada 385).
>>> 
>>> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
>>> on the APU2.
>>> 
>>> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
>>> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
>>> and tried it again, and even the APU2 can only do CAKE up to ~300 
>>> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
>>> OpenWrt).
>>> 
>>> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
>>> see when CPU based forwarding is maxed out. From my recollection of 
>>> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
>>> using more CPU than FQ_CODEL.
>>> 
>>> Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
>>> bufferbloat but... I thought CAKE supposedly should use less CPU, not 
>>> more?
>> 
>> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
>> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
>> output of tc -s qdisc).
>> 
>> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
>> 
>> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n
>> 
>> -Toke
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

Hi Toke,


> On Sep 1, 2020, at 18:11, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Jonathan Foulkes <jf@jonathanfoulkes.com> writes:
> 
>> Toke, that link returns a 404 for me.
> 
> Ah, seems an extra character snuck in at the end - try this:
> 
> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6
> 
>> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
>> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
>> 
>> That said, I’d love to find options to increase throughput for
>> single-tin configs.
> 
> Yeah, doing something about this is on my list, one way or another. Not
> sure how much more we can do in terms of overhead, so we may have to go
> for multi-q (and multi-CPU) support. How many CPU cores does the
> IQrouter have?

	It might be worth looking how the typical two cake instances distribute across the available CPUs, in some version of OpenWrt all cake's and ethernet interupt processing crowed up on a single CPU leading to "out of CPU" behaviour with 50% idle remaining... I think that usinf a different RPS scheme might work better.

Best Regards
	Sebastian

> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Toke,
>
>
>> On Sep 1, 2020, at 18:11, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> Jonathan Foulkes <jf@jonathanfoulkes.com> writes:
>> 
>>> Toke, that link returns a 404 for me.
>> 
>> Ah, seems an extra character snuck in at the end - try this:
>> 
>> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6
>> 
>>> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
>>> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
>>> 
>>> That said, I’d love to find options to increase throughput for
>>> single-tin configs.
>> 
>> Yeah, doing something about this is on my list, one way or another. Not
>> sure how much more we can do in terms of overhead, so we may have to go
>> for multi-q (and multi-CPU) support. How many CPU cores does the
>> IQrouter have?
>
> 	It might be worth looking how the typical two cake instances
> 	distribute across the available CPUs, in some version of OpenWrt
> 	all cake's and ethernet interupt processing crowed up on a
> 	single CPU leading to "out of CPU" behaviour with 50% idle
> 	remaining... I think that usinf a different RPS scheme might
> 	work better.

Well, many home routers don't have any functional RPS at all. Also, it
doesn't help since CAKE takes the global qdisc lock. Both of those
issues should be fixed, ideally :)

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Foulkes]

Thanks Toke, we currently are on an MT7621a @880, so a dual-core.
And we are looking for a good quad-core platform that will support 600Mbps or more with Cake enabled, hopefully with AX radios as well.

Jonathan

> On Sep 1, 2020, at 12:11 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Jonathan Foulkes <jf@jonathanfoulkes.com> writes:
> 
>> Toke, that link returns a 404 for me.
> 
> Ah, seems an extra character snuck in at the end - try this:
> 
> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6
> 
>> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
>> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
>> 
>> That said, I’d love to find options to increase throughput for
>> single-tin configs.
> 
> Yeah, doing something about this is on my list, one way or another. Not
> sure how much more we can do in terms of overhead, so we may have to go
> for multi-q (and multi-CPU) support. How many CPU cores does the
> IQrouter have?
> 
> -Toke

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Morton]

> On 1 Sep, 2020, at 9:45 pm, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> CAKE takes the global qdisc lock.

Presumably this is a default mechanism because CAKE doesn't handle any locking itself.

Obviously it would need to be replaced with at least a lock over CAKE's complete data structures, taking the lock on each entry point and releasing it at each return point, and I assume there is a flag we can set to indicate we do so.  Finer-grained locking might be possible, but CAKE is fairly complex so that might be hard to implement.  Locking per CAKE instance would at least allow running ingress and egress on different CPUs.

Is there an example anywhere on how to do this?

 - Jonathan Morton

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Foulkes]

[-- Attachment #1: Type: text/plain, Size: 10469 bytes --]

Hi Sebastian, Cake functions wonderfully, it’s a marvel in terms of goodput.

My comment was more oriented at the metrics process users use to evaluate results. Only those who spend time analyzing just how busy an ‘idle’ network can be know that there are a lot of processes in constant communications with their cloud services. 
The challenge are the end users, who only understand the silly ’speed’ metric, and feel anything that lowers that number is a ‘bad’ thing. It takes effort to get even technical users to get it.
But even beyond the basic, the further cuts induced by fairness is the new wrinkle in dealing with widely varying speed test results with isolation enabled on a busy network.

The high density of devices and constant chatter with cloud services means the average home has way more devices and connections than many realize. Keep a note of the number of ‘active connections’ displayed on the OpenWRT overview page, you might be surprised (well, not you Seb ;) )

As an example, on my network, I average 1,000 active connections all day, it rarely drops below 700. And it’s just two WFH professionals and 60+ network devices, not all of which are active at any one time.
I actually run some custom firewall rules to de-prioritize four IoT devices that generate a LOT of traffic to their services. Two of which power panel monitors with real-time updates. This is why my bulk tin on egress has such high traffic.

Since you like to see tc output, here’s the one from my system after nearly a week.
I run four-layer Cake as we do a lot of Zoom calls and our accounts are set up to do the appropriate DSCP marking.

root@IQrouter:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 51311363856 bytes 86785488 pkt (dropped 53, overlimits 0 requeues 9114) 
 backlog 0b 0p requeues 9114
  maxpacket 12112 drop_overlimit 0 new_flow_count 691740 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8005: dev eth0.2 root refcnt 2 bandwidth 22478Kbit diffserv4 dual-srchost nat nowash ack-filter split-gso rtt 100.0ms raw overhead 0 mpu 64 
 Sent 6943407136 bytes 35467722 pkt (dropped 51747, overlimits 3912091 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 843816b of 4Mb
 capacity estimate: 22478Kbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       64 /    1514
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh       1404Kbit    22478Kbit    11239Kbit     5619Kbit
  target         12.9ms        5.0ms        5.0ms        5.0ms
  interval      107.9ms      100.0ms      100.0ms      100.0ms
  pk_delay        5.9ms        6.4ms        3.7ms        1.6ms
  av_delay        426us        445us        124us        188us
  sp_delay         13us         13us         12us          8us
  backlog            0b           0b           0b           0b
  pkts          3984407     30899121       474818       161123
  bytes       789740113   5883832402    246917562     30556915
  way_inds        65175      2580935         1064            5
  way_miss         1427       918529        15960         1120
  way_cols            0            0            0            0
  drops               0         2966          511            7
  marks               0          105            0            0
  ack_drop            0        48263            0            0
  sp_flows            2            4            1            0
  bk_flows            0            0            0            0
  un_flows            0            0            0            0
  max_len          1035        43094         3094          590
  quantum           300          685          342          300

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 43188461026 bytes 67870269 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8006: dev ifb4eth0.2 root refcnt 2 bandwidth 289066Kbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 18 mpu 64 
 Sent 44692280901 bytes 67864800 pkt (dropped 5472, overlimits 5572964 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 7346339b of 14453300b
 capacity estimate: 289066Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh      18066Kbit   289066Kbit   144533Kbit    72266Kbit
  target          5.0ms        5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms      100.0ms
  pk_delay         47us        740us         42us         18us
  av_delay         24us         32us         22us          9us
  sp_delay         14us         11us         12us          5us
  backlog            0b           0b           0b           0b
  pkts             1389     45323600      3704409     18840874
  bytes          136046  43347299847    222296446   1130523693
  way_inds            0      3016679            0            0
  way_miss           17       903215         1053         2318
  way_cols            0            0            0            0
  drops               0         5471            0            1
  marks               0           27            0            0
  ack_drop            0            0            0            0
  sp_flows            1            4            2            1
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len            98        68338          136          221
  quantum           551         1514         1514         1514

root@IQrouter:~# uptime
 15:07:37 up 6 days,  3:23,  load average: 0.46, 0.21, 0.20
root@IQrouter:~# 


Cheers,

Jonathan

> On Sep 1, 2020, at 12:18 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> HI Jonathan,
> 
>> On Sep 1, 2020, at 17:41, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>> 
>> Toke, that link returns a 404 for me.
>> 
>> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
> 
> 	Are you talking about the fact that with competing connections, you only see the current isolation quantum's equivalent f the actual rate? In that case maybe parse the "tc -s qdisc" output to get an idea how much data/packets cake managed to push through in total in each direction instead of relaying on the measured goodput? I am probably barking up the wrong tree here...
> 
>> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
> 
> 	That sounds like solid advice ;) Especially in the light of the exceedingly useful "ingress" keyword, which under-load-will drop depending on a flow's "unresponsiveness" such that more responsive flows end up getting a somewhat bigger share of the post-cake throughput...
> 
>> 
>> That said, I’d love to find options to increase throughput for single-tin configs.
> 
> 	With or without isolation options?
> 
> Best Regards
> 	Sebastian
> 
>> 
>> Cheers,
>> 
>> Jonathan
>> 
>>> On Aug 31, 2020, at 7:35 AM, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
>>> 
>>> Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> writes:
>>> 
>>>> Hi,
>>>> 
>>>> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
>>>> router, from my previous WRT1200AC (marvell armada 385).
>>>> 
>>>> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
>>>> on the APU2.
>>>> 
>>>> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
>>>> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
>>>> and tried it again, and even the APU2 can only do CAKE up to ~300 
>>>> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
>>>> OpenWrt).
>>>> 
>>>> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
>>>> see when CPU based forwarding is maxed out. From my recollection of 
>>>> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
>>>> using more CPU than FQ_CODEL.
>>>> 
>>>> Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
>>>> bufferbloat but... I thought CAKE supposedly should use less CPU, not 
>>>> more?
>>> 
>>> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
>>> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
>>> output of tc -s qdisc).
>>> 
>>> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
>>> 
>>> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n
>>> 
>>> -Toke
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 


[-- Attachment #2: Type: text/html, Size: 41260 bytes --]

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

Hi Jonathan,



> On Sep 1, 2020, at 21:31, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
> Hi Sebastian, Cake functions wonderfully, it’s a marvel in terms of goodput.
> 
> My comment was more oriented at the metrics process users use to evaluate results. Only those who spend time analyzing just how busy an ‘idle’ network can be know that there are a lot of processes in constant communications with their cloud services. 

	True, intestinally, quite a number of speedtests seem to err on the side of too high, probably because that way users are happy to see something close to their contracted rates...

> The challenge are the end users, who only understand the silly ’speed’ metric, and feel anything that lowers that number is a ‘bad’ thing. It takes effort to get even technical users to get it.

	I repeatedly fall into that trap...

> But even beyond the basic, the further cuts induced by fairness is the new wrinkle in dealing with widely varying speed test results with isolation enabled on a busy network.

	Yes, but one can try to make lemonade out of it, by running speedtests from two devices while observing something like "sudo mtr -ezb4 -i 0.3 8.8.8.8" not budging much een though the tests come and go; demonstrating the quality of the isolation and that low queueing delay can "happen" even on a busy link.

> 
> The high density of devices and constant chatter with cloud services means the average home has way more devices and connections than many realize. Keep a note of the number of ‘active connections’ displayed on the OpenWRT overview page, you might be surprised (well, not you Seb ;) )

	Count me in, I just switched over to a turris omnia (which I had crowd-funded before I realized IQrouters will be delivered to Germany ;) ) and while playning with its pakon feature I was quite baffled by how many addresses are used even in a short amount of time. (All of this is just a hobby to me, so I keep forgetting stuff regularly, because I do approach things a bit casually at times).

> 
> As an example, on my network, I average 1,000 active connections all day, it rarely drops below 700. And it’s just two WFH professionals and 60+ network devices, not all of which are active at any one time.
> I actually run some custom firewall rules to de-prioritize four IoT devices that generate a LOT of traffic to their services. Two of which power panel monitors with real-time updates. This is why my bulk tin on egress has such high traffic.

	Nice, I think being able to deprioritize stuff is one of the best reasons for using diffserve.

> 
> Since you like to see tc output, here’s the one from my system after nearly a week.
> I run four-layer Cake as we do a lot of Zoom calls and our accounts are set up to do the appropriate DSCP marking.

	I saw your nice writeup of how to do that on the OpenWrt forum IIRC. Need to talk to our IT guys at work, whether they are willing to actually configure it in the first place.


> 
> root@IQrouter:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
>  Sent 51311363856 bytes 86785488 pkt (dropped 53, overlimits 0 requeues 9114) 
>  backlog 0b 0p requeues 9114
>   maxpacket 12112 drop_overlimit 0 new_flow_count 691740 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc noqueue 0: dev br-lan root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev eth0.1 root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc cake 8005: dev eth0.2 root refcnt 2 bandwidth 22478Kbit diffserv4 dual-srchost nat nowash ack-filter split-gso rtt 100.0ms raw overhead 0 mpu 64 
>  Sent 6943407136 bytes 35467722 pkt (dropped 51747, overlimits 3912091 requeues 0) 
>  backlog 0b 0p requeues 0
>  memory used: 843816b of 4Mb
>  capacity estimate: 22478Kbit
>  min/max network layer size:           42 /    1514
>  min/max overhead-adjusted size:       64 /    1514
>  average network hdr offset:           14
> 
>                    Bulk  Best Effort        Video        Voice
>   thresh       1404Kbit    22478Kbit    11239Kbit     5619Kbit
>   target         12.9ms        5.0ms        5.0ms        5.0ms
>   interval      107.9ms      100.0ms      100.0ms      100.0ms
>   pk_delay        5.9ms        6.4ms        3.7ms        1.6ms
>   av_delay        426us        445us        124us        188us
>   sp_delay         13us         13us         12us          8us
>   backlog            0b           0b           0b           0b
>   pkts          3984407     30899121       474818       161123
>   bytes       789740113   5883832402    246917562     30556915
>   way_inds        65175      2580935         1064            5
>   way_miss         1427       918529        15960         1120
>   way_cols            0            0            0            0
>   drops               0         2966          511            7
>   marks               0          105            0            0
>   ack_drop            0        48263            0            0
>   sp_flows            2            4            1            0
>   bk_flows            0            0            0            0
>   un_flows            0            0            0            0
>   max_len          1035        43094         3094          590
>   quantum           300          685          342          300
> 
> qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
>  Sent 43188461026 bytes 67870269 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev br-guest root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev wlan1 root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev wlan0 root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev wlan0-1 root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc noqueue 0: dev wlan1-1 root refcnt 2 
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
>  backlog 0b 0p requeues 0
> qdisc cake 8006: dev ifb4eth0.2 root refcnt 2 bandwidth 289066Kbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 18 mpu 64 
>  Sent 44692280901 bytes 67864800 pkt (dropped 5472, overlimits 5572964 requeues 0) 
>  backlog 0b 0p requeues 0
>  memory used: 7346339b of 14453300b
>  capacity estimate: 289066Kbit
>  min/max network layer size:           46 /    1500
>  min/max overhead-adjusted size:       64 /    1518
>  average network hdr offset:           14
> 
>                    Bulk  Best Effort        Video        Voice
>   thresh      18066Kbit   289066Kbit   144533Kbit    72266Kbit
>   target          5.0ms        5.0ms        5.0ms        5.0ms
>   interval      100.0ms      100.0ms      100.0ms      100.0ms
>   pk_delay         47us        740us         42us         18us
>   av_delay         24us         32us         22us          9us
>   sp_delay         14us         11us         12us          5us
>   backlog            0b           0b           0b           0b
>   pkts             1389     45323600      3704409     18840874
>   bytes          136046  43347299847    222296446   1130523693
>   way_inds            0      3016679            0            0
>   way_miss           17       903215         1053         2318
>   way_cols            0            0            0            0
>   drops               0         5471            0            1
>   marks               0           27            0            0
>   ack_drop            0            0            0            0
>   sp_flows            1            4            2            1
>   bk_flows            0            1            0            0
>   un_flows            0            0            0            0
>   max_len            98        68338          136          221
>   quantum           551         1514         1514         1514
> 
> root@IQrouter:~# uptime
>  15:07:37 up 6 days,  3:23,  load average: 0.46, 0.21, 0.20
> root@IQrouter:~# 
> 

	Thanks for sharing the stats, good reference.

Best Regards
	Sebastian


> 
> Cheers,
> 
> Jonathan
> 
>> On Sep 1, 2020, at 12:18 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> HI Jonathan,
>> 
>>> On Sep 1, 2020, at 17:41, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>>> 
>>> Toke, that link returns a 404 for me.
>>> 
>>> For others, I’ve found that testing cake throughput with isolation options enabled is tricky if there are many competing connections. 
>> 
>> 	Are you talking about the fact that with competing connections, you only see the current isolation quantum's equivalent f the actual rate? In that case maybe parse the "tc -s qdisc" output to get an idea how much data/packets cake managed to push through in total in each direction instead of relaying on the measured goodput? I am probably barking up the wrong tree here...
>> 
>>> Like I keep having to tell my customers, fairness algorithms mean no one device will ever gain 100% of the bandwidth so long as there are other open & active connections from other devices.
>> 
>> 	That sounds like solid advice ;) Especially in the light of the exceedingly useful "ingress" keyword, which under-load-will drop depending on a flow's "unresponsiveness" such that more responsive flows end up getting a somewhat bigger share of the post-cake throughput...
>> 
>>> 
>>> That said, I’d love to find options to increase throughput for single-tin configs.
>> 
>> 	With or without isolation options?
>> 
>> Best Regards
>> 	Sebastian
>> 
>>> 
>>> Cheers,
>>> 
>>> Jonathan
>>> 
>>>> On Aug 31, 2020, at 7:35 AM, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
>>>> 
>>>> Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> writes:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I migrated to an APU2 (https://www.pcengines.ch/apu2.htm) as residential 
>>>>> router, from my previous WRT1200AC (marvell armada 385).
>>>>> 
>>>>> I was running OpenWrt 18.06 on that one, now I am running latest 19.07.3 
>>>>> on the APU2.
>>>>> 
>>>>> Before I had 500/100 and I had to use FQ_CODEL because CAKE took too much 
>>>>> CPU to be able to do 500/100 on the WRT1200AC. Now I upgraded to 1000/1000 
>>>>> and tried it again, and even the APU2 can only do CAKE up to ~300 
>>>>> megabit/s. With FQ_CODEL I get full speed (configure 900/900 in SQM in 
>>>>> OpenWrt).
>>>>> 
>>>>> Looking in top, I see sirq% sitting at 50% pegged. This is typical what I 
>>>>> see when CPU based forwarding is maxed out. From my recollection of 
>>>>> running CAKE on earlier versions of openwrt (17.x) I don't remember CAKE 
>>>>> using more CPU than FQ_CODEL.
>>>>> 
>>>>> Anyone know what's up? I'm fine running FQ_CODEL, it solves any 
>>>>> bufferbloat but... I thought CAKE supposedly should use less CPU, not 
>>>>> more?
>>>> 
>>>> Hmm, you say CAKE and FQ-Codel - so you're not enabling the shaper (that
>>>> would be FQ-CoDel+HTB)? An exact config might be useful (or just the
>>>> output of tc -s qdisc).
>>>> 
>>>> If you are indeed not shaping, maybe you're hitting the issue fixed by this commit?
>>>> 
>>>> https://github.com/dtaht/sch_cake/commit/3152477235c934022049fcddc063c45d37ec10e6n
>>>> 
>>>> -Toke
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>> 
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> 
> 

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Morton]

> On 1 Sep, 2020, at 11:04 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> The challenge are the end users, who only understand the silly ’speed’ metric, and feel anything that lowers that number is a ‘bad’ thing. It takes effort to get even technical users to get it.
> 
> 	I repeatedly fall into that trap...

For a lot of users, I rather suspect that setting 40/10 Mbps would give them entirely sufficient speed, and most existing CPE would be able to keep up with those settings even with all of Cake's bells and whistles turned on.

The trouble is that that might be 10% of what the cable company is advertising to them.

 - Jonathan Morton

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 1 Sep, 2020, at 9:45 pm, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> CAKE takes the global qdisc lock.
>
> Presumably this is a default mechanism because CAKE doesn't handle any
> locking itself.
>
> Obviously it would need to be replaced with at least a lock over
> CAKE's complete data structures, taking the lock on each entry point
> and releasing it at each return point, and I assume there is a flag we
> can set to indicate we do so. Finer-grained locking might be possible,
> but CAKE is fairly complex so that might be hard to implement. Locking
> per CAKE instance would at least allow running ingress and egress on
> different CPUs.

What you're describing here is basically the existing qdisc root lock.
It is per instance of the qdisc, and it is held only while enqueueing
and dequeueing packets from that qdisc. So it is possible today to run
the ingress and egress instances of CAKE on different CPUs. All you have
to do is schedule the packets to be processed on different CPUs in the
different directions - which usually means messing with RPS settings for
the NIC, and as I remarked to Sebastian, for many OpenWrt SOCs this is
not really supported...

To make CAKE truly take advantage of multiple CPUs, there are to
options:

1. Make it aware of multiple hardware queues. To do this, we to
   implement the 'attach()' method in the Qdisc_ops struct (see sch_mq
   for an example). The idea here would be to create stub child qdiscs
   with a separate struct Qdisc_ops implementing enqueue() and
   dequeue(). These would be called separately for each hardware queue,
   with their separate locks held at the time; and with proper XPS
   steering, each hardware queue can be serviced by a separate CPU.

2. Set the TCQ_F_NOLOCK in the qdisc flags; this will cause the existing
   enqueue() and dequeue() functions to be called without the root lock
   being held, and the qdisc is responsible for dealing with that
   itself.

Of course in either case, the trick is to get the CAKE data structures
to play nice with concurrent access from multiple CPUs. For option 1.
above, we could just duplicate all the flow queues for each netdev queue
and take the hit in wasted space - or we could partition the data
structure, either statically at init, or dynamically as each flow
becomes active. But at a minimum there would need to be some way for the
shaper to enforce the maximum rate. Maybe a granular lock or an atomic
is good enough for this, though?

Note also that for 2. there's an ongoing issue[0] with packets getting
stuck which is still unresolved, as far as I can tell - so not sure if
this is the right way to go. However, apart from this, the benefit of 2.
is that CAKE could *potentially* process packets on multiple CPUs
without relying on hardware multi-Q. I'm not quite sure if the stack
will actually process packets on more than one CPU without them,
though.

Either way, I suppose some experimentation would be needed to find the
best solution.

-Toke

[0] https://lore.kernel.org/netdev/CACS=qq+a0H=e8yLFu95aE7Hr0bQ9ytCBBn2rFx82oJnPpkBpvg@mail.gmail.com/

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Jonathan Foulkes <jf@jonathanfoulkes.com> writes:

> Thanks Toke, we currently are on an MT7621a @880, so a dual-core.

Right, so some benefit might be possible here. Does the NIC have
multiple hardware queues (`ls /sys/class/net/$IFACE/queues` should tell
you)?

> And we are looking for a good quad-core platform that will support
> 600Mbps or more with Cake enabled, hopefully with AX radios as well.

Yup, the number of cores is only going to go up, so for CAKE to stay
relevant it'll need to be able to take advantage of this eventually :)

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Foulkes]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

> Right, so some benefit might be possible here. Does the NIC have
> multiple hardware queues (`ls /sys/class/net/$IFACE/queues` should tell
> you)?

Here is the output of:
/sys/devices/virtual/net/eth0.2/queues# ls
rx-0  tx-0
/sys/devices/virtual/net/eth0.2/queues/rx-0# cat rps_cpus 
0

/sys/devices/virtual/net/eth0.2/queues/tx-0# cat xps_cpus 
0

> Yup, the number of cores is only going to go up, so for CAKE to stay
> relevant it'll need to be able to take advantage of this eventually :)

True, the mid-range market is already there, and so soon will be the lower-end.
And with ISPs lighting up more and more capacity, the demand will be there to be able to shape higher and higher rates.

But I agree with Jonathan Morton that once every deice has sufficient capacity, more makes no difference. 
I went for 100/15 to 300/24 and never noticed the difference.

Hell, there are days I switch to my backup 10/0.7 DSL line for a test, and forget to switch back, and will work for hours and not notice I’m not on the 300Mbps line ;-)

Cheers,

Jonathan

> On Sep 1, 2020, at 5:11 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Jonathan Foulkes <jf@jonathanfoulkes.com> writes:
> 
>> Thanks Toke, we currently are on an MT7621a @880, so a dual-core.
> 
> Right, so some benefit might be possible here. Does the NIC have
> multiple hardware queues (`ls /sys/class/net/$IFACE/queues` should tell
> you)?
> 
>> And we are looking for a good quad-core platform that will support
>> 600Mbps or more with Cake enabled, hopefully with AX radios as well.
> 
> Yup, the number of cores is only going to go up, so for CAKE to stay
> relevant it'll need to be able to take advantage of this eventually :)
> 
> -Toke


[-- Attachment #2: Type: text/html, Size: 4629 bytes --]

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Jonathan Foulkes <jf@jonathanfoulkes.com> writes:

>> Right, so some benefit might be possible here. Does the NIC have
>> multiple hardware queues (`ls /sys/class/net/$IFACE/queues` should tell
>> you)?
>
> Here is the output of:
> /sys/devices/virtual/net/eth0.2/queues# ls
> rx-0  tx-0
> /sys/devices/virtual/net/eth0.2/queues/rx-0# cat rps_cpus 
> 0
>
> /sys/devices/virtual/net/eth0.2/queues/tx-0# cat xps_cpus 
> 0

Hmm, so no multiq support on this driver, it looks like. So not sure to
what extent it will be possible to effectively utilise both cores on
this box, sadly :/

>> Yup, the number of cores is only going to go up, so for CAKE to stay
>> relevant it'll need to be able to take advantage of this eventually :)
>
> True, the mid-range market is already there, and so soon will be the
> lower-end. And with ISPs lighting up more and more capacity, the
> demand will be there to be able to shape higher and higher rates.
>
> But I agree with Jonathan Morton that once every deice has sufficient
> capacity, more makes no difference. I went for 100/15 to 300/24 and
> never noticed the difference.
>
> Hell, there are days I switch to my backup 10/0.7 DSL line for a test,
> and forget to switch back, and will work for hours and not notice I’m
> not on the 300Mbps line ;-)

Heh, if you can live with a 10/0.7 line without noticing I think you're
more patient than me ;) But still, fair point; doesn't mean that people
will still not *want* to run a higher speeds, though... :)

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:

> And what about when you're running CAKE in 'unlimited' mode?

I tried this:

# tc qdisc add dev eth0 root cake bandwidth 900mbit

This seems fine from a performance point of view (not that high sirq%, 
around 35%) and does seem to limit my upstream traffic correctly. Not sure 
it helps though, at these speeds the bufferbloat problem is not that 
obvious and easy to test over the Internet :)

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
  backlog 0b 0p requeues 0
qdisc cake 8034: dev eth0 root refcnt 9 bandwidth 900Mbit diffserv3 
triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw 
overhead 0
  Sent 1111772001 bytes 959703 pkt (dropped 134, overlimits 221223 requeues 
179)
  backlog 0b 0p requeues 179
  memory used: 2751976b of 15140Kb
  capacity estimate: 900Mbit
  min/max network layer size:           42 /    1514
  min/max overhead-adjusted size:       42 /    1514
  average network hdr offset:           14

                    Bulk  Best Effort        Voice
   thresh      56250Kbit      900Mbit      225Mbit
   target          5.0ms        5.0ms        5.0ms
   interval      100.0ms      100.0ms      100.0ms
   pk_delay          0us         22us        232us
   av_delay          0us          6us          7us
   sp_delay          0us          4us          5us
   backlog            0b           0b           0b
   pkts                0       959747           90
   bytes               0   1111935437        39440
   way_inds            0        22964            0
   way_miss            0          275            2
   way_cols            0            0            0
   drops               0          134            0
   marks               0            0            0
   ack_drop            0            0            0
   sp_flows            0            3            1
   bk_flows            0            1            0
   un_flows            0            0            0
   max_len             0        68130         3714
   quantum          1514         1514         1514


-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] CAKE in openwrt high CPU

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:

> Yup, the number of cores is only going to go up, so for CAKE to stay 
> relevant it'll need to be able to take advantage of this eventually :)

https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform, 
it has a quad core machine with 2 x 2.5GbE NICs.

When using something like this for routing with HTB+CAKE for bidirectional 
shaping below line rate, what would be the main things that would need to 
be improved?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] CAKE in openwrt high CPU

[Luca Muscariello]

On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
<bloat@lists.bufferbloat.net> wrote:
>
> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
>
> > Yup, the number of cores is only going to go up, so for CAKE to stay
> > relevant it'll need to be able to take advantage of this eventually :)
>
> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform,
> it has a quad core machine with 2 x 2.5GbE NICs.
>
> When using something like this for routing with HTB+CAKE for bidirectional
> shaping below line rate, what would be the main things that would need to
> be improved?

IMO, hardware offloading for shaping, beyond this specific platform.
I ignore if there is any roadmap with that objective.

>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
>
>> Yup, the number of cores is only going to go up, so for CAKE to stay 
>> relevant it'll need to be able to take advantage of this eventually :)
>
> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform, 
> it has a quad core machine with 2 x 2.5GbE NICs.
>
> When using something like this for routing with HTB+CAKE for bidirectional 
> shaping below line rate, what would be the main things that would need to 
> be improved?

The aforementioned multi-processor support...

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:
>
>> And what about when you're running CAKE in 'unlimited' mode?
>
> I tried this:
>
> # tc qdisc add dev eth0 root cake bandwidth 900mbit

So the difference from before is just the lack of inbound shaping, or?

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

Hi Mikael,



> On Sep 3, 2020, at 15:10, Mikael Abrahamsson via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:
> 
>> And what about when you're running CAKE in 'unlimited' mode?
> 
> I tried this:
> 
> # tc qdisc add dev eth0 root cake bandwidth 900mbit

	That still employs the cake shaper, so is not equivalent with unlimited, I believe.

[PEDANT_MODE]

900 Mbps without explicit overhead will result in a typical maximum TCP/IPv4 goodput of

900 * ((1500-20-20)/(1500+14)) = 867.899603699 Mbps
but since ethernet overhead is actually 38 bytes instead of 14 this actually occupies 

(900 * ((1500-20-20)/(1500+14))) * ((1500+38)/(1500-20-20)) = 914.266842801 on the ethernet link

which for small packets will become problematic:
(900 * ((150-20-20)/(100+14))) * ((150+38)/(150-20-20)) = 1484.21052632 Mbps gross speed out of the 1000.0 Gigabit ethernet offers.

in fact, packet sizes below 202 will spend all the "credit" you got from reducing the shaper rate to 900 Mbps in the first place.
(900 * ((202-20-20)/(202 +14))) * ((202 +38)/(202-20-20)) = 1000  

Maybe tell cake that you run on ethernet by adding the "ethernet keyword" which will both take care of the per-packet overhead of 38 bytes and the minimum packet size on the link of 88 bytes?

Please note that for throughput this does not really matter that much, but latency-under-load is not going to be pretty when too many small packets are in flight...

[/PEDANT_MODE]


> 
> This seems fine from a performance point of view (not that high sirq%, around 35%) and does seem to limit my upstream traffic correctly. Not sure it helps though, at these speeds the bufferbloat problem is not that obvious and easy to test over the Internet :)

	Mmmh, how did you measure the sirq percentage? Some top versions show overall percentage with 100% meaning all CPUs, so 35% in a quadcore could mean 1 fully maxed out CPU (25%) plus an additional 10% spread over the other three, or something more benign. Better top (so not busybox's) or htop versions also can show the load per CPU which is helpful to pinpoint hotspots...

Best Regards
	Sebastian

> 
> root@OpenWrt:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc cake 8034: dev eth0 root refcnt 9 bandwidth 900Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
> Sent 1111772001 bytes 959703 pkt (dropped 134, overlimits 221223 requeues 179)
> backlog 0b 0p requeues 179
> memory used: 2751976b of 15140Kb
> capacity estimate: 900Mbit
> min/max network layer size:           42 /    1514
> min/max overhead-adjusted size:       42 /    1514
> average network hdr offset:           14
> 
>                   Bulk  Best Effort        Voice
>  thresh      56250Kbit      900Mbit      225Mbit
>  target          5.0ms        5.0ms        5.0ms
>  interval      100.0ms      100.0ms      100.0ms
>  pk_delay          0us         22us        232us
>  av_delay          0us          6us          7us
>  sp_delay          0us          4us          5us
>  backlog            0b           0b           0b
>  pkts                0       959747           90
>  bytes               0   1111935437        39440
>  way_inds            0        22964            0
>  way_miss            0          275            2
>  way_cols            0            0            0
>  drops               0          134            0
>  marks               0            0            0
>  ack_drop            0            0            0
>  sp_flows            0            3            1
>  bk_flows            0            1            0
>  un_flows            0            0            0
>  max_len             0        68130         3714
>  quantum          1514         1514         1514
> 
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

Ho Toke,

> On Sep 3, 2020, at 15:29, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Mikael Abrahamsson <swmike@swm.pp.se> writes:
> 
>> On Mon, 31 Aug 2020, Toke Høiland-Jørgensen wrote:
>> 
>>> And what about when you're running CAKE in 'unlimited' mode?
>> 
>> I tried this:
>> 
>> # tc qdisc add dev eth0 root cake bandwidth 900mbit
> 
> So the difference from before is just the lack of inbound shaping, or?

	Good point, so worst-case just half the load to handle, indicating that a single CPU is sufficient for gigabit shaping, but not for dual-gigabit shaping, no?

Best Regards
	Sebastian


> 
> -Toke
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

Luca Muscariello <muscariello@ieee.org> writes:

> On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
> <bloat@lists.bufferbloat.net> wrote:
>>
>> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
>>
>> > Yup, the number of cores is only going to go up, so for CAKE to stay
>> > relevant it'll need to be able to take advantage of this eventually :)
>>
>> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform,
>> it has a quad core machine with 2 x 2.5GbE NICs.
>>
>> When using something like this for routing with HTB+CAKE for bidirectional
>> shaping below line rate, what would be the main things that would need to
>> be improved?
>
> IMO, hardware offloading for shaping, beyond this specific platform.
> I ignore if there is any roadmap with that objective.

Yeah, offloading of some sort is another option, but I consider that
outside of the "CAKE stays relevant" territory, since that will most
likely involve an entirely programmable packet scheduler. There was some
discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
algorithm seems promising.

-Toke

[0] https://linuxplumbersconf.org/event/7/contributions/679/
[1] https://www.usenix.org/conference/nsdi19/presentation/saeed

Re: [Bloat] CAKE in openwrt high CPU

[Luca Muscariello]

On Thu, Sep 3, 2020 at 4:32 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Luca Muscariello <muscariello@ieee.org> writes:
>
> > On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
> > <bloat@lists.bufferbloat.net> wrote:
> >>
> >> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
> >>
> >> > Yup, the number of cores is only going to go up, so for CAKE to stay
> >> > relevant it'll need to be able to take advantage of this eventually :)
> >>
> >> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform,
> >> it has a quad core machine with 2 x 2.5GbE NICs.
> >>
> >> When using something like this for routing with HTB+CAKE for bidirectional
> >> shaping below line rate, what would be the main things that would need to
> >> be improved?
> >
> > IMO, hardware offloading for shaping, beyond this specific platform.
> > I ignore if there is any roadmap with that objective.
>
> Yeah, offloading of some sort is another option, but I consider that
> outside of the "CAKE stays relevant" territory, since that will most
> likely involve an entirely programmable packet scheduler. There was some
> discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
> algorithm seems promising.
>
> -Toke
>
> [0] https://linuxplumbersconf.org/event/7/contributions/679/
> [1] https://www.usenix.org/conference/nsdi19/presentation/saeed

These are all interesting efforts for scheduling but orthogonal to shaping
and not going to help make shaping more scalable.

Re: [Bloat] CAKE in openwrt high CPU

[Toke Høiland-Jørgensen]

On 3 September 2020 17:31:07 CEST, Luca Muscariello <muscariello@ieee.org> wrote:
>On Thu, Sep 3, 2020 at 4:32 PM Toke Høiland-Jørgensen <toke@toke.dk>
>wrote:
>>
>> Luca Muscariello <muscariello@ieee.org> writes:
>>
>> > On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
>> > <bloat@lists.bufferbloat.net> wrote:
>> >>
>> >> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
>> >>
>> >> > Yup, the number of cores is only going to go up, so for CAKE to
>stay
>> >> > relevant it'll need to be able to take advantage of this
>eventually :)
>> >>
>> >> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting
>platform,
>> >> it has a quad core machine with 2 x 2.5GbE NICs.
>> >>
>> >> When using something like this for routing with HTB+CAKE for
>bidirectional
>> >> shaping below line rate, what would be the main things that would
>need to
>> >> be improved?
>> >
>> > IMO, hardware offloading for shaping, beyond this specific
>platform.
>> > I ignore if there is any roadmap with that objective.
>>
>> Yeah, offloading of some sort is another option, but I consider that
>> outside of the "CAKE stays relevant" territory, since that will most
>> likely involve an entirely programmable packet scheduler. There was
>some
>> discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
>> algorithm seems promising.
>>
>> -Toke
>>
>> [0] https://linuxplumbersconf.org/event/7/contributions/679/
>> [1] https://www.usenix.org/conference/nsdi19/presentation/saeed
>
>These are all interesting efforts for scheduling but orthogonal to
>shaping
>and not going to help make shaping more scalable.

Eiffel says it can do shaping by way of a global calendar queue... Planning to put that to the test :)

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Jonathan Morton]

> On 3 Sep, 2020, at 5:32 pm, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Yeah, offloading of some sort is another option, but I consider that
> outside of the "CAKE stays relevant" territory, since that will most
> likely involve an entirely programmable packet scheduler.

Offload of *just* shaping could be valuable in itself at higher rates, when combined with BQL, as it would avoid having to interact with the CPU-side timer infrastructure so much.  It would also not be difficult at all to implement in hardware at line rate, even with overhead compensation.  It's the sort of thing you could sensibly do with 74-series logic and a lookup table in a cheap SRAM, up to millions of PPS, and considerably faster in FPGA or ASIC territory.

I think that's what the questions about combining "unlimited Cake" with some other shaper are angling towards, though I suspect that the way Cake's shaper is integrated is still better than having an external one in software.

With that said, it's also possible that something a bit lighter than Cake might be appropriate at cable speeds.  There is background work in this general area going on, so don't despair.

 - Jonathan Morton

[Bloat] Other CAKE territory (was: CAKE in openwrt high CPU)

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On 2020-09-03 10:32 a.m., Toke Høiland-Jørgensen via Bloat wrote

> Yeah, offloading of some sort is another option, but I consider that
> outside of the "CAKE stays relevant" territory, since that will most
> likely involve an entirely programmable packet scheduler. There was some
> discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
> algorithm seems promising.
>
> -Toke

I'm wondering if edge servers with 1Gb NICs are inside the "CAKE stays 
relevant" territory?

My main customer/employer has a gazillion of those, currently reporting

**

*qdisc mq 0: root*

*

qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 
1 1 1

...

*

because their OS is just a tiny bit elderly (;-)). We we're planning to 
roll forward this quarter to centos 8.2, where CAKE is an option.

It strikes me that the self-tuning capacity of CAKE might be valuable 
for a whole /class/ of small rack-mounted machines, but you just 
mentioned the desire for better multi-processor support.

Am I reaching for the moon, or is this something within reach?

--dave

-- 

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain


[-- Attachment #2: Type: text/html, Size: 3003 bytes --]

Re: [Bloat] Other CAKE territory (was: CAKE in openwrt high CPU)

[Jonathan Morton]

> On 4 Sep, 2020, at 1:14 am, David Collier-Brown <davecb.42@gmail.com> wrote:
> 
> I'm wondering if edge servers with 1Gb NICs are inside the "CAKE stays relevant" territory?  

Edge servers usually have strong enough CPUs and I/O - by which I mean anything from AMD K8 and Intel Core 2 onwards with PCIe attached NICs - to run Cake at 1Gbps without needing special measures.  I should run a test to see how much I can shove through an AMD Bobcat these days - not exactly a speed demon.

We're usually seeing problems with the smaller-scale CPUs found in CPE SoCs, which are very much geared to take advantage of hardware accelerated packet forwarding.  I think in some cases there might actually be insufficient internal I/O bandwidth to get 1Gbps out of the NIC, into the CPU, and back out to the NIC again, only through the dedicated forwarding path.  That could manifest itself as a lot of kernel time spent waiting for the hardware, and can only really be solved by redesigning the hardware.

 - Jonathan Morton

Re: [Bloat] Other CAKE territory (was: CAKE in openwrt high CPU)

[Toke Høiland-Jørgensen]

David Collier-Brown <davecb.42@gmail.com> writes:

> On 2020-09-03 10:32 a.m., Toke Høiland-Jørgensen via Bloat wrote
>
>> Yeah, offloading of some sort is another option, but I consider that
>> outside of the "CAKE stays relevant" territory, since that will most
>> likely involve an entirely programmable packet scheduler. There was some
>> discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
>> algorithm seems promising.
>>
>> -Toke
>
> I'm wondering if edge servers with 1Gb NICs are inside the "CAKE stays 
> relevant" territory?
>
> My main customer/employer has a gazillion of those, currently reporting
>
> **
>
> *qdisc mq 0: root*
>
> *
>
> qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 
> 1 1 1
>
> ...
>
> *
>
> because their OS is just a tiny bit elderly (;-)). We we're planning to 
> roll forward this quarter to centos 8.2, where CAKE is an option.
>
> It strikes me that the self-tuning capacity of CAKE might be valuable 
> for a whole /class/ of small rack-mounted machines, but you just 
> mentioned the desire for better multi-processor support.
>
> Am I reaching for the moon, or is this something within reach?

As Jonathan says, servers mostly have enough CPU that running at 1gbps
is not an issue. And especially if you're not shaping, running CAKE in
unlimited mode should not be an issue.

However, do consider what you're trying to achieve here. Most of the
specific features of CAKE are targeting gateway routers. For instance,
for a server you may be better off with sch_fq to also get efficient
pacing support. Depends on what the server is doing...

But please, get rid of pfifo_fast! Anything is better than that! ;)

-Toke

Re: [Bloat] CAKE in openwrt high CPU

[Mikael Abrahamsson]

On Thu, 3 Sep 2020, Sebastian Moeller wrote:

> 	Mmmh, how did you measure the sirq percentage? Some top versions 
> show overall percentage with 100% meaning all CPUs, so 35% in a quadcore 
> could mean 1 fully maxed out CPU (25%) plus an additional 10% spread 
> over the other three, or something more benign. Better top (so not 
> busybox's) or htop versions also can show the load per CPU which is 
> helpful to pinpoint hotspots...

If I run iperf3 with 10 parallel sessions then htop shows this (in the 
CAKE upstream direction I believe):

   1  [*                                                                                  0.7%]   Tasks: 19, 0 thr; 2 running
   2  [*********************************************************************************100.0%]   Load average: 0.48 0.16 0.05
   3  [#***************************************                                          44.4%]   Uptime: 10 days, 04:46:37
   4  [************************************************                                  54.2%]
   Mem[|#*                                                                         36.7M/3.84G]
   Swp[                                                                                  0K/0K]

The other direction (-R), typically this:

  1  [#***********                                                                      13.0%]   Tasks: 19, 0 thr; 2 running
  2  [***********************************************                                   53.9%]   Load average: 0.54 0.25 0.09
  3  [#*************************************************                                55.8%]   Uptime: 10 days, 04:47:36
  4  [**************************************************************************        84.4%]

Topology is:

PC - HGW -> Internet

iperf3 is run on the PC, HGW has CAKE in the -> Internet direction.

> Best Regards
> 	Sebastian
>
>>
>> root@OpenWrt:~# tc -s qdisc
>> qdisc noqueue 0: dev lo root refcnt 2
>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>> backlog 0b 0p requeues 0
>> qdisc cake 8034: dev eth0 root refcnt 9 bandwidth 900Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
>> Sent 1111772001 bytes 959703 pkt (dropped 134, overlimits 221223 requeues 179)
>> backlog 0b 0p requeues 179
>> memory used: 2751976b of 15140Kb
>> capacity estimate: 900Mbit
>> min/max network layer size:           42 /    1514
>> min/max overhead-adjusted size:       42 /    1514
>> average network hdr offset:           14
>>
>>                   Bulk  Best Effort        Voice
>>  thresh      56250Kbit      900Mbit      225Mbit
>>  target          5.0ms        5.0ms        5.0ms
>>  interval      100.0ms      100.0ms      100.0ms
>>  pk_delay          0us         22us        232us
>>  av_delay          0us          6us          7us
>>  sp_delay          0us          4us          5us
>>  backlog            0b           0b           0b
>>  pkts                0       959747           90
>>  bytes               0   1111935437        39440
>>  way_inds            0        22964            0
>>  way_miss            0          275            2
>>  way_cols            0            0            0
>>  drops               0          134            0
>>  marks               0            0            0
>>  ack_drop            0            0            0
>>  sp_flows            0            3            1
>>  bk_flows            0            1            0
>>  un_flows            0            0            0
>>  max_len             0        68130         3714
>>  quantum          1514         1514         1514
>>
>>
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] Other CAKE territory (was: CAKE in openwrt high CPU)

[Mikael Abrahamsson]

On Fri, 4 Sep 2020, Jonathan Morton wrote:

> We're usually seeing problems with the smaller-scale CPUs found in CPE 
> SoCs, which are very much geared to take advantage of hardware 
> accelerated packet forwarding.  I think in some cases there might 
> actually be insufficient internal I/O bandwidth to get 1Gbps out of the 
> NIC, into the CPU, and back out to the NIC again, only through the 
> dedicated forwarding path.  That could manifest itself as a lot of 
> kernel time spent waiting for the hardware, and can only really be 
> solved by redesigning the hardware.

There are lots of SoCs where CPU routing results in ~100 megabit/s of 
throughput, whilst the HW offload engine is perfectly capable of full gig 
speeds. MT7621 being one that actually is supported in OpenWrt.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] CAKE in openwrt high CPU

[Sebastian Moeller]

Hi Mikael,

Thanks! That looks like a fully saturated core, no? I do not know how to parse the symbols here, so not sure what "class" of load is denoted by the star, but I would guess something including sirqs? Anyway the average is ~49% load, while clearly CPU is pegged already. I assume the htop data is from the HGW...

best regards
	Sebastian

> On Sep 4, 2020, at 15:37, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Thu, 3 Sep 2020, Sebastian Moeller wrote:
> 
>> 	Mmmh, how did you measure the sirq percentage? Some top versions show overall percentage with 100% meaning all CPUs, so 35% in a quadcore could mean 1 fully maxed out CPU (25%) plus an additional 10% spread over the other three, or something more benign. Better top (so not busybox's) or htop versions also can show the load per CPU which is helpful to pinpoint hotspots...
> 
> If I run iperf3 with 10 parallel sessions then htop shows this (in the CAKE upstream direction I believe):
> 
>  1  [*                                                                                  0.7%]   Tasks: 19, 0 thr; 2 running
>  2  [*********************************************************************************100.0%]   Load average: 0.48 0.16 0.05
>  3  [#***************************************                                          44.4%]   Uptime: 10 days, 04:46:37
>  4  [************************************************                                  54.2%]
>  Mem[|#*                                                                         36.7M/3.84G]
>  Swp[                                                                                  0K/0K]
> 
> The other direction (-R), typically this:
> 
> 1  [#***********                                                                      13.0%]   Tasks: 19, 0 thr; 2 running
> 2  [***********************************************                                   53.9%]   Load average: 0.54 0.25 0.09
> 3  [#*************************************************                                55.8%]   Uptime: 10 days, 04:47:36
> 4  [**************************************************************************        84.4%]
> 
> Topology is:
> 
> PC - HGW -> Internet
> 
> iperf3 is run on the PC, HGW has CAKE in the -> Internet direction.
> 
>> Best Regards
>> 	Sebastian
>> 
>>> 
>>> root@OpenWrt:~# tc -s qdisc
>>> qdisc noqueue 0: dev lo root refcnt 2
>>> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>> backlog 0b 0p requeues 0
>>> qdisc cake 8034: dev eth0 root refcnt 9 bandwidth 900Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
>>> Sent 1111772001 bytes 959703 pkt (dropped 134, overlimits 221223 requeues 179)
>>> backlog 0b 0p requeues 179
>>> memory used: 2751976b of 15140Kb
>>> capacity estimate: 900Mbit
>>> min/max network layer size:           42 /    1514
>>> min/max overhead-adjusted size:       42 /    1514
>>> average network hdr offset:           14
>>> 
>>>                  Bulk  Best Effort        Voice
>>> thresh      56250Kbit      900Mbit      225Mbit
>>> target          5.0ms        5.0ms        5.0ms
>>> interval      100.0ms      100.0ms      100.0ms
>>> pk_delay          0us         22us        232us
>>> av_delay          0us          6us          7us
>>> sp_delay          0us          4us          5us
>>> backlog            0b           0b           0b
>>> pkts                0       959747           90
>>> bytes               0   1111935437        39440
>>> way_inds            0        22964            0
>>> way_miss            0          275            2
>>> way_cols            0            0            0
>>> drops               0          134            0
>>> marks               0            0            0
>>> ack_drop            0            0            0
>>> sp_flows            0            3            1
>>> bk_flows            0            1            0
>>> un_flows            0            0            0
>>> max_len             0        68130         3714
>>> quantum          1514         1514         1514
>>> 
>>> 
>>> --
>>> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> 
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se