[Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Jose Blanquicet]

Hi everyone,

We have an embedded system with limited CPU resources that acts as a
gateway to provide Internet access from LTE to a private USB-NCM
network (And also to a Wi-Fi private network but we will work on it
later). Our problem is that the bandwidth on LTE and USB link is
higher than what the system is able to handle thus it reaches 100% of
CPU load when we perform a simple speed test from a device on the
private network.

Therefore, we want to limit the bandwidth to avoid system getting
saturated in such use-case. To do so, we thought to use the CAKE on
the USB interface. For instance, we tried:

    tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
internet flowblind nonat besteffort nowash

It worked correctly and the maximum rate was limited but there are two
things that are worrying us:

1) The maximum rate reached after applying CAKE was in between 12Mbps
and 15Mbps which is quite lower than the 20Mbps we are configuring, we
were expecting around 18-19. Why? Is there something in the parameters
we are doing wrong? Please take into account that our goal is to limit
the rate but adding as little CPU load as possible.

2) The CPU load added by CAKE was not negligible for our system. In
fact, we compared the CPU load when limitation was done by CAKE and by
the device on the private network, e.g. curl tool with parameter
"--limit-rate". As a result, we found that the CPU load when using
CAKE was 30%. Is there any way to make it lighter with a different
configuration?

Thanks in advance for the support. Any suggestion is welcome.

Jose Blanquicet

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Y]

You should paste this result.

tc -s qdisc show dev eth0

Yutaka

On 22/06/2020 22:10, Jose Blanquicet wrote:
> Hi everyone,
> 
> We have an embedded system with limited CPU resources that acts as a
> gateway to provide Internet access from LTE to a private USB-NCM
> network (And also to a Wi-Fi private network but we will work on it
> later). Our problem is that the bandwidth on LTE and USB link is
> higher than what the system is able to handle thus it reaches 100% of
> CPU load when we perform a simple speed test from a device on the
> private network.
> 
> Therefore, we want to limit the bandwidth to avoid system getting
> saturated in such use-case. To do so, we thought to use the CAKE on
> the USB interface. For instance, we tried:
> 
>      tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
> internet flowblind nonat besteffort nowash
> 
> It worked correctly and the maximum rate was limited but there are two
> things that are worrying us:
> 
> 1) The maximum rate reached after applying CAKE was in between 12Mbps
> and 15Mbps which is quite lower than the 20Mbps we are configuring, we
> were expecting around 18-19. Why? Is there something in the parameters
> we are doing wrong? Please take into account that our goal is to limit
> the rate but adding as little CPU load as possible.
> 
> 2) The CPU load added by CAKE was not negligible for our system. In
> fact, we compared the CPU load when limitation was done by CAKE and by
> the device on the private network, e.g. curl tool with parameter
> "--limit-rate". As a result, we found that the CPU load when using
> CAKE was 30%. Is there any way to make it lighter with a different
> configuration?
> 
> Thanks in advance for the support. Any suggestion is welcome.
> 
> Jose Blanquicet
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> 

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Toke Høiland-Jørgensen]

Jose Blanquicet <blanquicet@gmail.com> writes:

> Hi everyone,
>
> We have an embedded system with limited CPU resources that acts as a
> gateway to provide Internet access from LTE to a private USB-NCM
> network (And also to a Wi-Fi private network but we will work on it
> later). Our problem is that the bandwidth on LTE and USB link is
> higher than what the system is able to handle thus it reaches 100% of
> CPU load when we perform a simple speed test from a device on the
> private network.

What speeds were you getting without shaping?

> Therefore, we want to limit the bandwidth to avoid system getting
> saturated in such use-case. To do so, we thought to use the CAKE on
> the USB interface. For instance, we tried:
>
>     tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
> internet flowblind nonat besteffort nowash
>
> It worked correctly and the maximum rate was limited but there are two
> things that are worrying us:
>
> 1) The maximum rate reached after applying CAKE was in between 12Mbps
> and 15Mbps which is quite lower than the 20Mbps we are configuring, we
> were expecting around 18-19. Why? Is there something in the parameters
> we are doing wrong? Please take into account that our goal is to limit
> the rate but adding as little CPU load as possible.

Hmm, are you actually running out of CPU? I.e., is the CPU pegged at
100% when you hit this limit? What kind of platform are you running on?
And what kernel and CAKE versions are you using?

> 2) The CPU load added by CAKE was not negligible for our system. In
> fact, we compared the CPU load when limitation was done by CAKE and by
> the device on the private network, e.g. curl tool with parameter
> "--limit-rate". As a result, we found that the CPU load when using
> CAKE was 30%. Is there any way to make it lighter with a different
> configuration?

No, you've already turned off most of the features that might incur
overhead, so I don't think there's anything more you can do
configuration-wise to improve CPU load. Shaping does tend to use up a
lot of CPU, so it's not too surprising you run into issues here.

We did recently get a pull request whose author states that he was
seeing a 1/3 improvement in performance from it. See:
https://github.com/dtaht/sch_cake/pull/136

You could try this; if your ingress network device driver has the same
issue with skbs being allocated in smaller bits, you may see a similar
increase with this patch. For a quick test you could also just try
commenting out the call to cake_handle_diffserv() entirely since you're
running in besteffort mode anyway :)

-Toke

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Jose Blanquicet]

Hi Toke,

Thanks for your reply.

On Mon, Jun 22, 2020 at 5:47 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> > We have an embedded system with limited CPU resources that acts as a
> > gateway to provide Internet access from LTE to a private USB-NCM
> > network (And also to a Wi-Fi private network but we will work on it
> > later). Our problem is that the bandwidth on LTE and USB link is
> > higher than what the system is able to handle thus it reaches 100% of
> > CPU load when we perform a simple speed test from a device on the
> > private network.
>
> What speeds were you getting without shaping?

Between 35 and 40Mbps.

> > Therefore, we want to limit the bandwidth to avoid system getting
> > saturated in such use-case. To do so, we thought to use the CAKE on
> > the USB interface. For instance, we tried:
> >
> >     tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
> > internet flowblind nonat besteffort nowash
> >
> > It worked correctly and the maximum rate was limited but there are two
> > things that are worrying us:
> >
> > 1) The maximum rate reached after applying CAKE was in between 12Mbps
> > and 15Mbps which is quite lower than the 20Mbps we are configuring, we
> > were expecting around 18-19. Why? Is there something in the parameters
> > we are doing wrong? Please take into account that our goal is to limit
> > the rate but adding as little CPU load as possible.
>
> Hmm, are you actually running out of CPU? I.e., is the CPU pegged at
> 100% when you hit this limit? What kind of platform are you running on?
> And what kernel and CAKE versions are you using?

I checked the CPU with top and there is still free CPU to be used. We
also tried with lower values like 10 and it is again far away from the
configured limit.

We have just a percentage of an ARM Cortex A7 (1.2GHz) because the
rest is reserved for modem. We are now trying to optimize all the
applications in the system but LTE<->WIFI/USB data transfer is indeed
the
use-case that puts our system in crisis.

The kernel version is 3.18 and for we are using the latest commit on
master branch (9d79e2b) for CAKE. In case, we could change CAKE but
not the kernel version, at most some specific patches.

> > 2) The CPU load added by CAKE was not negligible for our system. In
> > fact, we compared the CPU load when limitation was done by CAKE and by
> > the device on the private network, e.g. curl tool with parameter
> > "--limit-rate". As a result, we found that the CPU load when using
> > CAKE was 30%. Is there any way to make it lighter with a different
> > configuration?
>
> No, you've already turned off most of the features that might incur
> overhead, so I don't think there's anything more you can do
> configuration-wise to improve CPU load. Shaping does tend to use up a
> lot of CPU, so it's not too surprising you run into issues here.

Could you please help us to identify which one is still active? We
thought we had already turned off all the features not needed to apply
a limitation with a single queue (Besteffor mode).

> We did recently get a pull request whose author states that he was
> seeing a 1/3 improvement in performance from it. See:
> https://github.com/dtaht/sch_cake/pull/136
>
> You could try this; if your ingress network device driver has the same
> issue with skbs being allocated in smaller bits, you may see a similar
> increase with this patch. For a quick test you could also just try
> commenting out the call to cake_handle_diffserv() entirely since you're
> running in besteffort mode anyway :)

Interesting. We will try this, we commented out the call to
cake_handle_diffserv() as you said and just to be sure, we also
applied the 2nd commit of the PR. I will be back soon with news.

Thanks,
Jose Blanquicet

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Toke Høiland-Jørgensen]

Jose Blanquicet <blanquicet@gmail.com> writes:

> Hi Toke,
>
> Thanks for your reply.
>
> On Mon, Jun 22, 2020 at 5:47 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> > We have an embedded system with limited CPU resources that acts as a
>> > gateway to provide Internet access from LTE to a private USB-NCM
>> > network (And also to a Wi-Fi private network but we will work on it
>> > later). Our problem is that the bandwidth on LTE and USB link is
>> > higher than what the system is able to handle thus it reaches 100% of
>> > CPU load when we perform a simple speed test from a device on the
>> > private network.
>>
>> What speeds were you getting without shaping?
>
> Between 35 and 40Mbps.
>
>> > Therefore, we want to limit the bandwidth to avoid system getting
>> > saturated in such use-case. To do so, we thought to use the CAKE on
>> > the USB interface. For instance, we tried:
>> >
>> >     tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
>> > internet flowblind nonat besteffort nowash
>> >
>> > It worked correctly and the maximum rate was limited but there are two
>> > things that are worrying us:
>> >
>> > 1) The maximum rate reached after applying CAKE was in between 12Mbps
>> > and 15Mbps which is quite lower than the 20Mbps we are configuring, we
>> > were expecting around 18-19. Why? Is there something in the parameters
>> > we are doing wrong? Please take into account that our goal is to limit
>> > the rate but adding as little CPU load as possible.
>>
>> Hmm, are you actually running out of CPU? I.e., is the CPU pegged at
>> 100% when you hit this limit? What kind of platform are you running on?
>> And what kernel and CAKE versions are you using?
>
> I checked the CPU with top and there is still free CPU to be used. We
> also tried with lower values like 10 and it is again far away from the
> configured limit.
>
> We have just a percentage of an ARM Cortex A7 (1.2GHz) because the
> rest is reserved for modem. We are now trying to optimize all the
> applications in the system but LTE<->WIFI/USB data transfer is indeed
> the
> use-case that puts our system in crisis.
>
> The kernel version is 3.18 and for we are using the latest commit on
> master branch (9d79e2b) for CAKE. In case, we could change CAKE but
> not the kernel version, at most some specific patches.

Right, well if you're not running out of CPU I guess it could be a
timing issue. The CAKE shaper relies on accurate timestamps and the
qdisc watchdog timer to schedule the transmission of packets. A loaded
system can simply miss deadlines, which would be consistent with the
behaviour you're seeing.

In fact, when looking into this a bit more, I came across this commit
that seemed to observe the same behaviour in sch_fq:
https://git.kernel.org/torvalds/c/fefa569a9d4b

So I guess we could try to do something similar in CAKE.

Could you please post the output of 'tc -s qdisc' after a test run? That
should give some indication on how much the shaper is throttling...

>> > 2) The CPU load added by CAKE was not negligible for our system. In
>> > fact, we compared the CPU load when limitation was done by CAKE and by
>> > the device on the private network, e.g. curl tool with parameter
>> > "--limit-rate". As a result, we found that the CPU load when using
>> > CAKE was 30%. Is there any way to make it lighter with a different
>> > configuration?
>>
>> No, you've already turned off most of the features that might incur
>> overhead, so I don't think there's anything more you can do
>> configuration-wise to improve CPU load. Shaping does tend to use up a
>> lot of CPU, so it's not too surprising you run into issues here.
>
> Could you please help us to identify which one is still active? We
> thought we had already turned off all the features not needed to apply
> a limitation with a single queue (Besteffor mode).

Well the only thing more you can turn off by configuration is the shaper
itself :)

>> We did recently get a pull request whose author states that he was
>> seeing a 1/3 improvement in performance from it. See:
>> https://github.com/dtaht/sch_cake/pull/136
>>
>> You could try this; if your ingress network device driver has the same
>> issue with skbs being allocated in smaller bits, you may see a similar
>> increase with this patch. For a quick test you could also just try
>> commenting out the call to cake_handle_diffserv() entirely since you're
>> running in besteffort mode anyway :)
>
> Interesting. We will try this, we commented out the call to
> cake_handle_diffserv() as you said and just to be sure, we also
> applied the 2nd commit of the PR. I will be back soon with news.

OK, great!

-Toke

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Jonathan Morton]

> On 23 Jun, 2020, at 5:41 pm, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
> 
> Right, well if you're not running out of CPU I guess it could be a
> timing issue. The CAKE shaper relies on accurate timestamps and the
> qdisc watchdog timer to schedule the transmission of packets. A loaded
> system can simply miss deadlines, which would be consistent with the
> behaviour you're seeing.
> 
> In fact, when looking into this a bit more, I came across this commit
> that seemed to observe the same behaviour in sch_fq:
> https://git.kernel.org/torvalds/c/fefa569a9d4b
> 
> So I guess we could try to do something similar in CAKE.

Actually, we already do.  The first version of Cake's shaper was based closely on the one in sch_fq at the time, and I modified it after noticing that it had a very limited maximum throughput when timer resolution was poor (eg. at 1kHz on an old PC without HPET hardware, we could only get 1k pps).  Now, any late timer event will result in a burst being issued to catch up with the proper schedule.  The only time that wouldn't work is if the queue is empty.

If the patches currently being trialled are not sufficient, then perhaps we could try something counter-intuitive: switch on "flows" instead of "flowblind", and enable the ack-filter.  That should result in fewer small packets to process, as the ack-filter will coalesce some acks, especially under load.  It might also help to select "satellite" AQM parameters, reducing the amount of processing needed at that layer.

 - Jonathan Morton

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Sebastian Moeller]

Hi Jonathan,

> On Jun 23, 2020, at 17:21, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 23 Jun, 2020, at 5:41 pm, Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> 
>> Right, well if you're not running out of CPU I guess it could be a
>> timing issue. The CAKE shaper relies on accurate timestamps and the
>> qdisc watchdog timer to schedule the transmission of packets. A loaded
>> system can simply miss deadlines, which would be consistent with the
>> behaviour you're seeing.
>> 
>> In fact, when looking into this a bit more, I came across this commit
>> that seemed to observe the same behaviour in sch_fq:
>> https://git.kernel.org/torvalds/c/fefa569a9d4b
>> 
>> So I guess we could try to do something similar in CAKE.
> 
> Actually, we already do.  The first version of Cake's shaper was based closely on the one in sch_fq at the time, and I modified it after noticing that it had a very limited maximum throughput when timer resolution was poor (eg. at 1kHz on an old PC without HPET hardware, we could only get 1k pps).  Now, any late timer event will result in a burst being issued to catch up with the proper schedule.  The only time that wouldn't work is if the queue is empty.

	This nicely and effortlessly explains why cake unlike HTB+fq_codel maintains the set bandwidth better under CPU load (but then these burst also increase latency under load a bit more; then again again, we had to make the burst buffering for HTB configurable so it would not be as bad in dropping bandwidth on the floor). But I assume that you bound the bursts somehow, do you remember your bust sizing method by chance? (For HTB/TBF sqm now simply allows the user to configure a maximum burst service time im microseconds, that at least allows to make a conscious trade-off).


> 
> If the patches currently being trialled are not sufficient, then perhaps we could try something counter-intuitive: switch on "flows" instead of "flowblind", and enable the ack-filter.  That should result in fewer small packets to process, as the ack-filter will coalesce some acks, especially under load.  It might also help to select "satellite" AQM parameters, reducing the amount of processing needed at that layer.
> 
> - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

[Jonathan Morton]

> On 23 Jun, 2020, at 7:08 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> But I assume that you bound the bursts somehow, do you remember your bust sizing method by chance?

It bursts exactly enough to catch up to the schedule.  No more, no less - unless the queue is emptied in the process.

 - Jonathan Morton