Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Jose Blanquicet <blanquicet@gmail.com> writes:

> Hi Toke,
>
> Thanks for your reply.
>
> On Mon, Jun 22, 2020 at 5:47 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>> > We have an embedded system with limited CPU resources that acts as a
>> > gateway to provide Internet access from LTE to a private USB-NCM
>> > network (And also to a Wi-Fi private network but we will work on it
>> > later). Our problem is that the bandwidth on LTE and USB link is
>> > higher than what the system is able to handle thus it reaches 100% of
>> > CPU load when we perform a simple speed test from a device on the
>> > private network.
>>
>> What speeds were you getting without shaping?
>
> Between 35 and 40Mbps.
>
>> > Therefore, we want to limit the bandwidth to avoid system getting
>> > saturated in such use-case. To do so, we thought to use the CAKE on
>> > the USB interface. For instance, we tried:
>> >
>> >     tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet
>> > internet flowblind nonat besteffort nowash
>> >
>> > It worked correctly and the maximum rate was limited but there are two
>> > things that are worrying us:
>> >
>> > 1) The maximum rate reached after applying CAKE was in between 12Mbps
>> > and 15Mbps which is quite lower than the 20Mbps we are configuring, we
>> > were expecting around 18-19. Why? Is there something in the parameters
>> > we are doing wrong? Please take into account that our goal is to limit
>> > the rate but adding as little CPU load as possible.
>>
>> Hmm, are you actually running out of CPU? I.e., is the CPU pegged at
>> 100% when you hit this limit? What kind of platform are you running on?
>> And what kernel and CAKE versions are you using?
>
> I checked the CPU with top and there is still free CPU to be used. We
> also tried with lower values like 10 and it is again far away from the
> configured limit.
>
> We have just a percentage of an ARM Cortex A7 (1.2GHz) because the
> rest is reserved for modem. We are now trying to optimize all the
> applications in the system but LTE<->WIFI/USB data transfer is indeed
> the
> use-case that puts our system in crisis.
>
> The kernel version is 3.18 and for we are using the latest commit on
> master branch (9d79e2b) for CAKE. In case, we could change CAKE but
> not the kernel version, at most some specific patches.

Right, well if you're not running out of CPU I guess it could be a
timing issue. The CAKE shaper relies on accurate timestamps and the
qdisc watchdog timer to schedule the transmission of packets. A loaded
system can simply miss deadlines, which would be consistent with the
behaviour you're seeing.

In fact, when looking into this a bit more, I came across this commit
that seemed to observe the same behaviour in sch_fq:
https://git.kernel.org/torvalds/c/fefa569a9d4b

So I guess we could try to do something similar in CAKE.

Could you please post the output of 'tc -s qdisc' after a test run? That
should give some indication on how much the shaper is throttling...

>> > 2) The CPU load added by CAKE was not negligible for our system. In
>> > fact, we compared the CPU load when limitation was done by CAKE and by
>> > the device on the private network, e.g. curl tool with parameter
>> > "--limit-rate". As a result, we found that the CPU load when using
>> > CAKE was 30%. Is there any way to make it lighter with a different
>> > configuration?
>>
>> No, you've already turned off most of the features that might incur
>> overhead, so I don't think there's anything more you can do
>> configuration-wise to improve CPU load. Shaping does tend to use up a
>> lot of CPU, so it's not too surprising you run into issues here.
>
> Could you please help us to identify which one is still active? We
> thought we had already turned off all the features not needed to apply
> a limitation with a single queue (Besteffor mode).

Well the only thing more you can turn off by configuration is the shaper
itself :)

>> We did recently get a pull request whose author states that he was
>> seeing a 1/3 improvement in performance from it. See:
>> https://github.com/dtaht/sch_cake/pull/136
>>
>> You could try this; if your ingress network device driver has the same
>> issue with skbs being allocated in smaller bits, you may see a similar
>> increase with this patch. For a quick test you could also just try
>> commenting out the call to cake_handle_diffserv() entirely since you're
>> running in besteffort mode anyway :)
>
> Interesting. We will try this, we commented out the call to
> cake_handle_diffserv() as you said and just to be sure, we also
> applied the 2nd commit of the PR. I will be back soon with news.

OK, great!

-Toke