[Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Jose Blanquicet]

Hi everyone,

We have an embedded system with limited CPU resources that acts as
gateway to provide Internet access from LTE to a private Wi-Fi
network. Our problem is that the bandwidth on LTE and Wi-Fi links is
higher than what the system is able to handle thus it reaches 100% of
CPU load when we perform a simple speed test from a device connected
to our Wi-Fi Hotspot.

Therefore, we want to limit the bandwidth to avoid system gets
saturated is such use-case. To do so, we thought to use the QDISC-TBF
on the Wi-Fi interface. For instance, to have 10Mbps:

    tc qdisc add dev wlan0 root tbf rate 10mbit burst 12500b latency 50ms

It worked correctly and maximum rate was limited to 10Mbps. However,
we noticed that the CPU load added by the TBF was not negligible for
our system. In fact, we compared the CPU load when limitation was done
by TBF and by the device on the private network, e.g. wget tool with
parameter "--limit-rate". As result, we found that the CPU load when
using TBF was 10-15% higher.

Then, we thought that using traffic shaping in egress, packets need to
be un-natted (which takes CPU) and pass through the system to then get
dropped. Therefore, we tried to use an incoming policer instead of
egress traffic shaping as following:

    tc qdisc add dev eth0 ingress handle ffff:
    tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0
police rate 10mbit burst 1m drop

Unfortunately, as per egress traffic shaping, we still obtained a high
CPU cost because of rate limiting. However, also in this case, we are
not sure we chose the most efficient option in terms of CPU cost to
police in ingress.

Given that, we were wondering if we are doing wrong by choosing TBF?
Or maybe we are using wrong parameters? We found everywhere that TBF
is the simplest way to limit the rate thus we suppose it is also the
most efficient QDISC. Is our supposition correct? Or there is another
way to limit rate well-known by its low CPU consumption? Any
suggestion is welcome, just taking into account we are using
libnl-3.2.28 and linux-kernel 3.18. In case, we could change libnl but
not kernel version, at most some specific patches.

Thanks in advance for the support!

Jose Blanquicet

Re: [Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Y]

[-- Attachment #1: Type: text/plain, Size: 2588 bytes --]

Hello

I heard that IBF costs cpu load.

How about shaping only egress with TBF?

Yutaka.

On 2020/05/26 18:47, Jose Blanquicet wrote:
> Hi everyone,
>
> We have an embedded system with limited CPU resources that acts as
> gateway to provide Internet access from LTE to a private Wi-Fi
> network. Our problem is that the bandwidth on LTE and Wi-Fi links is
> higher than what the system is able to handle thus it reaches 100% of
> CPU load when we perform a simple speed test from a device connected
> to our Wi-Fi Hotspot.
>
> Therefore, we want to limit the bandwidth to avoid system gets
> saturated is such use-case. To do so, we thought to use the QDISC-TBF
> on the Wi-Fi interface. For instance, to have 10Mbps:
>
>      tc qdisc add dev wlan0 root tbf rate 10mbit burst 12500b latency 50ms
>
> It worked correctly and maximum rate was limited to 10Mbps. However,
> we noticed that the CPU load added by the TBF was not negligible for
> our system. In fact, we compared the CPU load when limitation was done
> by TBF and by the device on the private network, e.g. wget tool with
> parameter "--limit-rate". As result, we found that the CPU load when
> using TBF was 10-15% higher.
>
> Then, we thought that using traffic shaping in egress, packets need to
> be un-natted (which takes CPU) and pass through the system to then get
> dropped. Therefore, we tried to use an incoming policer instead of
> egress traffic shaping as following:
>
>      tc qdisc add dev eth0 ingress handle ffff:
>      tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0
> police rate 10mbit burst 1m drop
>
> Unfortunately, as per egress traffic shaping, we still obtained a high
> CPU cost because of rate limiting. However, also in this case, we are
> not sure we chose the most efficient option in terms of CPU cost to
> police in ingress.
>
> Given that, we were wondering if we are doing wrong by choosing TBF?
> Or maybe we are using wrong parameters? We found everywhere that TBF
> is the simplest way to limit the rate thus we suppose it is also the
> most efficient QDISC. Is our supposition correct? Or there is another
> way to limit rate well-known by its low CPU consumption? Any
> suggestion is welcome, just taking into account we are using
> libnl-3.2.28 and linux-kernel 3.18. In case, we could change libnl but
> not kernel version, at most some specific patches.
>
> Thanks in advance for the support!
>
> Jose Blanquicet
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: Type: text/html, Size: 3286 bytes --]

Re: [Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Jonathan Morton]

> On 26 May, 2020, at 12:47 pm, Jose Blanquicet <blanquicet@gmail.com> wrote:
> 
> We have an embedded system with limited CPU resources that acts as
> gateway to provide Internet access from LTE to a private Wi-Fi
> network. Our problem is that the bandwidth on LTE and Wi-Fi links is
> higher than what the system is able to handle thus it reaches 100% of
> CPU load when we perform a simple speed test from a device connected
> to our Wi-Fi Hotspot.
> 
> Therefore, we want to limit the bandwidth to avoid system gets
> saturated is such use-case. To do so, we thought to use the QDISC-TBF
> on the Wi-Fi interface. For instance, to have 10Mbps:
> 
>    tc qdisc add dev wlan0 root tbf rate 10mbit burst 12500b latency 50ms
> 
> It worked correctly and maximum rate was limited to 10Mbps. However,
> we noticed that the CPU load added by the TBF was not negligible for
> our system.

Just how limited is the CPU on this device?  I have successfully shaped at several tens of Mbps on a Pentium-MMX, where the limiting factor may have been the PCI bus rather than the CPU itself.

Assuming your CPU is of that order of capability, I would suggest installing Cake using the out-of-tree build process, and the latest stable version of the iproute2 tools to configure it.  Start with:

	git clone https://github.com/dtaht/sch_cake.git

This provides a more efficient and more effective shaper than TBF, and a more effective AQM than a policer, and good flow-isolation properties, all in a single bundle that will be more efficient than running two separate components.

Once installed, the following should set it up nicely for you:

	tc qdisc replace dev wlan0 root cake bandwidth 10Mbit besteffort flows ack-filter

Cake is considered quite a heavyweight solution, but very effective.  If it doesn't work well for this particular use case, it may be feasible to backport some more recent work which takes a simpler approach, though along similar lines.

 - Jonathan Morton

Re: [Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Jose Blanquicet]

Hi Jonathan,

Thanks for the response.

On Tue, May 26, 2020 at 2:50 PM Jonathan Morton <chromatix99@gmail.com> wrote:
> > On 26 May, 2020, at 12:47 pm, Jose Blanquicet <blanquicet@gmail.com> wrote:
> >
> > We have an embedded system with limited CPU resources that acts as
> > gateway to provide Internet access from LTE to a private Wi-Fi
> > network. Our problem is that the bandwidth on LTE and Wi-Fi links is
> > higher than what the system is able to handle thus it reaches 100% of
> > CPU load when we perform a simple speed test from a device connected
> > to our Wi-Fi Hotspot.
> >
> > Therefore, we want to limit the bandwidth to avoid system gets
> > saturated is such use-case. To do so, we thought to use the QDISC-TBF
> > on the Wi-Fi interface. For instance, to have 10Mbps:
> >
> >    tc qdisc add dev wlan0 root tbf rate 10mbit burst 12500b latency 50ms
> >
> > It worked correctly and maximum rate was limited to 10Mbps. However,
> > we noticed that the CPU load added by the TBF was not negligible for
> > our system.
>
> Just how limited is the CPU on this device?  I have successfully shaped at several tens of Mbps on a Pentium-MMX, where the limiting factor may have been the PCI bus rather than the CPU itself.

We have just a percentage of an ARM Cortex A7 (1.2GHz) because the
rest is reserved for modem. We are now trying to optimize all the
applications in the system but LTE<->WIFI data transfer is indeed the
use-case that puts our system in crisis.

> Assuming your CPU is of that order of capability, I would suggest installing Cake using the out-of-tree build process, and the latest stable version of the iproute2 tools to configure it.  Start with:
>
>         git clone https://github.com/dtaht/sch_cake.git
>
> This provides a more efficient and more effective shaper than TBF, and a more effective AQM than a policer, and good flow-isolation properties, all in a single bundle that will be more efficient than running two separate components.
>
> Once installed, the following should set it up nicely for you:
>
>         tc qdisc replace dev wlan0 root cake bandwidth 10Mbit besteffort flows ack-filter
>
> Cake is considered quite a heavyweight solution, but very effective.  If it doesn't work well for this particular use case, it may be feasible to backport some more recent work which takes a simpler approach, though along similar lines.

Interesting, we will have a look at this and do some test to see how
it behaves in our "particular" environment.

Thanks,
Jose

Re: [Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Jose Blanquicet]

Hi Yutaka,

Thanks for your response.

On Tue, May 26, 2020 at 2:04 PM Y via Bloat <bloat@lists.bufferbloat.net> wrote:
> I heard that IBF costs cpu load.

Sorry but what do you mean with "IBF"?

> How about shaping only egress with TBF?

It was our very first option but limiting traffic with this is costing
too much CPU in our system with limited CPU resources (Check the
results in my first email). So we found that we are sacrificing too
much rate to get few points of free CPU and that's why I am here
asking for suggestions.

Thanks,
Jose

Re: [Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Y]

Hi,Jose


I mean ifb sorry :P

But cake is awesome,try it.

On 2020/05/26 23:50, Jose Blanquicet wrote:
> Hi Yutaka,
>
> Thanks for your response.
>
> On Tue, May 26, 2020 at 2:04 PM Y via Bloat <bloat@lists.bufferbloat.net> wrote:
>> I heard that IBF costs cpu load.
> Sorry but what do you mean with "IBF"?
>
>> How about shaping only egress with TBF?
> It was our very first option but limiting traffic with this is costing
> too much CPU in our system with limited CPU resources (Check the
> results in my first email). So we found that we are sacrificing too
> much rate to get few points of free CPU and that's why I am here
> asking for suggestions.
>
> Thanks,
> Jose