[Bloat] CPU consumption using TC-TBF and TC-POLICE to limit rate

[Jose Blanquicet]

Hi everyone,

We have an embedded system with limited CPU resources that acts as
gateway to provide Internet access from LTE to a private Wi-Fi
network. Our problem is that the bandwidth on LTE and Wi-Fi links is
higher than what the system is able to handle thus it reaches 100% of
CPU load when we perform a simple speed test from a device connected
to our Wi-Fi Hotspot.

Therefore, we want to limit the bandwidth to avoid system gets
saturated is such use-case. To do so, we thought to use the QDISC-TBF
on the Wi-Fi interface. For instance, to have 10Mbps:

    tc qdisc add dev wlan0 root tbf rate 10mbit burst 12500b latency 50ms

It worked correctly and maximum rate was limited to 10Mbps. However,
we noticed that the CPU load added by the TBF was not negligible for
our system. In fact, we compared the CPU load when limitation was done
by TBF and by the device on the private network, e.g. wget tool with
parameter "--limit-rate". As result, we found that the CPU load when
using TBF was 10-15% higher.

Then, we thought that using traffic shaping in egress, packets need to
be un-natted (which takes CPU) and pass through the system to then get
dropped. Therefore, we tried to use an incoming policer instead of
egress traffic shaping as following:

    tc qdisc add dev eth0 ingress handle ffff:
    tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0
police rate 10mbit burst 1m drop

Unfortunately, as per egress traffic shaping, we still obtained a high
CPU cost because of rate limiting. However, also in this case, we are
not sure we chose the most efficient option in terms of CPU cost to
police in ingress.

Given that, we were wondering if we are doing wrong by choosing TBF?
Or maybe we are using wrong parameters? We found everywhere that TBF
is the simplest way to limit the rate thus we suppose it is also the
most efficient QDISC. Is our supposition correct? Or there is another
way to limit rate well-known by its low CPU consumption? Any
suggestion is welcome, just taking into account we are using
libnl-3.2.28 and linux-kernel 3.18. In case, we could change libnl but
not kernel version, at most some specific patches.

Thanks in advance for the support!

Jose Blanquicet