[Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 26, 2018, at 11:38 PM, Jonathan Morton <chromatix99 at gmail.com> wrote:
> 
> It would also be valuable to have a firmer handle on the actual requirements in the field.  For example, if it is feasible to focus only on current Linux kernels, then a lot of backwards compatibility cruft can be excised when importing existing code from Cake.

One more data point, for FreeNet Liberec (WISP co-op)...


There are some older backhaul routers still with 2.6.26.8(!) although those are being phased out so don’t count them. More current ones use 3.16.7 and there’s some discussion but I’m not sure what/when the upgrade plan is. I think the Internet router uses a more modern Debian 9 which is likely to have a 4.9 series, but I don’t have access to it.

FreeNet might be unusual in that administration is distributed. Volunteer members administer backhaul routers which carry some number of customers, average a few dozen or so. These do routing/firewalling/monitoring and QoS, when they’re a bottleneck. They use either an ancient esfq (has per-IP fairness though) on older kernels and sfq when it’s not available, which is now more common. There are arguments (heated ones, I’ve heard) about centralizing administrative functions, including QoS, at or near the Internet gateway, but I think this would have to depend on over-provisioning the backhaul. I’ve volunteered to modernize the QoS when I can. If ISP flavored Cake doesn’t happen, it will end up being HTB+sfq|fq_codel|cake depending on what the kernel supports.

>  If all users will have the same link-layer technology (with the same overhead parameters), then these can be set globally - or if not, they can be set per-tier.

Afaik almost all members use WiFi, possibly some Ethernet. There are no tiers and no per-member rate limits.

>  Is the Diffserv support from Cake likely to be useful, and if so how flexible should the configuration be?  

Currently the root qdisc on backhaul routers is a prio with "bands 3 priomap 2 2 2 2 2 2 0 0 2 2 2 2 2 2 2 2”. I don’t think we’d want DSCP for anything, with the possible exception of voip, and even then there might just be a special IP/port rule for the asterisk server.

> And are there only a few discrete settings for bandwidth per user, or do we have to be more flexible to handle a BRAS environment?  

Mainly in this case only fairness between members is needed. There’s a db mapping members to their MAC addresses (usually one to one but not always).

> Is it also necessary to account per-user traffic accurately, or will an external tool be used for that?

It would be better than what is done now (counting per-IP, which when a customer has multiple MACs makes it harder to interpret).

> Many other low-level considerations can be adjusted on the fly during the conversion, so they are not in themselves a problem.  In this category are questions like "how many simultaneous users and flows can be supported”.

In this case something that works to a few thousand users is likely to be fine (currently, 800 members).


Lastly, do you think a better shaper for point-to-point WiFi is within the scope of this project? This might be more needed for WISPs, but here’s why:

- If we still have bottlenecks in the backhaul, we’ll need to keep doing QoS there, and for FreeNet that means soft rate limiting, because the WiFi devices have to keep running airOS for its management tools.
- If you only rate limit on egress, you lose at least half your available throughput.
- You can run egress and ingress through a common IFB, but in my testing, TCP RTT (rrul_be with —socket-stats) gets higher.
- I find it works better to use HTB at the root and have HTB+cake as leaf queues, then TCP RTT is roughly cut in half.
- But it would be even better if the shaper understood an approximation of airtime, because aggregate throughput changes based on the balance of up/down traffic in the case where up/down rates are stable but asymmetric, which FreeNet sometimes has.
- And, I’d rather not have to use HTB, and be able to use the better deficit mode shaper in Cake.

I know that WiFi has many complexities where soft rate limiting can’t ever be perfect without knowledge from the driver, but I still think it could be useful to have something that does a better approximation than what we have today. The question is, is that something that could be part of this project, or not… :)