[Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 29, 2018, at 9:14 PM, Toke Høiland-Jørgensen <toke at toke.dk> wrote:
>> 
>> Caveats that I know of:
>> - Limited to 1024 members
>> - No fairness between flows
> 
> You could assign more than one queue per customer and hash traffic
> between them in BPF…

True. There will always be that limit of 1024 (in my case I’ll need 800).

>> - Non-member traffic would have to be dealt with somehow, maybe put in
>> its own queue or split among multiple queues, otherwise there can be
>> hash collisions with member queues
> 
> Yeah, an "overflow queue" is definitely needed in this kind of
> deployment :)

Yep, I’d like to hash the non-member flows across the remaining queues.

> I actually wrote an eBPF classifier a few months back, that can lookup
> subnets in a BPF map and map them into different classes:
> https://github.com/tohojo/tc-classifier <https://github.com/tohojo/tc-classifier>

Nice! That along with Dave’s ack classifier will help me write one for MAC addresses. I got as far as “my first no-panic bpf”, but for starters I wasn’t sure of the right way to set classid, so I see TC_H_MAKE. The documentation one finds on BPF varies a lot in correctness, so I messed around a while.

I think regardless of whether ISP cake is a new qdisc or changes to the current one, it would be good to provide a common tool like this for mapping both MACs and IP subnets. Maybe I can just expand tc-classifier a bit for my needs and try to think of others also? Here’s how it could work:

Userspace tool:
- accepts as input from stdin or file, space or comma separated mappings of one of (MAC address, IPv4 subnet or IPv6 subnet) to both classid (flow) and priority (tin), so three fields total
- accepts as an optional argument tin to place unclassified traffic in (defaults to 0)
- returns an error if no queues available for unclassified traffic
- puts mappings into up to three global BPF maps (for MAC, IPv4 and IPv6)
- puts unclassified traffic tin, if non-zero, into a global
- should lock globals here so updates can be made without removing / re-adding qdisc

BPF filter:
- tries to classify first using MAC address map, then IPv4 or IPv6 maps for IP traffic
- spreads any unclassified traffic in unclassified traffic tin across remaining classids from max(classid)+1 to 1023

Lastly, although it's natural to use classid for flow (subscriber) and priority for tin, we have a hard maximum of 2^16 subscribers in a given tin (minor classid is 16 bits). It doesn’t matter now because we only have 1024 flows per tin, but for ISP cake, is a limit of ~2^16 subscribers in one tin enough? Otherwise we’d have to change the way we specify this.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20180730/79e5bb9e/attachment-0001.html>