[Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF

[Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF

[Pete Heist]

I've always wanted a way to customize Cake's host and flow isolation in
a way that would be usable e.g. for small ISPs, and this is what I came
up with:

https://github.com/heistp/cake-custom-isolation

ipsets are used to set the skb priority or mark, then tc-flow or a
simple eBPF classifier is used in a child filter of cake to get the
major and minor class IDs set, which override the host and flow hashes.

To show it in action, the cakeiso.sh script sets up a netns environment
and runs competition between two "subscribers" and three flows, two TCP
flows and one unresponsive UDP flow. Several configurations are run to
show what is and isn't possible.

If anyone knows of a simpler way than eBPF to get both the major and
minor class ID set from ipsets, I'd like to hear it, but the included
classifiers are at least very simple one-liners...

Pete

Re: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

> I've always wanted a way to customize Cake's host and flow isolation in
> a way that would be usable e.g. for small ISPs, and this is what I came
> up with:
>
> https://github.com/heistp/cake-custom-isolation
>
> ipsets are used to set the skb priority or mark, then tc-flow or a
> simple eBPF classifier is used in a child filter of cake to get the
> major and minor class IDs set, which override the host and flow hashes.

Very cool! Awesome to see the customisation options being used for
something neat like this! :)

> To show it in action, the cakeiso.sh script sets up a netns environment
> and runs competition between two "subscribers" and three flows, two TCP
> flows and one unresponsive UDP flow. Several configurations are run to
> show what is and isn't possible.
>
> If anyone knows of a simpler way than eBPF to get both the major and
> minor class ID set from ipsets, I'd like to hear it, but the included
> classifiers are at least very simple one-liners...

Well, you could go the other way? Instead of ipset, just do the
classification in eBPF and use a BPF map to store the IP addresses.
There's even an LPM map type, so you can use arbitrary prefix lengths
for each class (or not, and just use a hashmap)...

-Toke

Re: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF

[Pete Heist]

On Sun, 2021-06-06 at 21:59 +0200, Toke Høiland-Jørgensen wrote:
> Pete Heist <pete@heistp.net> writes:
> 
> > I've always wanted a way to customize Cake's host and flow isolation
> > in
> > a way that would be usable e.g. for small ISPs, and this is what I
> > came
> > up with:
> > 
> > https://github.com/heistp/cake-custom-isolation
> > 
> > ipsets are used to set the skb priority or mark, then tc-flow or a
> > simple eBPF classifier is used in a child filter of cake to get the
> > major and minor class IDs set, which override the host and flow
> > hashes.
> 
> Very cool! Awesome to see the customisation options being used for
> something neat like this! :)
> 
> > To show it in action, the cakeiso.sh script sets up a netns
> > environment
> > and runs competition between two "subscribers" and three flows, two
> > TCP
> > flows and one unresponsive UDP flow. Several configurations are run
> > to
> > show what is and isn't possible.
> > 
> > If anyone knows of a simpler way than eBPF to get both the major and
> > minor class ID set from ipsets, I'd like to hear it, but the included
> > classifiers are at least very simple one-liners...
> 
> Well, you could go the other way? Instead of ipset, just do the
> classification in eBPF and use a BPF map to store the IP addresses.
> There's even an LPM map type, so you can use arbitrary prefix lengths
> for each class (or not, and just use a hashmap)...

True that, I started something like that at some point:

https://github.com/heistp/tc-users/

but I think I got a little overzealous with it. I'm not sure if/when
I'll get back to that, but the ipset solution seems to be "good enough"
for what I (and my ISP) needs. I'm glad you slipped the tc filter
overrides in before Cake went out the door. :)

This doesn't do away with the possible need for a full-blown ISP qdisc
one day, with configurable subscriber tiers, handling of higher loads,
etc, but at least it's something for the little guys.

Pete

> -Toke

Re: [Cake] customizing Cake's isolation with ipsets, tc-flow and eBPF

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

> On Sun, 2021-06-06 at 21:59 +0200, Toke Høiland-Jørgensen wrote:
>> Pete Heist <pete@heistp.net> writes:
>> 
>> > I've always wanted a way to customize Cake's host and flow isolation
>> > in
>> > a way that would be usable e.g. for small ISPs, and this is what I
>> > came
>> > up with:
>> > 
>> > https://github.com/heistp/cake-custom-isolation
>> > 
>> > ipsets are used to set the skb priority or mark, then tc-flow or a
>> > simple eBPF classifier is used in a child filter of cake to get the
>> > major and minor class IDs set, which override the host and flow
>> > hashes.
>> 
>> Very cool! Awesome to see the customisation options being used for
>> something neat like this! :)
>> 
>> > To show it in action, the cakeiso.sh script sets up a netns
>> > environment
>> > and runs competition between two "subscribers" and three flows, two
>> > TCP
>> > flows and one unresponsive UDP flow. Several configurations are run
>> > to
>> > show what is and isn't possible.
>> > 
>> > If anyone knows of a simpler way than eBPF to get both the major and
>> > minor class ID set from ipsets, I'd like to hear it, but the included
>> > classifiers are at least very simple one-liners...
>> 
>> Well, you could go the other way? Instead of ipset, just do the
>> classification in eBPF and use a BPF map to store the IP addresses.
>> There's even an LPM map type, so you can use arbitrary prefix lengths
>> for each class (or not, and just use a hashmap)...
>
> True that, I started something like that at some point:
>
> https://github.com/heistp/tc-users/
>
> but I think I got a little overzealous with it. I'm not sure if/when
> I'll get back to that, but the ipset solution seems to be "good enough"
> for what I (and my ISP) needs. I'm glad you slipped the tc filter
> overrides in before Cake went out the door. :)

Yes! I would love to take credit for the idea, but this came from
the netdev review. I agree though, great that it ended up in there!

> This doesn't do away with the possible need for a full-blown ISP qdisc
> one day, with configurable subscriber tiers, handling of higher loads,
> etc, but at least it's something for the little guys.

Yeah; I have something in the pipeline that will hopefully end up being
useable for that, but may take a little while to get there :)

-Toke