[Cake] flow isolation for ISPs

Fri Apr 7 07:13:15 EDT 2017

Hi Peter,

> On Apr 7, 2017, at 11:37, Pete Heist <peteheist at gmail.com> wrote:
> 
>> 
>> On Apr 7, 2017, at 10:28 AM, Jonathan Morton <ciately 600 users IIRChromatix99 at gmail.com> wrote:
>>> 
>>> On 7 Apr, 2017, at 11:13, Pete Heist <peteheist at gmail.com> wrote:
>>> 
>>>> On Apr 6, 2017, at 11:26 AM, Pete Heist <peteheist at gmail.com> wrote:
>>>> 
>>>>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99 at gmail.com> wrote:
>>>>> 
>>>>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist at gmail.com> wrote:
>>>>>> 
>>>>>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>>>>> 
>>>>> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.
>>> 
>>> One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?
>>> 
>>> I know it’s additional functionality, and another keyword, but it could get you out of the business of the myriad of ways people might want to do flow isolation, and you’d still have a catch-all answer for such cases.
>>> 
>>> There could be a keyword ‘hash-mark’, let’s say, which first includes the mark in the hash, then does on to deal with any other flow isolation keywords as usual. So for example if I have ‘hash-mark’ and ‘dual-srchost’, the hash is first on the mark, then by source host, then by flow. I could set the mark to be the member number with iptables.
>> 
>> That isn’t really how hashing works; there is no “first, second, third” structure, just an accumulation of entropy which is all mashed together.  In order to run the triple-isolation algorithm at all, I have to take separate hashes of the relevant host addresses, alongside the general 5-tuple hash.
>> 
>> However, it would be possible to use the “mark” directly as one of the host identifiers which triple-isolate operates on to provide that layer of fairness.  That’s probably what you meant.
>> 
>> Since this wouldn’t unduly complicate the configuration interface, it could be a feasible way of adding this functionality for modest installations, up to a strict maximum of 1024 subscribers (and a recommended maximum somewhat below that).
> 
> Ok, I’m still getting familiar with how triple-isolate is implemented. For example, I was surprised in my test setup that no fairness is enforced when four client IPs connect to a single server IP, but I understand from this discussion (https://github.com/dtaht/sch_cake/issues/46) that that is actually what is expected. We would probably use dual-srchost and dual-dsthost in the backhaul, which seems to work very well, and in the backhaul we have the information to specify that in both directions. (Also, there is no NAT to deal with at this level.)
> 
> Just to see if I understand the marking proposal, here's the behavior I would expect: if there are two TCP flows (on egress) with mark 1 and one with mark 2, that together saturate the link, the measured rate of the two flows with mark 1 will add up to the rate of the single flow with mark 2. Is that right? And would you still add a keyword to specify that the mark should be used at all?
> 
> I’m not sure where the 1024 limit comes from, but it would probably be fine in our case as of now, with 800 members. Even in the future, I don’t think occasional collisions would be a big problem, and I think there are things we could do to minimize them.

Seeing your 800 members I remember a discussion over at the lede forum, https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/27?u=moeller0 where orangetek, used cake on a wired backhaul for approximately 600 end users. He reported for number of concurrent flows: “As far as i can tell, around 25k-30k during busy hours.” 
He also increased the number of CAKE_BINs in the code to 64k. So depending on your user’s 1024 might be a bit tight, given that you still ideally want flows to not share bins if possible (sure cake is great in avoiding sharing unless impossible, but with enough flows you might want/need to simply hard code your cake instances for higher limits).

Best Regards

> 
>>> It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.
>> 
>> It has been suggested, in the context of using the “mark” for Diffserv purposes, that Linux’ conntrack facility could preserve the mark between directions of flow.  Cake can already query conntrack for NAT awareness.
> 
> That would be nice for the future, but for now I guess this wouldn’t work on ingress. It shouldn’t be much of a problem in the backhaul though, because we’re the ones sending the downstream traffic, and we can set the marks on that.
> 
> Overall, I think this could be a nice feature. Let me know if I can help in some way and thank you for your feedback. :)
> 
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake