Re: [Cake] flow isolation for ISPs

Re: [Cake] flow isolation for ISPs

[Konstantin Shalygin]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

Hello.

If you don't know how to shape gigabits via HTB and u32 flow filters, 
maybe for you will be useful to sc <https://bitbucket.org/sky/sc> - this 
is tool for ISP, do all work for you.
Hope this helps.

> I get the part about matching with tc-filter and the u32 selector (as intuitive as that is :), but am not sure of the action the filter needs to take. However, I do see the example towards the bottom of the tc-u32 man page where a hash table is created and filters move packets into the right buckets. Perhaps it will be eventually decipherable from this… :)



[-- Attachment #2: Type: text/html, Size: 898 bytes --]

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]


>> On Apr 7, 2017, at 10:28 AM, Jonathan Morton <chromatix99@gmail.com <mailto:chromatix99@gmail.com>> wrote:
>>> 
>>> On 7 Apr, 2017, at 11:13, Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> wrote:
>>> 
>>> One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?
>> 
>> However, it would be possible to use the “mark” directly as one of the host identifiers which triple-isolate operates on to provide that layer of fairness.  That’s probably what you meant.

Ok, I’m starting to understand in cake_hash how triple-isolate is done. I had some misconceptions, and this is the first time I’ve seen a set-associative hash. In case there's time, I’ll see if I can get the mark idea working. It's probably something useful for only a minority of users...

[-- Attachment #2: Type: text/html, Size: 2522 bytes --]

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 3230 bytes --]


> On Apr 7, 2017, at 1:13 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Peter,
> 
>> On Apr 7, 2017, at 11:37, Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> wrote:
>> 
>> Ok, I’m still getting familiar with how triple-isolate is implemented. For example, I was surprised in my test setup that no fairness is enforced when four client IPs connect to a single server IP, but I understand from this discussion (https://github.com/dtaht/sch_cake/issues/46) that that is actually what is expected. We would probably use dual-srchost and dual-dsthost in the backhaul, which seems to work very well, and in the backhaul we have the information to specify that in both directions. (Also, there is no NAT to deal with at this level.)

I didn’t write that very well before, so just to clarify, there’s nothing more we need to specify for dual-srchost and dual-dsthost to work, just that we control both directions of the flow so can use dual-srchost on upstream egress and dual-dsthost on downstream egress.

>> Just to see if I understand the marking proposal, here's the behavior I would expect: if there are two TCP flows (on egress) with mark 1 and one with mark 2, that together saturate the link, the measured rate of the two flows with mark 1 will add up to the rate of the single flow with mark 2. Is that right? And would you still add a keyword to specify that the mark should be used at all?
>> 
>> I’m not sure where the 1024 limit comes from, but it would probably be fine in our case as of now, with 800 members. Even in the future, I don’t think occasional collisions would be a big problem, and I think there are things we could do to minimize them.
> 
> Seeing your 800 members I remember a discussion over at the lede forum, https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/27?u=moeller0 <https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/27?u=moeller0> where orangetek, used cake on a wired backhaul for approximately 600 end users. He reported for number of concurrent flows: “As far as i can tell, around 25k-30k during busy hours.” 
> He also increased the number of CAKE_BINs in the code to 64k. So depending on your user’s 1024 might be a bit tight, given that you still ideally want flows to not share bins if possible (sure cake is great in avoiding sharing unless impossible, but with enough flows you might want/need to simply hard code your cake instances for higher limits).

I see, so the 1024 limit probably comes from the CAKE_QUEUES define. :)

So far, we’re not looking to use Cake on the main Internet router. I’m just not sure yet if it would be appropriate for a gigabit uplink (that also doesn’t reach saturation, as far as can be discerned from mrtg plots). We’re taking things step by step, and looking at some of the backhaul routers first, where there can sometimes be congestion.

As for an average peak number of concurrent flows on the Internet router I could find out from the admin. Total throughput for the gigabit Internet uplink is on a public page: https://www.lbcfree.net/mrtg/10.101.254.194_24.html, but not flow counts.

Thanks for the tip...

[-- Attachment #2: Type: text/html, Size: 7951 bytes --]

Re: [Cake] flow isolation for ISPs

[Sebastian Moeller]

Hi Peter,



> On Apr 7, 2017, at 11:37, Pete Heist <peteheist@gmail.com> wrote:
> 
>> 
>> On Apr 7, 2017, at 10:28 AM, Jonathan Morton <ciately 600 users IIRChromatix99@gmail.com> wrote:
>>> 
>>> On 7 Apr, 2017, at 11:13, Pete Heist <peteheist@gmail.com> wrote:
>>> 
>>>> On Apr 6, 2017, at 11:26 AM, Pete Heist <peteheist@gmail.com> wrote:
>>>> 
>>>>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>>>> 
>>>>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>>>>>> 
>>>>>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>>>>> 
>>>>> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.
>>> 
>>> One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?
>>> 
>>> I know it’s additional functionality, and another keyword, but it could get you out of the business of the myriad of ways people might want to do flow isolation, and you’d still have a catch-all answer for such cases.
>>> 
>>> There could be a keyword ‘hash-mark’, let’s say, which first includes the mark in the hash, then does on to deal with any other flow isolation keywords as usual. So for example if I have ‘hash-mark’ and ‘dual-srchost’, the hash is first on the mark, then by source host, then by flow. I could set the mark to be the member number with iptables.
>> 
>> That isn’t really how hashing works; there is no “first, second, third” structure, just an accumulation of entropy which is all mashed together.  In order to run the triple-isolation algorithm at all, I have to take separate hashes of the relevant host addresses, alongside the general 5-tuple hash.
>> 
>> However, it would be possible to use the “mark” directly as one of the host identifiers which triple-isolate operates on to provide that layer of fairness.  That’s probably what you meant.
>> 
>> Since this wouldn’t unduly complicate the configuration interface, it could be a feasible way of adding this functionality for modest installations, up to a strict maximum of 1024 subscribers (and a recommended maximum somewhat below that).
> 
> Ok, I’m still getting familiar with how triple-isolate is implemented. For example, I was surprised in my test setup that no fairness is enforced when four client IPs connect to a single server IP, but I understand from this discussion (https://github.com/dtaht/sch_cake/issues/46) that that is actually what is expected. We would probably use dual-srchost and dual-dsthost in the backhaul, which seems to work very well, and in the backhaul we have the information to specify that in both directions. (Also, there is no NAT to deal with at this level.)
> 
> Just to see if I understand the marking proposal, here's the behavior I would expect: if there are two TCP flows (on egress) with mark 1 and one with mark 2, that together saturate the link, the measured rate of the two flows with mark 1 will add up to the rate of the single flow with mark 2. Is that right? And would you still add a keyword to specify that the mark should be used at all?
> 
> I’m not sure where the 1024 limit comes from, but it would probably be fine in our case as of now, with 800 members. Even in the future, I don’t think occasional collisions would be a big problem, and I think there are things we could do to minimize them.


Seeing your 800 members I remember a discussion over at the lede forum, https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appliance/1861/27?u=moeller0 where orangetek, used cake on a wired backhaul for approximately 600 end users. He reported for number of concurrent flows: “As far as i can tell, around 25k-30k during busy hours.” 
He also increased the number of CAKE_BINs in the code to 64k. So depending on your user’s 1024 might be a bit tight, given that you still ideally want flows to not share bins if possible (sure cake is great in avoiding sharing unless impossible, but with enough flows you might want/need to simply hard code your cake instances for higher limits).

Best Regards


> 
>>> It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.
>> 
>> It has been suggested, in the context of using the “mark” for Diffserv purposes, that Linux’ conntrack facility could preserve the mark between directions of flow.  Cake can already query conntrack for NAT awareness.
> 
> That would be nice for the future, but for now I guess this wouldn’t work on ingress. It shouldn’t be much of a problem in the backhaul though, because we’re the ones sending the downstream traffic, and we can set the marks on that.
> 
> Overall, I think this could be a nice feature. Let me know if I can help in some way and thank you for your feedback. :)
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] flow isolation for ISPs

[John Sager]

On 07/04/17 09:28, Jonathan Morton wrote:

>> It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.
> 
> It has been suggested, in the context of using the “mark” for Diffserv purposes, that Linux’ conntrack facility could preserve the mark between directions of flow.  Cake can already query conntrack for NAT awareness.
> 

That's how I use marks, though for guaranteeing bandwidth to classes rather
than fairness per se - it prevents big file downloads killing video
streaming. See this fraction from my setup script:

# add ingress qdisc
$TC qdisc add dev $IFACE handle ffff: ingress

# set pkt mark from connmark & send all traffic to ifb interface
$TC filter add dev $IFACE parent ffff: protocol ip u32 match u32 0 0 \
 action connmark action mirred egress redirect dev $IFB

Marks are set up by iptables rules on egress and copied to conntrack.

The qdiscs on egress and ifb are htb+fq_codel.

John

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 4440 bytes --]


> On Apr 7, 2017, at 10:28 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>> On 7 Apr, 2017, at 11:13, Pete Heist <peteheist@gmail.com> wrote:
>> 
>>> On Apr 6, 2017, at 11:26 AM, Pete Heist <peteheist@gmail.com> wrote:
>>> 
>>>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>>> 
>>>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>>>>> 
>>>>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>>>> 
>>>> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.
>> 
>> One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?
>> 
>> I know it’s additional functionality, and another keyword, but it could get you out of the business of the myriad of ways people might want to do flow isolation, and you’d still have a catch-all answer for such cases.
>> 
>> There could be a keyword ‘hash-mark’, let’s say, which first includes the mark in the hash, then does on to deal with any other flow isolation keywords as usual. So for example if I have ‘hash-mark’ and ‘dual-srchost’, the hash is first on the mark, then by source host, then by flow. I could set the mark to be the member number with iptables.
> 
> That isn’t really how hashing works; there is no “first, second, third” structure, just an accumulation of entropy which is all mashed together.  In order to run the triple-isolation algorithm at all, I have to take separate hashes of the relevant host addresses, alongside the general 5-tuple hash.
> 
> However, it would be possible to use the “mark” directly as one of the host identifiers which triple-isolate operates on to provide that layer of fairness.  That’s probably what you meant.
> 
> Since this wouldn’t unduly complicate the configuration interface, it could be a feasible way of adding this functionality for modest installations, up to a strict maximum of 1024 subscribers (and a recommended maximum somewhat below that).

Ok, I’m still getting familiar with how triple-isolate is implemented. For example, I was surprised in my test setup that no fairness is enforced when four client IPs connect to a single server IP, but I understand from this discussion (https://github.com/dtaht/sch_cake/issues/46) that that is actually what is expected. We would probably use dual-srchost and dual-dsthost in the backhaul, which seems to work very well, and in the backhaul we have the information to specify that in both directions. (Also, there is no NAT to deal with at this level.)

Just to see if I understand the marking proposal, here's the behavior I would expect: if there are two TCP flows (on egress) with mark 1 and one with mark 2, that together saturate the link, the measured rate of the two flows with mark 1 will add up to the rate of the single flow with mark 2. Is that right? And would you still add a keyword to specify that the mark should be used at all?

I’m not sure where the 1024 limit comes from, but it would probably be fine in our case as of now, with 800 members. Even in the future, I don’t think occasional collisions would be a big problem, and I think there are things we could do to minimize them.

>> It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.
> 
> It has been suggested, in the context of using the “mark” for Diffserv purposes, that Linux’ conntrack facility could preserve the mark between directions of flow.  Cake can already query conntrack for NAT awareness.

That would be nice for the future, but for now I guess this wouldn’t work on ingress. It shouldn’t be much of a problem in the backhaul though, because we’re the ones sending the downstream traffic, and we can set the marks on that.

Overall, I think this could be a nice feature. Let me know if I can help in some way and thank you for your feedback. :)


[-- Attachment #2: Type: text/html, Size: 10342 bytes --]

Re: [Cake] flow isolation for ISPs

[Jonathan Morton]

> On 7 Apr, 2017, at 11:13, Pete Heist <peteheist@gmail.com> wrote:
> 
>> On Apr 6, 2017, at 11:26 AM, Pete Heist <peteheist@gmail.com> wrote:
>> 
>>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> 
>>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>>>> 
>>>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>>> 
>>> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.
> 
> One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?
> 
> I know it’s additional functionality, and another keyword, but it could get you out of the business of the myriad of ways people might want to do flow isolation, and you’d still have a catch-all answer for such cases.
> 
> There could be a keyword ‘hash-mark’, let’s say, which first includes the mark in the hash, then does on to deal with any other flow isolation keywords as usual. So for example if I have ‘hash-mark’ and ‘dual-srchost’, the hash is first on the mark, then by source host, then by flow. I could set the mark to be the member number with iptables.

That isn’t really how hashing works; there is no “first, second, third” structure, just an accumulation of entropy which is all mashed together.  In order to run the triple-isolation algorithm at all, I have to take separate hashes of the relevant host addresses, alongside the general 5-tuple hash.

However, it would be possible to use the “mark” directly as one of the host identifiers which triple-isolate operates on to provide that layer of fairness.  That’s probably what you meant.

Since this wouldn’t unduly complicate the configuration interface, it could be a feasible way of adding this functionality for modest installations, up to a strict maximum of 1024 subscribers (and a recommended maximum somewhat below that).

> It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.

It has been suggested, in the context of using the “mark” for Diffserv purposes, that Linux’ conntrack facility could preserve the mark between directions of flow.  Cake can already query conntrack for NAT awareness.

 - Jonathan Morton

Re: [Cake] flow isolation for ISPs

[Pete Heist]

> On Apr 6, 2017, at 11:26 AM, Pete Heist <peteheist@gmail.com> wrote:
> 
>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>>> 
>>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>> 
>> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.

One more thought, would it be possible for Cake to optionally include the packet’s mark in the hash?

I know it’s additional functionality, and another keyword, but it could get you out of the business of the myriad of ways people might want to do flow isolation, and you’d still have a catch-all answer for such cases.

There could be a keyword ‘hash-mark’, let’s say, which first includes the mark in the hash, then does on to deal with any other flow isolation keywords as usual. So for example if I have ‘hash-mark’ and ‘dual-srchost’, the hash is first on the mark, then by source host, then by flow. I could set the mark to be the member number with iptables.

It looks like the mark could be obtained from the ‘mark' field of the sk_buff struct, but I don’t know the validity of the field in various cases. For example, I don’t think I can set the mark on ingress before it reaches a qdisc on an IFB device.

Re: [Cake] flow isolation for ISPs

[Andy Furniss]

Pete Heist wrote:
>
>> Date: Thu, 6 Apr 2017 13:48:00 +0100 From: Andy Furniss
>> <adf.lists@gmail.com> To: cake@lists.bufferbloat.net Subject: Re:
>> [Cake] flow isolation for ISPs
>>
>> Pete Heist wrote:
>>
>>> Cake is not a requirement yet. I like it for several of its
>>> attributes (good performance with high numbers of flows, and also
>>> when “over-limiting”, which I’ll explain more in my next round of
>>> point-to-point WiFi results).
>>
>> Would be nicer for your users though?
>>
>> I mean in the sense that if the "outer" hash could be done on some
>> mark then the inner hash on connection as normal then they would
>> get more than a fifo, which seems to be how current solutions are
>> heading.
>>
>> TBH, though I don't even get your set up - I mean is WISP like some
>> giant lan, or does anyone that asks get a real IP, do subscribers
>> normally have more than one access point? Just curious.
>
> Cake has some nice qualities, yes, so we’ll see.
>
> The network has been assembled over a number of years, by a number
> of people (not by me personally), so it has a life of its own, but
> works quite well, from a member’s perspective. :) I’m still figuring
> out more about it myself, but it consists of a series of nodes, each
> of which has at least a router, a WiFi AP for clients and an uplink
> to the Internet connection either with point-to-point WiFi or some
> other means (fiber, licensed radios, etc). If you want an idea:
>
> http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=satellite&geolocate=98%7C114%7C111%7C117%7C109%7C111%7C118%7C115%7C107%7C97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&tilt=0&heading=0&
>
>
<http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=satellite&geolocate=98|114|111|117|109|111|118|115|107|97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&tilt=0&heading=0&>
>
> In some cases, like mine, my CPE has a single IP address from the
> node’s router. In others, where there are nodes on top of apartment
> buildings for example, members connect with Ethernet straight to the
> router on the roof of the building, and IP addresses come from the
> DHCP server on that router, and they get multiple IPs.
>
> So it’s a mixture in this case, and that’s what makes hashing only
> by IP address, or by IP address and flow, not ideal.

Ahh, OK, complicated then.

FWIW you may already know, but as I haven't read u32 docs for years I
don't know if it says but -

To match macs you use (IIRC from a decade ago) negative offsets, so -2
will get ethertype, -8  for 6 bytes is src -14 for dst.

Re: [Cake] flow isolation for ISPs

[Dave Taht]

If I had a "vision" here, it was to create a more advanced version
of the the ipset facility, where a customer was represented by
all the IP addresses they might have.

Mac addresses might be hard to get at especially if you were
trying to do this at a layer back from the DSLAM or CMTS.

IP 222.22.22.1 -> 1 customer address
IP 222::1/128  -> 1 v6 p2p on the external interface
IP 222::2/128  -> 1 customer router
IP 222:1::/56  -> 1 customer internal delegation

IP 222.22.22.2 -> 2 customer address
IP 222::3/128  -> 2 v6 p2p on the external interface
IP 222::4/128  -> 2 customer router
IP 222:2::/56  -> 2 customer internal delegation

These would all get thrown into one big ipset (and route)
table, and you'd use the index to hand off to a cake-like
qdisc hanging off of something htb-like.

Sadly:

A) Ipset doesn't work this way (you only get matches)
B) Line conditions vary
C) Total throughput possible on the link varies
as a function of the total bandwidth being used.

Who do you throttle when you are out?

PS (this is my first test of switching back to emacs
again to send mail, let me know if it looks funny)

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2356 bytes --]


> Date: Thu, 6 Apr 2017 13:48:00 +0100
> From: Andy Furniss <adf.lists@gmail.com>
> To: cake@lists.bufferbloat.net
> Subject: Re: [Cake] flow isolation for ISPs
> 
> Pete Heist wrote:
> 
>> Cake is not a requirement yet. I like it for several of its
>> attributes (good performance with high numbers of flows, and also
>> when “over-limiting”, which I’ll explain more in my next round of
>> point-to-point WiFi results).
> 
> Would be nicer for your users though?
> 
> I mean in the sense that if the "outer" hash could be done on some mark
> then the inner hash on connection as normal then they would get more
> than a fifo, which seems to be how current solutions are heading.
> 
> TBH, though I don't even get your set up - I mean is WISP like some
> giant lan, or does anyone that asks get a real IP, do subscribers
> normally have more than one access point? Just curious.

Cake has some nice qualities, yes, so we’ll see.

The network has been assembled over a number of years, by a number of people (not by me personally), so it has a life of its own, but works quite well, from a member’s perspective. :) I’m still figuring out more about it myself, but it consists of a series of nodes, each of which has at least a router, a WiFi AP for clients and an uplink to the Internet connection either with point-to-point WiFi or some other means (fiber, licensed radios, etc). If you want an idea:

http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=satellite&geolocate=98%7C114%7C111%7C117%7C109%7C111%7C118%7C115%7C107%7C97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&tilt=0&heading=0& <http://mapa.czfree.net/#lat=50.76816800116274&lng=15.066890716552734&zoom=13&autofilter=1&type=satellite&geolocate=98|114|111|117|109|111|118|115|107|97&node=6101&aponly=1&bbonly=1&actlink=1&actnode=1&tilt=0&heading=0&>

In some cases, like mine, my CPE has a single IP address from the node’s router. In others, where there are nodes on top of apartment buildings for example, members connect with Ethernet straight to the router on the roof of the building, and IP addresses come from the DHCP server on that router, and they get multiple IPs.

So it’s a mixture in this case, and that’s what makes hashing only by IP address, or by IP address and flow, not ideal.

[-- Attachment #2: Type: text/html, Size: 3370 bytes --]

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]


> On Apr 6, 2017, at 3:42 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Ah, right; you can't get the two-level scheduling that Cake does with
> just FQ-CoDel. Didn't realise you were looking for that, sorry...
> 
> You could assign a fixed number of hash buckets to each member (i.e.
> member #1 gets buckets 1-10, say, hashing flows into those). But the
> FQ-CoDel scheduler would be oblivious to the hierarchy, so a member with
> 10 active queues would get service for each of those each time another
> member with just one active queue gets service for his queue.
> 
> To get the hierarchical sub-division, you'd need to have a two-level
> scheme where you have a separate instance of fq-codel per member.

Well, it's only perhaps a requirement. I’m in the middle of some flow isolation tests to look at the difference between Cake’s ‘srchost’ and ‘dual-srchost’ keywords and try to figure out whether one or the other is “better" to use on a ISP backhaul’s egress. I’m not sure yet, but if there are any opinions on it it could help.

But this is also helpful, because either per-member scheduling is good enough, or we’d need multiple fq_codel instances, and at that point it could also be Cake as well… :)

[-- Attachment #2: Type: text/html, Size: 8700 bytes --]

Re: [Cake] flow isolation for ISPs

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Apr 6, 2017, at 2:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Pete Heist <peteheist@gmail.com> writes:
>
>  but I’m still a little confused. Is fq_codel actually a classful
>  qdisc?
>
>  Yup. The hash buckets are the classes, basically. You can get per-flow
>  stats by doing `tc -s class show dev eth0` (only works when a flow has
>  built a queue).
>
> I see now, never knew that… :)

Well, it's not a terribly well-published feature ;)

>  I get the part about matching with tc-filter and the u32 selector (as
>  intuitive as that is :), but am not sure of the action the filter
>  needs to take. However, I do see the example towards the bottom of the
>  tc-u32 man page where a hash table is created and filters move packets
>  into the right buckets. Perhaps it will be eventually decipherable
>  from this… :)
>
>  The filter classifies a packet into a class. The 'minor number' of this
>  is the queue number (so needs to be less than the number of flows you
>  configured for fq_codel - 1024 by default).
>
> It’s clearer to me now how to split traffic by member (using their MAC
> addresses- and by clear I mean not totally clear yet but should become
> clear :) but to get both per-member and per-flow hashing together
> isn’t as clear yet (which I think is what Cake’s dual-srchost does).
> Maybe a separate table per-member could be created. ‘u32’ looks
> arcane, but flexible...

Ah, right; you can't get the two-level scheduling that Cake does with
just FQ-CoDel. Didn't realise you were looking for that, sorry...

You could assign a fixed number of hash buckets to each member (i.e.
member #1 gets buckets 1-10, say, hashing flows into those). But the
FQ-CoDel scheduler would be oblivious to the hierarchy, so a member with
10 active queues would get service for each of those each time another
member with just one active queue gets service for his queue.

To get the hierarchical sub-division, you'd need to have a two-level
scheme where you have a separate instance of fq-codel per member.

-Toke

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]


> On Apr 6, 2017, at 2:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
>> Pete Heist <peteheist@gmail.com> writes:
>> but I’m still a little confused. Is fq_codel actually a classful
>> qdisc?
> 
> Yup. The hash buckets are the classes, basically. You can get per-flow
> stats by doing `tc -s class show dev eth0` (only works when a flow has
> built a queue).

I see now, never knew that… :)

>> I get the part about matching with tc-filter and the u32 selector (as
>> intuitive as that is :), but am not sure of the action the filter
>> needs to take. However, I do see the example towards the bottom of the
>> tc-u32 man page where a hash table is created and filters move packets
>> into the right buckets. Perhaps it will be eventually decipherable
>> from this… :)
> 
> The filter classifies a packet into a class. The 'minor number' of this
> is the queue number (so needs to be less than the number of flows you
> configured for fq_codel - 1024 by default).

It’s clearer to me now how to split traffic by member (using their MAC addresses- and by clear I mean not totally clear yet but should become clear :) but to get both per-member and per-flow hashing together isn’t as clear yet (which I think is what Cake’s dual-srchost does). Maybe a separate table per-member could be created. ‘u32’ looks arcane, but flexible...

[-- Attachment #2: Type: text/html, Size: 7425 bytes --]

Re: [Cake] flow isolation for ISPs

[Andy Furniss]

Pete Heist wrote:

> Cake is not a requirement yet. I like it for several of its
> attributes (good performance with high numbers of flows, and also
> when “over-limiting”, which I’ll explain more in my next round of
> point-to-point WiFi results).

Would be nicer for your users though?

I mean in the sense that if the "outer" hash could be done on some mark
then the inner hash on connection as normal then they would get more
than a fifo, which seems to be how current solutions are heading.

TBH, though I don't even get your set up - I mean is WISP like some
giant lan, or does anyone that asks get a real IP, do subscribers
normally have more than one access point? Just curious.

Re: [Cake] flow isolation for ISPs

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Apr 6, 2017, at 12:50 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Pete Heist <peteheist@gmail.com> writes:
>
>  The fq_codel qdisc does have support for arbitrary tc filters to replace
>  the default hashing, BTW. If you don't need the cake shaper, that might
>  be a solution?
>
>  I see, I found mention of it in Chapter 6 of a draft RFC that it looks
>  like you wrote, actually
>  (https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06#section-6). :)
>  To try it out, am I heading the right direction by looking at tc
>  filter’s skbedit action, or is that just for MQ devices?
>  (http://man7.org/linux/man-pages/man8/tc-skbedit.8.html)
>
>  I also saw this mention of “We are not aware of any deployments
>  utilising the custom classification feature"
>  https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-02#section-5.1.1,
>  so not sure how often this has been tried. :)
>
>  Yeah, haven't actually heard of anyone using the feature in production.
>  It's basically this section from the 'classful qdiscs' section of 'man
>  tc':
>
>        When a packet enters a classful qdisc it can be classified to one of the classes within. Three criteria are available, although not all qdiscs will use all three:
>
>        tc filters
>               If tc filters are attached to a class, they are consulted first for relevant instructions. Filters can match on all fields of a packet header, as well as on the firewall mark applied by ipchains or iptables.
>
>  So you can basically use the full capabilities of tc-filter in place of
>  the built-in hashing of fq_codel. The tc-u32 man page has some examples,
>  which is probably a good starting point.
>
>  If you do try this out and feel like writing up a small
>  example/tutorial, I'm happy to add a link (or the whole thing) somewhere
>  on bufferbloat.net :)
>
> Sure, if I get it working I’ll include an example in my paper,

Awesome!

> but I’m still a little confused. Is fq_codel actually a classful
> qdisc?

Yup. The hash buckets are the classes, basically. You can get per-flow
stats by doing `tc -s class show dev eth0` (only works when a flow has
built a queue).

> I get the part about matching with tc-filter and the u32 selector (as
> intuitive as that is :), but am not sure of the action the filter
> needs to take. However, I do see the example towards the bottom of the
> tc-u32 man page where a hash table is created and filters move packets
> into the right buckets. Perhaps it will be eventually decipherable
> from this… :)

The filter classifies a packet into a class. The 'minor number' of this
is the queue number (so needs to be less than the number of flows you
configured for fq_codel - 1024 by default).

-Toke

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2437 bytes --]


> On Apr 6, 2017, at 12:50 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <peteheist@gmail.com <mailto:peteheist@gmail.com>> writes:
> 
>> The fq_codel qdisc does have support for arbitrary tc filters to replace
>> the default hashing, BTW. If you don't need the cake shaper, that might
>> be a solution?
>> 
>> I see, I found mention of it in Chapter 6 of a draft RFC that it looks
>> like you wrote, actually
>> (https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06#section-6). :)
>> To try it out, am I heading the right direction by looking at tc
>> filter’s skbedit action, or is that just for MQ devices?
>> (http://man7.org/linux/man-pages/man8/tc-skbedit.8.html)
>> 
>> I also saw this mention of “We are not aware of any deployments
>> utilising the custom classification feature"
>> https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-02#section-5.1.1,
>> so not sure how often this has been tried. :)
> 
> Yeah, haven't actually heard of anyone using the feature in production.
> It's basically this section from the 'classful qdiscs' section of 'man
> tc':
> 
>       When a packet enters a classful qdisc it can be classified to one of the classes within. Three criteria are available, although not all qdiscs will use all three:
> 
>       tc filters
>              If tc filters are attached to a class, they are consulted first for relevant instructions. Filters can match on all fields of a packet header, as well as on the firewall mark applied by ipchains or iptables.
> 
> So you can basically use the full capabilities of tc-filter in place of
> the built-in hashing of fq_codel. The tc-u32 man page has some examples,
> which is probably a good starting point.
> 
> If you do try this out and feel like writing up a small
> example/tutorial, I'm happy to add a link (or the whole thing) somewhere
> on bufferbloat.net <http://bufferbloat.net/> :)

Sure, if I get it working I’ll include an example in my paper, but I’m still a little confused. Is fq_codel actually a classful qdisc?

I get the part about matching with tc-filter and the u32 selector (as intuitive as that is :), but am not sure of the action the filter needs to take. However, I do see the example towards the bottom of the tc-u32 man page where a hash table is created and filters move packets into the right buckets. Perhaps it will be eventually decipherable from this… :)

[-- Attachment #2: Type: text/html, Size: 15860 bytes --]

Re: [Cake] flow isolation for ISPs

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

>  On Apr 6, 2017, at 11:33 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>  Once upon a time I implemented something like this; it was basically a
>  PHP script that would generate an HTB bucket (with sfq as leaf qdisc;
>  this was pre-fq_codel) per subscriber ID and use tc filter to map the
>  list of IPs registered to that customer into the right bucket. The HTB
>  shaper was used to enforce the bandwidth each customer was paying for.
>
>  Did it work? Yup, mostly. Was it ugly? Oh boy, yes!
>
> Oh my, ok, so it is possible. It can take a while to apply many qdiscs
> and filters on lower end devices, so I picture some delay while
> modifying the list or restarting the routers, but I’m just exploring
> options now, so it is one.

What I did was 10 years ago on what was fairly high-end x86 hardware at
the time; scaling to 10s (or maybe low 100s) of users. So depending on
the scale, it's probably doable on somewhat cheaper hardware now. But
yeah, if you need to modify things often, you may have problems.

>  The fq_codel qdisc does have support for arbitrary tc filters to replace
>  the default hashing, BTW. If you don't need the cake shaper, that might
>  be a solution?
>
> I see, I found mention of it in Chapter 6 of a draft RFC that it looks
> like you wrote, actually
> (https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06#section-6). :)
> To try it out, am I heading the right direction by looking at tc
> filter’s skbedit action, or is that just for MQ devices?
> (http://man7.org/linux/man-pages/man8/tc-skbedit.8.html)
>
> I also saw this mention of “We are not aware of any deployments
> utilising the custom classification feature"
> https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-02#section-5.1.1,
> so not sure how often this has been tried. :)

Yeah, haven't actually heard of anyone using the feature in production.
It's basically this section from the 'classful qdiscs' section of 'man
tc':

       When a packet enters a classful qdisc it can be classified to one of the classes within. Three criteria are available, although not all qdiscs will use all three:

       tc filters
              If tc filters are attached to a class, they are consulted first for relevant instructions. Filters can match on all fields of a packet header, as well as on the firewall mark applied by ipchains or iptables.

So you can basically use the full capabilities of tc-filter in place of
the built-in hashing of fq_codel. The tc-u32 man page has some examples,
which is probably a good starting point.

If you do try this out and feel like writing up a small
example/tutorial, I'm happy to add a link (or the whole thing) somewhere
on bufferbloat.net :)

-Toke

Re: [Cake] flow isolation for ISPs

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1916 bytes --]


> On Apr 6, 2017, at 11:33 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Once upon a time I implemented something like this; it was basically a
> PHP script that would generate an HTB bucket (with sfq as leaf qdisc;
> this was pre-fq_codel) per subscriber ID and use tc filter to map the
> list of IPs registered to that customer into the right bucket. The HTB
> shaper was used to enforce the bandwidth each customer was paying for.
> 
> Did it work? Yup, mostly. Was it ugly? Oh boy, yes!

Oh my, ok, so it is possible. It can take a while to apply many qdiscs and filters on lower end devices, so I picture some delay while modifying the list or restarting the routers, but I’m just exploring options now, so it is one.

> The fq_codel qdisc does have support for arbitrary tc filters to replace
> the default hashing, BTW. If you don't need the cake shaper, that might
> be a solution?


I see, I found mention of it in Chapter 6 of a draft RFC that it looks like you wrote, actually (https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06#section-6 <https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06>). :) To try it out, am I heading the right direction by looking at tc filter’s skbedit action, or is that just for MQ devices? (http://man7.org/linux/man-pages/man8/tc-skbedit.8.html <http://man7.org/linux/man-pages/man8/tc-skbedit.8.html>)

I also saw this mention of “We are not aware of any deployments utilising the custom classification feature" https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-02#section-5.1.1 <https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-02#section-5.1.1>, so not sure how often this has been tried. :)

Cake is not a requirement yet. I like it for several of its attributes (good performance with high numbers of flows, and also when “over-limiting”, which I’ll explain more in my next round of point-to-point WiFi results).

[-- Attachment #2: Type: text/html, Size: 9268 bytes --]

Re: [Cake] flow isolation for ISPs

[Andy Furniss]

Jonathan Morton wrote:
> Also, Cake’s general philosophy of simplifying configuration means that it’s unlikely to ever support “lists” or “tables” of explicit parameters.  This is a conscious design decision to enable its use by relative non-experts.  Arguably, even some of the existing options could reasonably be streamlined away.

Maybe something as simple as hash on nf-mark?

Re: [Cake] flow isolation for ISPs

[Toke Høiland-Jørgensen]

Pete Heist <peteheist@gmail.com> writes:

> Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)
>
> There’s fairness at the IP address level (currently with esfq, maybe soon with
> Cake), but it's not fair that members with multiple devices effectively get one
> hash bucket per device, so if you have more devices connected at once, you win.
> There is a table of member ID to a list of MAC addresses for the member, so if
> there could somehow be fairness based on that table and by MAC address, that
> could solve it, but I don’t see how it could be implemented.
>
> Is it possible to customize the hashing algorithm used for flow isolation, either with Cake or some other way?
>
> The only options I can think of now:
>
> - force each member to use only one IP address (probably impractical at this point with hundreds of members)
> - use one queue per member in an HTB hierarchy, for example, with
> filters matching each member’s devices, but that seems difficult to
> manage

Once upon a time I implemented something like this; it was basically a
PHP script that would generate an HTB bucket (with sfq as leaf qdisc;
this was pre-fq_codel) per subscriber ID and use tc filter to map the
list of IPs registered to that customer into the right bucket. The HTB
shaper was used to enforce the bandwidth each customer was paying for.

Did it work? Yup, mostly. Was it ugly? Oh boy, yes!


The fq_codel qdisc does have support for arbitrary tc filters to replace
the default hashing, BTW. If you don't need the cake shaper, that might
be a solution?

-Toke

Re: [Cake] flow isolation for ISPs

[Pete Heist]

> On Apr 6, 2017, at 11:11 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>> 
>> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
> 
> One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.

Looking at ‘man tc-flower’, it does sound like an option. I suppose that one instance of Cake per member will be somewhat similar computationally to one instance for multiple members. There will be additional memory used, and additional processing for the filters. This would also probably only be practical if members only use one access point, which I do think is typical. A few dozen filters and Cake instances might be ok, but I’m not sure about 800. :) We’ll see what we come up with. Thanks…

Re: [Cake] flow isolation for ISPs

[Jonathan Morton]

> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
> 
> There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.

One option would be to use HTB with FLOWER filters to sort out the subscribers into classes, and use Cake or fq_codel as a child qdisc per class.  Remember that Cake can be used in “unlimited” mode to rely on an external shaping source.

 - Jonathan Morton

Re: [Cake] flow isolation for ISPs

[Pete Heist]

> On Apr 6, 2017, at 10:57 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
>> 
>> Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)
>> 
>> There’s fairness at the IP address level (currently with esfq, maybe soon with Cake), but it's not fair that members with multiple devices effectively get one hash bucket per device, so if you have more devices connected at once, you win. There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
>> 
>> Is it possible to customize the hashing algorithm used for flow isolation, either with Cake or some other way?
> 
> That is an important use-case, and one that Cake is not presently designed to explicitly accommodate.  Currently, the design assumes a single Cake instance per subscriber or household, and fairness between hosts within a household is assumed to be a relatively simple problem.
> 
> Also, Cake’s general philosophy of simplifying configuration means that it’s unlikely to ever support “lists” or “tables” of explicit parameters.  This is a conscious design decision to enable its use by relative non-experts.  Arguably, even some of the existing options could reasonably be streamlined away.
> 
> With that said, a related qdisc *with* such support is eminently feasible, and could easily be the focus of a project.  I think it would be worth gathering requirements for such a thing and considering potential funding sources.

I figured as much, but it’s good to know for sure, thanks!

Pete

Re: [Cake] flow isolation for ISPs

[Jonathan Morton]

> On 6 Apr, 2017, at 11:27, Pete Heist <peteheist@gmail.com> wrote:
> 
> Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)
> 
> There’s fairness at the IP address level (currently with esfq, maybe soon with Cake), but it's not fair that members with multiple devices effectively get one hash bucket per device, so if you have more devices connected at once, you win. There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
> 
> Is it possible to customize the hashing algorithm used for flow isolation, either with Cake or some other way?

That is an important use-case, and one that Cake is not presently designed to explicitly accommodate.  Currently, the design assumes a single Cake instance per subscriber or household, and fairness between hosts within a household is assumed to be a relatively simple problem.

Also, Cake’s general philosophy of simplifying configuration means that it’s unlikely to ever support “lists” or “tables” of explicit parameters.  This is a conscious design decision to enable its use by relative non-experts.  Arguably, even some of the existing options could reasonably be streamlined away.

With that said, a related qdisc *with* such support is eminently feasible, and could easily be the focus of a project.  I think it would be worth gathering requirements for such a thing and considering potential funding sources.

 - Jonathan Morton

Re: [Cake] flow isolation for ISPs

[Pete Heist]

> On Apr 6, 2017, at 10:39 AM, David Lang <david@lang.hm> wrote:
> 
> On Thu, 6 Apr 2017, Pete Heist wrote:
> 
>> Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)
>> 
>> There’s fairness at the IP address level (currently with esfq, maybe soon with Cake), but it's not fair that members with multiple devices effectively get one hash bucket per device, so if you have more devices connected at once, you win. There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.
> 
> well, if the congested link is not the last-mile link to a user, the right answer is probably to increase the capacity of the link, and fairness issues would be a temporary thing until the link was upgraded.
> 
> remember that the fairness is a means to an end, good reponsive service. As long as the result is responsive for all users, a bit of unfairness is acceptable.

It’s a good point, but for a non-profit cooperative WISP, each hardware upgrade/installation can be a decision. In some locations and times there’s congestion in the backhaul, in others not. So it’s not always practical in this case to solve it by adding more hardware…

Re: [Cake] flow isolation for ISPs

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]

On Thu, 6 Apr 2017, Pete Heist wrote:

> Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)
>
> There’s fairness at the IP address level (currently with esfq, maybe soon with Cake), but it's not fair that members with multiple devices effectively get one hash bucket per device, so if you have more devices connected at once, you win. There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.

well, if the congested link is not the last-mile link to a user, the right 
answer is probably to increase the capacity of the link, and fairness issues 
would be a temporary thing until the link was upgraded.

remember that the fairness is a means to an end, good reponsive service. As long 
as the result is responsive for all users, a bit of unfairness is acceptable.

David Lang

> Is it possible to customize the hashing algorithm used for flow isolation, either with Cake or some other way?
>
> The only options I can think of now:
>
> - force each member to use only one IP address (probably impractical at this point with hundreds of members)
> - use one queue per member in an HTB hierarchy, for example, with filters matching each member’s devices, but that seems difficult to manage
> - wait years for IPv6 deployment, allocate subnets to each member and wait for qdiscs that have the ability to hash by IPv6 subnet :)
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

[Cake] flow isolation for ISPs

[Pete Heist]

Suppose there is a cooperative ISP that has some members who access the network through a single device (like a router with NAT), while others use multiple devices and leave routing to the ISPs routers. (No need to suppose, actually.)

There’s fairness at the IP address level (currently with esfq, maybe soon with Cake), but it's not fair that members with multiple devices effectively get one hash bucket per device, so if you have more devices connected at once, you win. There is a table of member ID to a list of MAC addresses for the member, so if there could somehow be fairness based on that table and by MAC address, that could solve it, but I don’t see how it could be implemented.

Is it possible to customize the hashing algorithm used for flow isolation, either with Cake or some other way?

The only options I can think of now:

- force each member to use only one IP address (probably impractical at this point with hundreds of members)
- use one queue per member in an HTB hierarchy, for example, with filters matching each member’s devices, but that seems difficult to manage
- wait years for IPv6 deployment, allocate subnets to each member and wait for qdiscs that have the ability to hash by IPv6 subnet :)