Re: [Cake] Using cake to shape 1000’s of users.

Re: [Cake] Using cake to shape 1000’s of users.

[Felix Resch]

since commercial interest is involved, see here https://lists.bufferbloat.net/pipermail/cake/2018-June/003861.html

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Tue, Jul 17, 2018 at 12:24 AM Felix Resch <fuller@beif.de> wrote:
>
> since commercial interest is involved, see here https://lists.bufferbloat.net/pipermail/cake/2018-June/003861.html

I grew that list substantially in the ending talk. It was motivating.
:) I am thinking of doing something similar (with editorial comments)
pointing
to each of dslreports' values per ISP, like, for example, leveraging

http://www.dslreports.com/speedtest/results/bufferbloat?up=1

To kvetch while pointing further to stuff like:

http://www.dslreports.com/speedtest/results/isp/r3910-google-fiber
http://www.dslreports.com/speedtest/results/isp/r2784-t-mobile
http://www.dslreports.com/speedtest/results/isp/r896-sonic
http://www.dslreports.com/speedtest/results/isp/r1579-Comcast%20XFINITY

That angst out, ISPs and vendors that want to work with us to
establish requirements and code for a transparent
bridge/veth/cake-like thing are very welcome at any point! The core
factor stopping us from even trying isn't lack of money or time... it
is not knowing of *any* head-end equipment (DSLAM/CMTS/GPON/etc) that
could be modified by us to have better queue management in the first
place, and a transparent bridge seems second best, at best, with
complexities involving ipv6 and ipv4 support, sag, nat, vlans, etc,
etc - so we've focused on fixing the edge device itself, and making
available published open source code and standards in the hope that
some head-end vendor would pick it up... after being suitably nagged
by their ISP customers. Or the sandvine type middlebox folk.

Also, in terms of angst, we hope merely that ISPs would start
supplying CPE that has our stuff in it (particularly since the
aftermarket already has it, and it works in even the cheapest boxes
made today), and either autoconfigure to their set rates, with cake,
or supply that information to their end users to configure. It's been
6 years since the code hit the embedded world.... I'm pleased to say
that "fq_codel for wifi" derivatives seem to propagating rapidly, at
least. Maybe a fortigate or barracuda will ship some kind of smart
queue management one day soon... if enough customers ask for it.
example: https://forum.fortinet.com/tm.aspx?m=163978

on the transparent bridge front... Linux has issues scaling to high
rates *on the receive path* at 10gigE+ speeds.

I've certainly lain awake at night dreaming of what we could do with a
smart line card for a cmts, or even a small dslam.

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

Tiny bit of self promotion here but Preseem (https://www.preseem.com)
is a transparent bridge that leverages HTB/FQ-CoDel to make subscriber plan enforcement provide much better QoE. Leaving enforcement up to the deep queues in most network equipment has comparably very bad results. We focus on WISPs but we have customers that provide service via DSL and cable as well.

We leverage an eBPF classifier to direct each subscriber's traffic into
an HTB class that matches their plan rate and within that is an FQ-
CoDel instance. This classifier handles the various encapsulations we
need to support in ISP networks.

I haven't had time to try Cake in this context yet but hope to get to
that in the next couple months. I believe this will require one Cake
instance per-subscriber like we do with FQ-CoDel today.


On Tue, 2018-07-17 at 09:59 -0700, Dave Taht wrote:
> On Tue, Jul 17, 2018 at 12:24 AM Felix Resch <fuller@beif.de> wrote:
> > 
> > since commercial interest is involved, see here 
> > https://lists.bufferbloat.net/pipermail/cake/2018-June/003861.html
> 
> I grew that list substantially in the ending talk. It was motivating.
> :) I am thinking of doing something similar (with editorial comments)
> pointing
> to each of dslreports' values per ISP, like, for example, leveraging
> 
> http://www.dslreports.com/speedtest/results/bufferbloat?up=1
> 
> To kvetch while pointing further to stuff like:
> 
> http://www.dslreports.com/speedtest/results/isp/r3910-google-fiber
> http://www.dslreports.com/speedtest/results/isp/r2784-t-mobile
> http://www.dslreports.com/speedtest/results/isp/r896-sonic
> 
http://www.dslreports.com/speedtest/results/isp/r1579-Comcast%20XFINITY
> 
> That angst out, ISPs and vendors that want to work with us to
> establish requirements and code for a transparent
> bridge/veth/cake-like thing are very welcome at any point! The core
> factor stopping us from even trying isn't lack of money or time... it
> is not knowing of *any* head-end equipment (DSLAM/CMTS/GPON/etc) that
> could be modified by us to have better queue management in the first
> place, and a transparent bridge seems second best, at best, with
> complexities involving ipv6 and ipv4 support, sag, nat, vlans, etc,
> etc - so we've focused on fixing the edge device itself, and making
> available published open source code and standards in the hope that
> some head-end vendor would pick it up... after being suitably nagged
> by their ISP customers. Or the sandvine type middlebox folk.
> 
> Also, in terms of angst, we hope merely that ISPs would start
> supplying CPE that has our stuff in it (particularly since the
> aftermarket already has it, and it works in even the cheapest boxes
> made today), and either autoconfigure to their set rates, with cake,
> or supply that information to their end users to configure. It's been
> 6 years since the code hit the embedded world.... I'm pleased to say
> that "fq_codel for wifi" derivatives seem to propagating rapidly, at
> least. Maybe a fortigate or barracuda will ship some kind of smart
> queue management one day soon... if enough customers ask for it.
> example: https://forum.fortinet.com/tm.aspx?m=163978
> 
> on the transparent bridge front... Linux has issues scaling to high
> rates *on the receive path* at 10gigE+ speeds.
> 
> I've certainly lain awake at night dreaming of what we could do with
> a
> smart line card for a cmts, or even a small dslam.
> 
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Thu, Jul 26, 2018 at 8:46 AM Dan Siemon <dan@coverfire.com> wrote:
>
> Tiny bit of self promotion here but Preseem (https://www.preseem.com)
> is a transparent bridge that leverages HTB/FQ-CoDel to make subscriber plan enforcement provide much better QoE. Leaving enforcement up to the deep queues in most network equipment has comparably very bad results. We focus on WISPs but we have customers that provide service via DSL and cable as well.

AWESOME. I am curious as to how high you can scale currently?
(subscribers, bandwidths)

> We leverage an eBPF classifier to direct each subscriber's traffic into
> an HTB class that matches their plan rate and within that is an FQ-
> CoDel instance. This classifier handles the various encapsulations we
> need to support in ISP networks.
>
> I haven't had time to try Cake in this context yet but hope to get to
> that in the next couple months. I believe this will require one Cake
> instance per-subscriber like we do with FQ-CoDel today.
>
>
> On Tue, 2018-07-17 at 09:59 -0700, Dave Taht wrote:
> > On Tue, Jul 17, 2018 at 12:24 AM Felix Resch <fuller@beif.de> wrote:
> > >
> > > since commercial interest is involved, see here
> > > https://lists.bufferbloat.net/pipermail/cake/2018-June/003861.html
> >
> > I grew that list substantially in the ending talk. It was motivating.
> > :) I am thinking of doing something similar (with editorial comments)
> > pointing
> > to each of dslreports' values per ISP, like, for example, leveraging
> >
> > http://www.dslreports.com/speedtest/results/bufferbloat?up=1
> >
> > To kvetch while pointing further to stuff like:
> >
> > http://www.dslreports.com/speedtest/results/isp/r3910-google-fiber
> > http://www.dslreports.com/speedtest/results/isp/r2784-t-mobile
> > http://www.dslreports.com/speedtest/results/isp/r896-sonic
> >
> http://www.dslreports.com/speedtest/results/isp/r1579-Comcast%20XFINITY
> >
> > That angst out, ISPs and vendors that want to work with us to
> > establish requirements and code for a transparent
> > bridge/veth/cake-like thing are very welcome at any point! The core
> > factor stopping us from even trying isn't lack of money or time... it
> > is not knowing of *any* head-end equipment (DSLAM/CMTS/GPON/etc) that
> > could be modified by us to have better queue management in the first
> > place, and a transparent bridge seems second best, at best, with
> > complexities involving ipv6 and ipv4 support, sag, nat, vlans, etc,
> > etc - so we've focused on fixing the edge device itself, and making
> > available published open source code and standards in the hope that
> > some head-end vendor would pick it up... after being suitably nagged
> > by their ISP customers. Or the sandvine type middlebox folk.
> >
> > Also, in terms of angst, we hope merely that ISPs would start
> > supplying CPE that has our stuff in it (particularly since the
> > aftermarket already has it, and it works in even the cheapest boxes
> > made today), and either autoconfigure to their set rates, with cake,
> > or supply that information to their end users to configure. It's been
> > 6 years since the code hit the embedded world.... I'm pleased to say
> > that "fq_codel for wifi" derivatives seem to propagating rapidly, at
> > least. Maybe a fortigate or barracuda will ship some kind of smart
> > queue management one day soon... if enough customers ask for it.
> > example: https://forum.fortinet.com/tm.aspx?m=163978
> >
> > on the transparent bridge front... Linux has issues scaling to high
> > rates *on the receive path* at 10gigE+ speeds.
> >
> > I've certainly lain awake at night dreaming of what we could do with
> > a
> > smart line card for a cmts, or even a small dslam.
> >
> > > _______________________________________________
> > > Cake mailing list
> > > Cake@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/cake
> >
> >
> >
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Dan Siemon <dan@coverfire.com> writes:

> Tiny bit of self promotion here but Preseem (https://www.preseem.com)
> is a transparent bridge that leverages HTB/FQ-CoDel to make subscriber
> plan enforcement provide much better QoE. Leaving enforcement up to
> the deep queues in most network equipment has comparably very bad
> results. We focus on WISPs but we have customers that provide service
> via DSL and cable as well.
>
> We leverage an eBPF classifier to direct each subscriber's traffic
> into an HTB class that matches their plan rate and within that is an
> FQ- CoDel instance. This classifier handles the various encapsulations
> we need to support in ISP networks.

This sounds really cool, and is definitely the right way to do things!
Neat :)

> I haven't had time to try Cake in this context yet but hope to get to
> that in the next couple months. I believe this will require one Cake
> instance per-subscriber like we do with FQ-CoDel today.

Yup, currently it would. It might be possible to extend CAKE's
architecture to not require this, though. If you are interested in
pursuing this in the future, let us know; having someone actually using
it would be quite useful if we were to pursue development of this :)

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

On Thu, 2018-07-26 at 08:48 -0700, Dave Taht wrote:
> On Thu, Jul 26, 2018 at 8:46 AM Dan Siemon <dan@coverfire.com> wrote:
> > 
> > Tiny bit of self promotion here but Preseem (
> > https://www.preseem.com)
> > is a transparent bridge that leverages HTB/FQ-CoDel to make
> > subscriber plan enforcement provide much better QoE. Leaving
> > enforcement up to the deep queues in most network equipment has
> > comparably very bad results. We focus on WISPs but we have
> > customers that provide service via DSL and cable as well.
> 
> AWESOME. I am curious as to how high you can scale currently?
> (subscribers, bandwidths)

Since our boxes are low cost, we tend towards slightly more distributed
deployments with smaller boxes.

However, a i7-7700 w/ X710 NICs can do about 4Gb/sec with 4000
subscribers with ~25% CPU usage.

Re: [Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

On Thu, 2018-07-26 at 19:42 +0200, Toke Høiland-Jørgensen wrote:
> Dan Siemon <dan@coverfire.com> writes:
> 
> > I haven't had time to try Cake in this context yet but hope to get
> > to
> > that in the next couple months. I believe this will require one
> > Cake
> > instance per-subscriber like we do with FQ-CoDel today.
> 
> Yup, currently it would. It might be possible to extend CAKE's
> architecture to not require this, though. If you are interested in
> pursuing this in the future, let us know; having someone actually
> using
> it would be quite useful if we were to pursue development of this :)

Yes, I'm very interested in this. A single QDisc would simplify things
quite a bit and hopefully be a perf win.

I'm happy to test this locally and with real subscribers once it is
stable enough.

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Dan Siemon <dan@coverfire.com> writes:

> On Thu, 2018-07-26 at 19:42 +0200, Toke Høiland-Jørgensen wrote:
>> Dan Siemon <dan@coverfire.com> writes:
>> 
>> > I haven't had time to try Cake in this context yet but hope to get
>> > to
>> > that in the next couple months. I believe this will require one
>> > Cake
>> > instance per-subscriber like we do with FQ-CoDel today.
>> 
>> Yup, currently it would. It might be possible to extend CAKE's
>> architecture to not require this, though. If you are interested in
>> pursuing this in the future, let us know; having someone actually
>> using
>> it would be quite useful if we were to pursue development of this :)
>
> Yes, I'm very interested in this. A single QDisc would simplify things
> quite a bit and hopefully be a perf win.
>
> I'm happy to test this locally and with real subscribers once it is
> stable enough.

Cool. No promises as to when (or even if) I get around to looking at
this; but will ping you when/if I do :)

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> On 27 Jul, 2018, at 12:09 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
>>>> I haven't had time to try Cake in this context yet but hope to get to
>>>> that in the next couple months. I believe this will require one Cake
>>>> instance per-subscriber like we do with FQ-CoDel today.
>>> 
>>> Yup, currently it would. It might be possible to extend CAKE's
>>> architecture to not require this, though. If you are interested in
>>> pursuing this in the future, let us know; having someone actually
>>> using it would be quite useful if we were to pursue development of this :)
>> 
>> Yes, I'm very interested in this. A single QDisc would simplify things
>> quite a bit and hopefully be a perf win.
>> 
>> I'm happy to test this locally and with real subscribers once it is
>> stable enough.
> 
> Cool. No promises as to when (or even if) I get around to looking at
> this; but will ping you when/if I do :)

Alternatively, I could look into it.  I have time, but funding would be nice.  Perhaps if several organisations have some commercial interest in the idea, they could pool some resources to keep my fridge stocked and a roof over my head?

It would also be valuable to have a firmer handle on the actual requirements in the field.  For example, if it is feasible to focus only on current Linux kernels, then a lot of backwards compatibility cruft can be excised when importing existing code from Cake.  If all users will have the same link-layer technology (with the same overhead parameters), then these can be set globally - or if not, they can be set per-tier.  Is the Diffserv support from Cake likely to be useful, and if so how flexible should the configuration be?  And are there only a few discrete settings for bandwidth per user, or do we have to be more flexible to handle a BRAS environment?  Is it also necessary to account per-user traffic accurately, or will an external tool be used for that?

Many other low-level considerations can be adjusted on the fly during the conversion, so they are not in themselves a problem.  In this category are questions like "how many simultaneous users and flows can be supported".  There are ways to design the code so that it scales up much more nicely in these dimensions than Cake does, at the expense of a few extra CPU cycles per packet.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 26, 2018, at 11:38 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 27 Jul, 2018, at 12:09 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Cool. No promises as to when (or even if) I get around to looking at
>> this; but will ping you when/if I do :)
> 
> Alternatively, I could look into it.  I have time, but funding would be nice.  Perhaps if several organisations have some commercial interest in the idea, they could pool some resources to keep my fridge stocked and a roof over my head?
> It would also be valuable to have a firmer handle on the actual requirements in the field.

Perhaps taking your questions (plus any others we think of) and putting up a survey, maybe with Google Forms for ease, would both gather requirements and generate organizational interest. Shall I start one with your questions that we can collaboratively edit before soliciting responses?

Pete

(also trying to take a summer bloat break too but proving difficult with many interesting things going on- might need to unsubscribe or bury some devices :)

Re: [Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

On Fri, 2018-07-27 at 00:38 +0300, Jonathan Morton wrote:
> It would also be valuable to have a firmer handle on the actual
> requirements in the field.  For example, if it is feasible to focus
> only on current Linux kernels, then a lot of backwards compatibility
> cruft can be excised when importing existing code from Cake.  If all
> users will have the same link-layer technology (with the same
> overhead parameters), then these can be set globally - or if not,
> they can be set per-tier.  Is the Diffserv support from Cake likely
> to be useful, and if so how flexible should the configuration
> be?  And are there only a few discrete settings for bandwidth per
> user, or do we have to be more flexible to handle a BRAS
> environment?  Is it also necessary to account per-user traffic
> accurately, or will an external tool be used for that?

Obviously I can't speak for other potential users but we follow the
upstream kernel very aggressively and have no interest in porting
something like this to older kernels.

We have some deployments with multiple access technologies (eg DOCSIS,
DSL and wireless) behind the same box so per customer overhead would be
useful.

I am interested in the diffserv class ideas in Cake but need to do some
experimentation to see if that helps in this environment. It would be
interesting to be able to target sub-classes (not sure what the proper
Cake terminology is) based on a eBPF classifier.

Re accounting, we currently count per-IP via eBPF not via qdisc counts.

Each subscriber can have several IPs in which case the traffic for
those IPs needs to go to a single class so that their entire traffic
envelope is shaped to the desired plan rate. At present we do this via
eBPF since it is essentially a map lookup operation.

A few of the above points touch on something that may be somewhat
unique service provider deployments vs homes, there are many situations
where the classification logic has to be a map lookup and cannot be
done just by looking at a given packet.

> Many other low-level considerations can be adjusted on the fly during
> the conversion, so they are not in themselves a problem.  In this
> category are questions like "how many simultaneous users and flows
> can be supported".  There are ways to design the code so that it
> scales up much more nicely in these dimensions than Cake does, at the
> expense of a few extra CPU cycles per packet.
> 
>  - Jonathan Morton
> 
> 

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> On 27 Jul, 2018, at 5:04 pm, Dan Siemon <dan@coverfire.com> wrote:
> 
> Obviously I can't speak for other potential users but we follow the
> upstream kernel very aggressively and have no interest in porting
> something like this to older kernels.

That's a useful data point.  Honestly it shouldn't be too difficult to install the latest kernels on a dedicated box in general.

> We have some deployments with multiple access technologies (eg DOCSIS,
> DSL and wireless) behind the same box so per customer overhead would be
> useful.

The design I presently have in mind would allow setting the overhead per speed tier.  Thus you could have "10Mbit DOCSIS" and "10Mbit ADSL" as separate tiers.  That is, there's likely to be far fewer unique overhead settings than customers.

> I am interested in the diffserv class ideas in Cake but need to do some
> experimentation to see if that helps in this environment. It would be
> interesting to be able to target sub-classes (not sure what the proper
> Cake terminology is) based on a eBPF classifier.

The trick with Diffserv is classifying the traffic correctly, given that most traffic in the wild is not properly marked, and some actively tries to hide its nature.  As an ISP, applying your own rules could be seen as questionable according to some interpretations of Net Neutrality.  With Cake that's not an issue by default, since it relies entirely on the DSCP on the packet itself, but it does mean that a lot of traffic which *should* probably be classified other than best-effort is not.

Cake does cope well with unclassified latency-sensitive traffic, due to DRR++, so it's not actually necessary to specifically identify such applications.  What it has trouble with, in common with all other flow-isolating qdiscs, is applications which open many parallel connections for bulk transfer (eg. BitTorrent, which is used by many software-update mechanisms as well as file-sharing).  Classifying this traffic as Bulk (CS1 DSCP) gets it out of the way of other applications while still permitting them to use available capacity.

I have some small hope that wider deployment of sensible, basic Diffserv at bottlenecks will encourage applications to start using it as intended, thereby solving a decades-long chicken-egg problem.  Presently Cake has such an implementation by default, but is not yet widely deployed enough to have any visible impact.

This is something we should probably discuss in more detail later.

> Re accounting, we currently count per-IP via eBPF not via qdisc counts.

Fine, that's a feature I'm happy to leave out.

> Each subscriber can have several IPs in which case the traffic for
> those IPs needs to go to a single class so that their entire traffic
> envelope is shaped to the desired plan rate. At present we do this via
> eBPF since it is essentially a map lookup operation.
> 
> A few of the above points touch on something that may be somewhat
> unique service provider deployments vs homes, there are many situations
> where the classification logic has to be a map lookup and cannot be
> done just by looking at a given packet.

Yes, eBPF does seem to be a good fit for that.

So in summary, the logical flow of a packet should be:

1: Map dst or src IP to subscriber (eBPF).
2: Map subscriber to speed/overhead tier (eBPF).
3: (optional) Classify Diffserv (???).
4: Enqueue per flow, handle global queue overflow (rare!) by dropping from head of longest queue (like Cake).
--- enqueue/dequeue split ---
5: Wait for shaper on earliest-scheduled subscriber link with waiting traffic (borrow sch_fq's rbtree?).
6: Wait for shaper on aggregate backhaul link (congestion can happen here too).
7: Choose one of subscriber's queues, apply AQM and deliver a packet (mostly like Cake does).

If that seems reasonable, we can treat it as a baseline spec for others' input.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 26, 2018, at 11:38 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> It would also be valuable to have a firmer handle on the actual requirements in the field.  For example, if it is feasible to focus only on current Linux kernels, then a lot of backwards compatibility cruft can be excised when importing existing code from Cake.

One more data point, for FreeNet Liberec (WISP co-op)...


There are some older backhaul routers still with 2.6.26.8(!) although those are being phased out so don’t count them. More current ones use 3.16.7 and there’s some discussion but I’m not sure what/when the upgrade plan is. I think the Internet router uses a more modern Debian 9 which is likely to have a 4.9 series, but I don’t have access to it.

FreeNet might be unusual in that administration is distributed. Volunteer members administer backhaul routers which carry some number of customers, average a few dozen or so. These do routing/firewalling/monitoring and QoS, when they’re a bottleneck. They use either an ancient esfq (has per-IP fairness though) on older kernels and sfq when it’s not available, which is now more common. There are arguments (heated ones, I’ve heard) about centralizing administrative functions, including QoS, at or near the Internet gateway, but I think this would have to depend on over-provisioning the backhaul. I’ve volunteered to modernize the QoS when I can. If ISP flavored Cake doesn’t happen, it will end up being HTB+sfq|fq_codel|cake depending on what the kernel supports.

>  If all users will have the same link-layer technology (with the same overhead parameters), then these can be set globally - or if not, they can be set per-tier.

Afaik almost all members use WiFi, possibly some Ethernet. There are no tiers and no per-member rate limits.

>  Is the Diffserv support from Cake likely to be useful, and if so how flexible should the configuration be?  

Currently the root qdisc on backhaul routers is a prio with "bands 3 priomap 2 2 2 2 2 2 0 0 2 2 2 2 2 2 2 2”. I don’t think we’d want DSCP for anything, with the possible exception of voip, and even then there might just be a special IP/port rule for the asterisk server.

> And are there only a few discrete settings for bandwidth per user, or do we have to be more flexible to handle a BRAS environment?  

Mainly in this case only fairness between members is needed. There’s a db mapping members to their MAC addresses (usually one to one but not always).

> Is it also necessary to account per-user traffic accurately, or will an external tool be used for that?

It would be better than what is done now (counting per-IP, which when a customer has multiple MACs makes it harder to interpret).

> Many other low-level considerations can be adjusted on the fly during the conversion, so they are not in themselves a problem.  In this category are questions like "how many simultaneous users and flows can be supported”.

In this case something that works to a few thousand users is likely to be fine (currently, 800 members).


Lastly, do you think a better shaper for point-to-point WiFi is within the scope of this project? This might be more needed for WISPs, but here’s why:

- If we still have bottlenecks in the backhaul, we’ll need to keep doing QoS there, and for FreeNet that means soft rate limiting, because the WiFi devices have to keep running airOS for its management tools.
- If you only rate limit on egress, you lose at least half your available throughput.
- You can run egress and ingress through a common IFB, but in my testing, TCP RTT (rrul_be with —socket-stats) gets higher.
- I find it works better to use HTB at the root and have HTB+cake as leaf queues, then TCP RTT is roughly cut in half.
- But it would be even better if the shaper understood an approximation of airtime, because aggregate throughput changes based on the balance of up/down traffic in the case where up/down rates are stable but asymmetric, which FreeNet sometimes has.
- And, I’d rather not have to use HTB, and be able to use the better deficit mode shaper in Cake.

I know that WiFi has many complexities where soft rate limiting can’t ever be perfect without knowledge from the driver, but I still think it could be useful to have something that does a better approximation than what we have today. The question is, is that something that could be part of this project, or not… :)

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> There are some older backhaul routers still with 2.6.26.8(!) although those are being phased out so don’t count them. More current ones use 3.16.7 and there’s some discussion but I’m not sure what/when the upgrade plan is. I think the Internet router uses a more modern Debian 9 which is likely to have a 4.9 series, but I don’t have access to it.
> 
> FreeNet might be unusual in that administration is distributed. Volunteer members administer backhaul routers which carry some number of customers, average a few dozen or so. These do routing/firewalling/monitoring and QoS, when they’re a bottleneck. They use either an ancient esfq (has per-IP fairness though) on older kernels and sfq when it’s not available, which is now more common. There are arguments (heated ones, I’ve heard) about centralizing administrative functions, including QoS, at or near the Internet gateway, but I think this would have to depend on over-provisioning the backhaul. I’ve volunteered to modernize the QoS when I can. If ISP flavored Cake doesn’t happen, it will end up being HTB+sfq|fq_codel|cake depending on what the kernel supports.

This sounds like a relatively complex network topology, in which there are a lot of different potential bottlenecks, depending on the dynamic state of the network.  But it's encouraging to hear that you have *some* sort of solution in place, even if it's using rather old techniques at present.

I think we should treat wired and wireless backhaul separately here.  By wireless I specifically mean shared-medium links which are, by design, half-duplex.

>> If all users will have the same link-layer technology (with the same overhead parameters), then these can be set globally - or if not, they can be set per-tier.
> 
> Afaik almost all members use WiFi, possibly some Ethernet. There are no tiers and no per-member rate limits.

Okay, so straight away we're in a significantly different regime from CoverFire's situation, where the subscribers each have a defined link bandwidth (which may or may not be related to the capabilities of the underlying physical link).  You simply need to share some backhaul link fairly between subscribers using it at any given moment.

I haven't yet considered whether this capability might fall naturally out of the implementation of a shaped-subscriber model set to infinite rate, or some other sane default.  If not, a separate algorithm will be required for that case.  But it's useful to have it on the radar.

>> Is the Diffserv support from Cake likely to be useful, and if so how flexible should the configuration be?  
> 
> Currently the root qdisc on backhaul routers is a prio with "bands 3 priomap 2 2 2 2 2 2 0 0 2 2 2 2 2 2 2 2”. I don’t think we’d want DSCP for anything, with the possible exception of voip, and even then there might just be a special IP/port rule for the asterisk server.

Let's call this one a vote for "diffserv not required", since DRR++ copes well by prioritising sparse traffic.

>> And are there only a few discrete settings for bandwidth per user, or do we have to be more flexible to handle a BRAS environment?  
> 
> Mainly in this case only fairness between members is needed. There’s a db mapping members to their MAC addresses (usually one to one but not always).

Could you convert that DB to eBPF rules?  This would let you use the same configuration interface as CoverFire's situation.

>> Is it also necessary to account per-user traffic accurately, or will an external tool be used for that?
> 
> It would be better than what is done now (counting per-IP, which when a customer has multiple MACs makes it harder to interpret).

It appears that this can also be done within eBPF.  Don't ask me the details of how; I haven't yet looked into what eBPF is actually capable of.

> Lastly, do you think a better shaper for point-to-point WiFi is within the scope of this project? This might be more needed for WISPs, but here’s why:
> 
> - If we still have bottlenecks in the backhaul, we’ll need to keep doing QoS there, and for FreeNet that means soft rate limiting, because the WiFi devices have to keep running airOS for its management tools.
> - If you only rate limit on egress, you lose at least half your available throughput.
> - You can run egress and ingress through a common IFB, but in my testing, TCP RTT (rrul_be with —socket-stats) gets higher.
> - I find it works better to use HTB at the root and have HTB+cake as leaf queues, then TCP RTT is roughly cut in half.
> - But it would be even better if the shaper understood an approximation of airtime, because aggregate throughput changes based on the balance of up/down traffic in the case where up/down rates are stable but asymmetric, which FreeNet sometimes has.
> - And, I’d rather not have to use HTB, and be able to use the better deficit mode shaper in Cake.
> 
> I know that WiFi has many complexities where soft rate limiting can’t ever be perfect without knowledge from the driver, but I still think it could be useful to have something that does a better approximation than what we have today. The question is, is that something that could be part of this project, or not… :)

In general, I think the make-wifi-fast team has a better handle on the subtleties of half-duplex links.  If there's any way to get their work into the airOS devices, so much the better.

Otherwise, your problem basically boils down to the problem of choosing a suitable rate limit, which you can dynamically update Cake's configuration with from userspace (tc qdisc change…) without losing any packets or fairness state.  Since the out-of-tree version of "normal" Cake already works with relatively old kernels, this seems like a good way to deploy a worthwhile improvement.  If the traffic load on backhaul links is complex enough (more than about several hundred *simultaneous* flows) to overstress Cake's flow-isolation capabilities, you might try using its "src-host" and "dst-host" modes instead of "flows" or "dual-src/dst".

But I think there's also a use for "ISP-type" Cake in your network, especially in the wired backhauls where link bandwidth is relatively predictable, and I suspect most of these will be in the core parts of your network where the traffic is most complex.  As long as you can get around to running a suitable kernel version, there should also be no inherent problem with replicating the same dynamic adjustment of global shaper rate as "normal" Cake has, which will allow it to be used on some of your wireless links as well.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

> Yes, eBPF does seem to be a good fit for that.
>
> So in summary, the logical flow of a packet should be:
>
> 1: Map dst or src IP to subscriber (eBPF).
> 2: Map subscriber to speed/overhead tier (eBPF).
> 3: (optional) Classify Diffserv (???).
> 4: Enqueue per flow, handle global queue overflow (rare!) by dropping
> from head of longest queue (like Cake).

Note that with the existing tc classifier stuff we already added to
Cake, we basically have this already (eBPF can map traffic to tin and
flow however it pleases).

> --- enqueue/dequeue split ---
> 5: Wait for shaper on earliest-scheduled subscriber link with waiting traffic (borrow sch_fq's rbtree?).
> 6: Wait for shaper on aggregate backhaul link (congestion can happen here too).
> 7: Choose one of subscriber's queues, apply AQM and deliver a packet (mostly like Cake does).
>
> If that seems reasonable, we can treat it as a baseline spec for
> others' input.

I think that the minimum modification to the existing CAKE code base
would be to just support an arbitrary (configurable) number of tins that
(eBPF) tc filters can map traffic into. You'd need a better way of
selecting the next tin to service; I think rbtree is a reasonable choice
here. And probably some way of avoiding allocating CAKE_FLOWS queues per
tin if there are a lot of them. The fq structure used for WiFi (which
was inspired by CAKE in the first place) solves this by allocating one
big batch of queues and mapping them to tins as packets are assigned to
them; this is in fq_impl.h.

The above is obviously minded on what would be the minimum required to
support an "ISP shaper" use case in the existing CAKE qdisc. If you're
designing a whole new qdisc, obviously there are other ways of
structuring things; so see the above more as inspiration... :)

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

I'm lovin this discussion.

a couple notes:

1) IF you go the full monty and create an isp oriented qdisc, for
gawd's sake come up with a googleable name.
Things like pie, cake, bobbie, tart are good codenames, fq_codel
horrific, "streamboost" is a canonical example of a great name. At the
moment I'm rather liking what can be done with ebpf and the new tc
priority stuff and for all I know a veth + cake per subscriber works
and thus not feeling the need to rework cake itself. But keep
brainstorming and gathering requirements! It would be good to talk at
a WISP convention, and to folks like wlan-sloviena, guifi and freifunk
as to the problems they have currently.

2) per host allocations based on macaddr rather than IP works on many
network types, and is cheaper than hashing, and essentially required
if you

3) It would be good to look into offloadable hw in general at this
point, like bigfoot, cavium, mellonox, etc. It's not actually sender
side processing that worries me all that much, it's the rx path.

4) anybody for timer wheels?

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Thu, Jul 26, 2018 at 11:07 AM Dan Siemon <dan@coverfire.com> wrote:
>
> On Thu, 2018-07-26 at 08:48 -0700, Dave Taht wrote:
> > On Thu, Jul 26, 2018 at 8:46 AM Dan Siemon <dan@coverfire.com> wrote:
> > >
> > > Tiny bit of self promotion here but Preseem (
> > > https://www.preseem.com)
> > > is a transparent bridge that leverages HTB/FQ-CoDel to make
> > > subscriber plan enforcement provide much better QoE. Leaving
> > > enforcement up to the deep queues in most network equipment has
> > > comparably very bad results. We focus on WISPs but we have
> > > customers that provide service via DSL and cable as well.
> >
> > AWESOME. I am curious as to how high you can scale currently?
> > (subscribers, bandwidths)
>
> Since our boxes are low cost, we tend towards slightly more distributed
> deployments with smaller boxes.
>
> However, a i7-7700 w/ X710 NICs can do about 4Gb/sec with 4000
> subscribers with ~25% CPU usage.
>

(I'm really behind on email)

I'd like to know what the same level processor can do (rx) with
higher end cards than that. Toke was getting > 50gbit outbound with
a single cake instance.

That's also pretty low end. On the high end nowadays there's stuff like this:

https://www.amazon.com/Intel-Xeon-E5-2698-Hexadeca-core-Processor/dp/B00PDD1QES

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> On 28 Jul, 2018, at 6:51 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> That's also pretty low end. On the high end nowadays there's stuff like this:
> 
> https://www.amazon.com/Intel-Xeon-E5-2698-Hexadeca-core-Processor/dp/B00PDD1QES

Intel is no longer high-end for x86 CPUs.  Not all of the market has realised that yet, but it's true.

Look at Threadripper 2 which scales up to 32 cores, 64 threads in a single socket at HEDT prices, and EPYC which just goes bonkers in terms of I/O capabilities and still costs less than its nearest Intel competitor.  None of which has any serious concerns with the recent series of Meltdown/Spectre speculation bugs, unlike Intel.

AMD is moving to a 7nm silicon process which apparently works pretty well already, and is theoretically on par with Intel's 10nm process which they still haven't got working reliably after how many years of delays now?  And they're already beating Intel over the head with a 14nm process which is theoretically *inferior* to Intel's 14nm process, which they'll be stuck with in practice for *at least* the next year even by their own wildly optimistic latest estimates.

The only place Intel temporarily holds a real advantage is in maximum single-threaded turbo clock speed.  This is relevant to a shrinking minority of users these days.  AMD's next CPUs are supposed to make that wholly irrelevant with a significant further jump in IPC - because they were designed to compete with 10nm Intel CPUs that Intel now looks very unlikely to be capable of manufacturing.

And is this relevant to "Super Mega Turbo Cake Edition XLRi"?  Well, one of the nice things about having lots of users is that you can statistically multiplex them across multiple hardware queues more easily.  Each subscriber's traffic can sanely end up on the same queue each time, and each queue can have a separate "Super Cake" instance allocated an even division of the total backhaul bandwidth, and in theory each of *those* can run on its own CPU core.  Instant throughput boost.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> On 28 Jul, 2018, at 6:04 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> for gawd's sake come up with a googleable name.
> Things like pie, cake, bobbie, tart are good codenames, fq_codel
> horrific, "streamboost" is a canonical example of a great name.

Suggestions on a postcard.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Sat, Jul 28, 2018 at 9:11 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 28 Jul, 2018, at 6:51 pm, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > That's also pretty low end. On the high end nowadays there's stuff like this:
> >
> > https://www.amazon.com/Intel-Xeon-E5-2698-Hexadeca-core-Processor/dp/B00PDD1QES
>
> Intel is no longer high-end for x86 CPUs.  Not all of the market has realised that yet, but it's true.
>
> Look at Threadripper 2 which scales up to 32 cores, 64 threads in a single socket at HEDT prices, and EPYC which just goes bonkers in terms of I/O capabilities and still costs less than its nearest Intel competitor.  None of which has any serious concerns with the recent series of Meltdown/Spectre speculation bugs, unlike Intel.

That haunts me.

> AMD is moving to a 7nm silicon process which apparently works pretty well already, and is theoretically on par with Intel's 10nm process which they still haven't got working reliably after how many years of delays now?  And they're already beating Intel over the head with a 14nm process which is theoretically *inferior* to Intel's 14nm process, which they'll be stuck with in practice for *at least* the next year even by their own wildly optimistic latest estimates.
>
> The only place Intel temporarily holds a real advantage is in maximum single-threaded turbo clock speed.

Which is important. Myself, I dream of cpus that can context switch in
5 cycles like the mill. For I/O
heavy workloads it's context switching that is rather important. There
is also a new generation of
multi-core arms, some with devkits like netronome's and caviums.

My take on all this would be to try and benchmark some stuff. Some
other bottleneck will surface. We have one benchmark of 4000 users at
4GigE to work with at the moment.

I've long pointed out that one big reason for GRO/GSO is the cost of a
routing lookup. mellonox, I think(?), *finally* added TCAM support to
their cards in  a patch that just went by on the netdev mailing list.

> This is relevant to a shrinking minority of users these days.  AMD's next CPUs are supposed to make that wholly irrelevant with a significant further jump in IPC - because they were designed to compete with 10nm Intel CPUs that Intel now looks very unlikely to be capable of manufacturing.

One huge advantage intel had was dma into cache. Still?

>
> And is this relevant to "Super Mega Turbo Cake Edition XLRi"?

Hahahaha. That comes up with 7000 hits for a hair dryer.

https://www.google.com/search?q=Super+Mega+Turbo+Cake+Edition+XLRi&rlz=1C5CHFA_enUS749US749&oq=Super+Mega+Turbo+Cake+Edition+XLRi&aqs=chrome..69i57.381j0j4&sourceid=chrome&ie=UTF-8

> Well, one of the nice things about having lots of users is that you can statistically multiplex them across multiple hardware queues more easily.  Each subscriber's traffic can sanely end up on the same queue each time, and each queue can have a separate "Super Cake" instance allocated an even division of the total backhaul bandwidth, and in theory each of *those* can run on its own CPU core.  Instant throughput boost.

I'd believe it when I benchmarked it, not before.

>
>  - Jonathan Morton
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Sat, Jul 28, 2018 at 9:19 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 28 Jul, 2018, at 6:04 pm, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > for gawd's sake come up with a googleable name.
> > Things like pie, cake, bobbie, tart are good codenames, fq_codel
> > horrific, "streamboost" is a canonical example of a great name.
>
> Suggestions on a postcard.

I was going to send a dollar to schenactady but it came back address unknown.

https://bookviewcafe.com/blog/2010/04/02/schenectady-or-where-do-you-get-your-ideas/

BTW, "tart" stood for "tri-color aware rate translator", and was the
fq-ish component of "bobbie", where bobbie was the aqm-ish side.


>  - Jonathan Morton
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2941 bytes --]

> On Jul 28, 2018, at 10:06 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> This sounds like a relatively complex network topology, in which there are a lot of different potential bottlenecks, depending on the dynamic state of the network.

It is, which is the argument from those who want a more centralized approach. Backhaul link types include 5 GHz, 10 GHz, 60 GHz, licensed spectrum (apparently), and some fiber. Side note: I’ve heard that aligning the 60 GHz p2p links is a chore.

> Okay, so straight away we're in a significantly different regime from CoverFire's situation, where the subscribers each have a defined link bandwidth (which may or may not be related to the capabilities of the underlying physical link).  You simply need to share some backhaul link fairly between subscribers using it at any given moment.

Exactly. Many members, including myself, are limited by our CPE links during off hours, and by the backhaul during high traffic hours.

> Let's call this one a vote for "diffserv not required", since DRR++ copes well by prioritising sparse traffic.

Yep, I don’t know if you’ll hear “diffserv required” for an ISP, but we’ll see.

> Could you convert that DB to eBPF rules?  This would let you use the same configuration interface as CoverFire's situation.

That should be no problem at all.

> In general, I think the make-wifi-fast team has a better handle on the subtleties of half-duplex links.  If there's any way to get their work into the airOS devices, so much the better.

I wish that were the case, but am not holding my breath. As things stand now, I may still use HTB then, which is ok. The key is not just the dynamic rates, but also, it appears, having egress and ingress through a common IFB that’s split into sub-queues for egress and ingress. I’ll punt on WiFi for now and come back to it later.

> But I think there's also a use for "ISP-type" Cake in your network, especially in the wired backhauls where link bandwidth is relatively predictable, and I suspect most of these will be in the core parts of your network where the traffic is most complex.  As long as you can get around to running a suitable kernel version, there should also be no inherent problem with replicating the same dynamic adjustment of global shaper rate as "normal" Cake has, which will allow it to be used on some of your wireless links as well.

Yes, the biggest benefit for us would probably be “member fairness” instead of “IP fairness”. One of the long-time admins was excited that this might be possible. Also, I’ll be interested to see if improving queue management gets us better utilization of the Internet link. If so, we may want fairness at the gateway, which may be a job more for an ISP Cake given the number of flows.

Lastly, if ISP Cake could support SMP, that would be all the better for the dual and quad core APU backhaul routers. :)


[-- Attachment #2: Type: text/html, Size: 8441 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]


> On Jul 28, 2018, at 5:04 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> I'm lovin this discussion.
> 
> a couple notes:
> 
> 1) IF you go the full monty and create an isp oriented qdisc, for
> gawd's sake come up with a googleable name.
> Things like pie, cake, bobbie, tart are good codenames, fq_codel
> horrific, "streamboost" is a canonical example of a great name.

If it’s separate from Cake but still has shared heritage, just throwing "Icing" out there:

ISP Centered Insanity Nullifying Goodness

The acronym could be better. :)

> At the
> moment I'm rather liking what can be done with ebpf and the new tc
> priority stuff and for all I know a veth + cake per subscriber works
> and thus not feeling the need to rework cake itself. But keep
> brainstorming and gathering requirements! It would be good to talk at
> a WISP convention, and to folks like wlan-sloviena, guifi and freifunk
> as to the problems they have currently.

FreeNet Liberec admins would likely have more input. I should be at an admins meeting in a month or so and see what input I can get. There’s also a very similar but larger network in Prague, here in WISP-land (https://spoje.net/internet/czfree/ <http://www.czf-praha.net/>). Enjoy this:

http://mapa.czfree.net/#lat=50.087803&lng=14.406480999999985&zoom=11&tilt=0&heading=0&autofilter=1&type=satellite&aponly=1&bbonly=1&actlink=1&actnode=1& <http://mapa.czfree.net/#lat=50.087803&lng=14.406480999999985&zoom=11&tilt=0&heading=0&autofilter=1&type=satellite&aponly=1&bbonly=1&actlink=1&actnode=1&>

> 4) anybody for timer wheels?

Just saying, could that be prototyped on one of those FPGA-powered NICs?

[-- Attachment #2: Type: text/html, Size: 2749 bytes --]

[Cake] isp economics

[Dave Taht]

On Sat, Jul 28, 2018 at 9:41 AM Pete Heist <pete@heistp.net> wrote:
>
> On Jul 28, 2018, at 10:06 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> This sounds like a relatively complex network topology, in which there are a lot of different potential bottlenecks, depending on the dynamic state of the network.
>
>
> It is, which is the argument from those who want a more centralized approach. Backhaul link types include 5 GHz, 10 GHz, 60 GHz, licensed spectrum (apparently), and some fiber. Side note: I’ve heard that aligning the 60 GHz p2p links is a chore.
>
> Okay, so straight away we're in a significantly different regime from CoverFire's situation, where the subscribers each have a defined link bandwidth (which may or may not be related to the capabilities of the underlying physical link).  You simply need to share some backhaul link fairly between subscribers using it at any given moment.
>
>
> Exactly. Many members, including myself, are limited by our CPE links during off hours, and by the backhaul during high traffic hours.

3 items

1) Co-locating some essential services (like netflix) might be of help.

https://openconnect.netflix.com/en/

2) If the members voted for more backhaul, the costs are understandable...

3) Philosophically I vastly prefer the concept of "everyone sharing
the network", rather than rate plans, to create some market tension as
to the available bandwidth at any given time of day. An extreme
viewpoint was that you'd bill for bandwidth more during times of the
day when it was more valuable, and less when it was unused.

Instead we get lying rate "up to X" plans that force "building
overcapacity because thats what I bought", and arbitrary ratios of
10x1 or worse overselling actual capacity to subscribers (which got
blown up by the rise of streaming)

However, if everyone always shared equally in the fate of the network,
everyone has an incentive to make it as a good as possible, and in
off-hours anybody could get more than "X", if available.

Back in the good ole days you'd find me in the lab at 2am because "the
network was better then". Incentivtising off peak use is what torrent
did. Backups were also frequently done then. etc.

> Let's call this one a vote for "diffserv not required", since DRR++ copes well by prioritising sparse traffic.
>
>
> Yep, I don’t know if you’ll hear “diffserv required” for an ISP, but we’ll see.
>
> Could you convert that DB to eBPF rules?  This would let you use the same configuration interface as CoverFire's situation.
>
>
> That should be no problem at all.
>
> In general, I think the make-wifi-fast team has a better handle on the subtleties of half-duplex links.  If there's any way to get their work into the airOS devices, so much the better.

ubnt did add at least airtime fairness to airos a year or two back. It
was not on by default. Their "TDMA" qos system was this insane mess of
sfq rules when I tore it apart.... 8 years back.

> I wish that were the case, but am not holding my breath. As things stand now, I may still use HTB then, which is ok.

>The key is not just the dynamic rates, but also, it appears, having egress and ingress through a common IFB that’s split into sub-queues for egress and ingress. I’ll punt on WiFi for now and come back to it later.
>
> But I think there's also a use for "ISP-type" Cake in your network, especially in the wired backhauls where link bandwidth is relatively predictable, and I suspect most of these will be in the core parts of your network where the traffic is most complex.  As long as you can get around to running a suitable kernel version, there should also be no inherent problem with replicating the same dynamic adjustment of global shaper rate as "normal" Cake has, which will allow it to be used on some of your wireless links as well.
>
>
> Yes, the biggest benefit for us would probably be “member fairness” instead of “IP fairness”. One of the long-time admins was excited that this might be possible. Also, I’ll be interested to see if improving queue management gets us better utilization of the Internet link. If so, we may want fairness at the gateway, which may be a job more for an ISP Cake given the number of flows.
>
> Lastly, if ISP Cake could support SMP, that would be all the better for the dual and quad core APU backhaul routers. :)
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]


> On Jul 28, 2018, at 10:56 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Note that with the existing tc classifier stuff we already added to
> Cake, we basically have this already (eBPF can map traffic to tin and
> flow however it pleases).

Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html <https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html>

George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html <https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html>

How does one send packets to a specific flow / queue?

This wouldn’t give both per-member and per-flow fairness, but at least per-member fairness might be possible. There are 1024(?) queues available and 800 members, so I’m just speculating that I could map members to a number from 0 to 800 (active member IDs packed and zero-based would work) and assign each member to their own flow. Thanks... :)


[-- Attachment #2: Type: text/html, Size: 1763 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

On Sat, Jul 28, 2018 at 10:38 AM Pete Heist <pete@heistp.net> wrote:
>
>
> On Jul 28, 2018, at 10:56 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Note that with the existing tc classifier stuff we already added to
> Cake, we basically have this already (eBPF can map traffic to tin and
> flow however it pleases).
>
>
> Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
>
> George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
>
> How does one send packets to a specific flow / queue?

It's essentially above. I think you can actually do it in pure bpf
without skbedit, I'd written a tc bpf flow classifier for acks quite
some time ago. The not current version is attached. I really need to
finish up some ack related stuff.
Using a bpf map to this then setting the flowid directly?


> This wouldn’t give both per-member and per-flow fairness, but at least per-member fairness might be possible. There are 1024(?) queues available and 800 members, so I’m just speculating that I could map members to a number from 0 to 800 (active member IDs packed and zero-based would work) and assign each member to their own flow. Thanks... :)
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


--

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #2: ack_recognise.c --]
[-- Type: application/octet-stream, Size: 2164 bytes --]

/* ack_recogniser: An eBPF program that correctly recognises modern TCP ACKs,
   with tcp option fields.
 */

#include "bpf_api.h"
#include "uapi/linux/if_ether.h"
#include "uapi/linux/ip.h"
#include "uapi/linux/in.h"
// #include "uapi/linux/ipv6.h" // iproute2 doesn't import this header
#include "uapi/linux/tcp.h"

/* Idealized syntax example to move acks into a priority queue:

tc qdisc del dev $IFACE root 2> /dev/null
tc qdisc add dev $IFACE root handle 1: prio bands 3 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
tc qdisc add dev $IFACE parent 1:1 handle 10:1 fq_codel
tc qdisc add dev $IFACE parent 1:2 handle 20:1 fq_codel
tc qdisc add dev $IFACE parent 1:3 handle 30:1 fq_codel

tc filter add dev $IFACE  prio 1 bpf object-file ack_recognise.o da ack_match flowid 10:1 # 1:1?

*/

/* A pure ack contains the ip header, the tcp header + options, flags with the
 * ack field set, and no additional payload. That last bit is what every prior
 * ack filter gets wrong, they typically assume an obsolete 64 bytes.
 */

__section_cls_entry
int ack_match(struct __sk_buff *skb)
{
	void *data = (void *)(long)skb->data;
	void *data_end = (void *)(long)skb->data_end;
	struct ethhdr *eth = data;
	struct iphdr *iph = data + sizeof(*eth);
	struct tcphdr *tcp;
	int len, isize;

	if (data + sizeof(*eth) + sizeof(*iph) + sizeof(*tcp) > data_end)
		return 0;

	if (eth->h_proto != htons(ETH_P_IP))
		return 0;

	if (iph->version == 4) {
		if(iph->protocol != IPPROTO_TCP)
			return 0;
		len = iph->tot_len; // htons?
		if(iph->ihl > 4) {
			isize = (iph->ihl+1) * 4 - sizeof(*iph);
			tcp = data + sizeof(*eth) + isize;
		} else {
			return 0;
		}
/* grump: iproute2 does not export ip6hdrs
	} else  if (iph->version == 6) {
		struct ipv6hdr iph6 = (struct ipv6hdr *) iph;
// FIXME: walk ip6hdrs appropriately
		if(iph6->nexthdr != IPPROTO_TCP)
			return 0;
		len = iph6->payload_len;
		isize = sizeof(*iph6) + ??;
		tcp = data + sizeof(*eth) + isize;
		*/
	} else
		return 0;

	if (!tcp->ack) return 0;

	if (isize + sizeof(*tcp) + tcp->doff*4 == len)
		return -1; // We matched, return TC_ACT_RECLASSIFY?

	return 0;
}

char __license[] __section("license") = "GPL";

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

>> Note that with the existing tc classifier stuff we already added to
>> Cake, we basically have this already (eBPF can map traffic to tin and
>> flow however it pleases).
> 
> Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
> 
> George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
> 
> How does one send packets to a specific flow / queue?

The trouble here is that there's only 8 tins max in Cake.  At that level selection is done with a linear search, which doesn't scale up, but is efficient for N=8.  The host/flow mapping is hardcoded for speed with no override hook, because no consumer needs custom mapping of this sort.

Fixing these problems to make them more ISP-friendly necessarily makes it less consumer-friendly.  Hence the new project.  Much code can be reused.

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

https://github.com/iovisor/bcc/blob/master/src/cc/compat/linux/bpf.h#L2222
says you can get at the priority field.

On Sat, Jul 28, 2018 at 10:52 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sat, Jul 28, 2018 at 10:38 AM Pete Heist <pete@heistp.net> wrote:
> >
> >
> > On Jul 28, 2018, at 10:56 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >
> > Note that with the existing tc classifier stuff we already added to
> > Cake, we basically have this already (eBPF can map traffic to tin and
> > flow however it pleases).
> >
> >
> > Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
> >
> > George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
> >
> > How does one send packets to a specific flow / queue?
>
> It's essentially above. I think you can actually do it in pure bpf
> without skbedit, I'd written a tc bpf flow classifier for acks quite
> some time ago. The not current version is attached. I really need to
> finish up some ack related stuff.
> Using a bpf map to this then setting the flowid directly?
>
>
> > This wouldn’t give both per-member and per-flow fairness, but at least per-member fairness might be possible. There are 1024(?) queues available and 800 members, so I’m just speculating that I could map members to a number from 0 to 800 (active member IDs packed and zero-based would work) and assign each member to their own flow. Thanks... :)
> >
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Sat, Jul 28, 2018 at 10:54 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> >> Note that with the existing tc classifier stuff we already added to
> >> Cake, we basically have this already (eBPF can map traffic to tin and
> >> flow however it pleases).
> >
> > Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
> >
> > George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
> >
> > How does one send packets to a specific flow / queue?
>
> The trouble here is that there's only 8 tins max in Cake.  At that level selection is done with a linear search, which doesn't scale up, but is efficient for N=8.  The host/flow mapping is hardcoded for speed with no override hook, because no consumer needs custom mapping of this sort.

Ah. Sigh. I forgot that we only did tins, not queues (which is fraught
with peril). Do think that initial classifier could be entirely
bpf....

> Fixing these problems to make them more ISP-friendly necessarily makes it less consumer-friendly.  Hence the new project.  Much code can be reused.

It can be modeled with fq_codel for basic per-member fairness via the
prio field. Or drr. Or htb.

Meh, I'm leaning now towards a new qdisc (after benching the veth
approach which is more multicore, as cake is not multicore enough
presently)

here's an idea for qdisc name: isp. Come up with your own backronym.
integral sane provisioning.

>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Priority field sets tin, class sets flow. Both need the qdisc is as its major number, iirc. And both can be set from the same bpf filter which can be run in direct action mode...

-Toke

On 28 July 2018 19:56:35 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>https://github.com/iovisor/bcc/blob/master/src/cc/compat/linux/bpf.h#L2222
>says you can get at the priority field.
>
>On Sat, Jul 28, 2018 at 10:52 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Sat, Jul 28, 2018 at 10:38 AM Pete Heist <pete@heistp.net> wrote:
>> >
>> >
>> > On Jul 28, 2018, at 10:56 AM, Toke Høiland-Jørgensen <toke@toke.dk>
>wrote:
>> >
>> > Note that with the existing tc classifier stuff we already added to
>> > Cake, we basically have this already (eBPF can map traffic to tin
>and
>> > flow however it pleases).
>> >
>> >
>> > Sorry, this just jostled in my brain now that I may be able to
>implement member fairness today, based on what you wrote earlier in a
>thread that I entirely missed:
>https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
>> >
>> > George posted an example of assigning packets to a tin:
>https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
>> >
>> > How does one send packets to a specific flow / queue?
>>
>> It's essentially above. I think you can actually do it in pure bpf
>> without skbedit, I'd written a tc bpf flow classifier for acks quite
>> some time ago. The not current version is attached. I really need to
>> finish up some ack related stuff.
>> Using a bpf map to this then setting the flowid directly?
>>
>>
>> > This wouldn’t give both per-member and per-flow fairness, but at
>least per-member fairness might be possible. There are 1024(?) queues
>available and 800 members, so I’m just speculating that I could map
>members to a number from 0 to 800 (active member IDs packed and
>zero-based would work) and assign each member to their own flow.
>Thanks... :)
>> >
>> > _______________________________________________
>> > Cake mailing list
>> > Cake@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cake
>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

On 28 July 2018 19:53:58 CEST, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> Note that with the existing tc classifier stuff we already added to
>>> Cake, we basically have this already (eBPF can map traffic to tin
>and
>>> flow however it pleases).
>> 
>> Sorry, this just jostled in my brain now that I may be able to
>implement member fairness today, based on what you wrote earlier in a
>thread that I entirely missed:
>https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
>> 
>> George posted an example of assigning packets to a tin:
>https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
>> 
>> How does one send packets to a specific flow / queue?
>
>The trouble here is that there's only 8 tins max in Cake.  At that
>level selection is done with a linear search, which doesn't scale up,
>but is efficient for N=8.  

Yeah, but replacing that with an rbtree should be straight forward.

> The flow mapping is hardcoded for speed
>with no override hook, because no consumer needs custom mapping of this
>sort.

Getting this to work is probably the most work, actually. I guess the 'tc class' config would be the obvious way to do express this, API-wise.


>Fixing these problems to make them more ISP-friendly necessarily makes
>it less consumer-friendly.  Hence the new project.  Much code can be
>reused.

If more features are needed, perhaps... But for just adding more classes I don't actually think it has to impact the UX for the current use cases. The existing keywords could be retained and map to the same configs.

-Toke

Re: [Cake] isp economics

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2365 bytes --]


> On Jul 28, 2018, at 7:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> Exactly. Many members, including myself, are limited by our CPE links during off hours, and by the backhaul during high traffic hours.
> 
> 3 items
> 
> 1) Co-locating some essential services (like netflix) might be of help.

That’s a good idea, will bring that up with the admins.

Netflix is here now, but many folks seem to use “O2 TV Air”, which is a way to watch ordinary cable packages over any Internet connection (not just from O2). SD streams are 3Mbit and HD 6Mbit. Weirdly, they don’t pace out their data but there’s a characteristic “pulsing” of the streams every 5 seconds that when I see it on the router's throughput graph I know, yep, that’s O2 TV. I’ll find out if their on-demand stuff has such a co-location option.

> https://openconnect.netflix.com/en/
> 
> 2) If the members voted for more backhaul, the costs are understandable…

Backhaul upgrades are coming into focus gradually, and upgrading those ALIX boxes which are tanks but...tanks.

I think some of the higher costs come from physical installations / placement rights. Surprising what towers can cost.

> 3) Philosophically I vastly prefer the concept of "everyone sharing
> the network", rather than rate plans, to create some market tension as
> to the available bandwidth at any given time of day.

Agree wholeheartedly. Rate limiting members isn’t necessary for a non-profit (no profit motive) and especially if we can get fairness right, there’s no technical reason to do it that I know of either.

> ubnt did add at least airtime fairness to airos a year or two back. It
> was not on by default. Their "TDMA" qos system was this insane mess of
> sfq rules when I tore it apart.... 8 years back.

I don’t know what they’re doing in their newer AC stuff, but I'm surprised by what appears to be ~6-8ms latency under load on the NanoStation 5 AC Loco’s I got for the camp’s backhaul. Is it really that good? This is in contrast to the 50+ms I see with rrul_be on the NanoStation M5 (without controlling the queue). This test is straight AP to AP though, with probably 1 flow up and 1 down plus ping, so I want to get 2-4 more of these and do rrul_be through the Ethernet ports, to get more flows and UDP, and see how it looks then.




[-- Attachment #2.1: Type: text/html, Size: 3652 bytes --]

[-- Attachment #2.2: ns5ac_loco.png --]
[-- Type: image/png, Size: 25819 bytes --]

Re: [Cake] isp economics

[Dave Taht]

[-- Attachment #1.1.1: Type: text/plain, Size: 3605 bytes --]

On Sat, Jul 28, 2018 at 11:39 AM Pete Heist <pete@heistp.net> wrote:

>
> On Jul 28, 2018, at 7:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>
> Exactly. Many members, including myself, are limited by our CPE links
> during off hours, and by the backhaul during high traffic hours.
>
>
> 3 items
>
> 1) Co-locating some essential services (like netflix) might be of help.
>
>
> That’s a good idea, will bring that up with the admins.
>
> Netflix is here now, but many folks seem to use “O2 TV Air”, which is a
> way to watch ordinary cable packages over any Internet connection (not just
> from O2). SD streams are 3Mbit and HD 6Mbit. Weirdly, they don’t pace out
> their data but there’s a characteristic “pulsing” of the streams every 5
> seconds that when I see it on the router's throughput graph I know, yep,
> that’s O2 TV. I’ll find out if their on-demand stuff has such a co-location
> option.
>
> https://openconnect.netflix.com/en/
>
> 2) If the members voted for more backhaul, the costs are understandable…
>
>
> Backhaul upgrades are coming into focus gradually, and upgrading those
> ALIX boxes which are tanks but...tanks.
>

apu2s are thus far being tanks for me. wndr3800s with multi-year cerowrt
uptimes now exist, too.


>
> I think some of the higher costs come from physical installations /
> placement rights. Surprising what towers can cost.
>
> 3) Philosophically I vastly prefer the concept of "everyone sharing
> the network", rather than rate plans, to create some market tension as
> to the available bandwidth at any given time of day.
>
>
> Agree wholeheartedly. Rate limiting members isn’t necessary for a
> non-profit (no profit motive) and especially if we can get fairness right,
> there’s no technical reason to do it that I know of either.
>
>
It would be a better world if more isps served their users and didn't buy
film studios.


> ubnt did add at least airtime fairness to airos a year or two back. It
> was not on by default. Their "TDMA" qos system was this insane mess of
> sfq rules when I tore it apart.... 8 years back.
>
>
> I don’t know what they’re doing in their newer AC stuff, but I'm surprised
> by what appears to be ~6-8ms latency
>

Something fq_codel-like is now in qualcomms proprietary firmware I'm told.
No ecn


> under load on the NanoStation 5 AC Loco’s I got for the camp’s backhaul.
> Is it really that good? This is in contrast to the 50+ms I see with rrul_be
> on the NanoStation M5 (without controlling the queue).
>

ubnt both cases? doubt it's the same bandwidth both cases. should be
proportional to the latency you are seeing. running 2x2 incurs a latency
penalty also.


> This test is straight AP to AP though, with probably 1 flow up and 1 down
> plus ping, so I want to get 2-4 more of these and do rrul_be through the
> Ethernet ports, to get more flows and UDP, and see how it looks then.
>

Run more flows. SFQ is per packet fq. They have right-sized buffers when
the link is running at close to the configured rate, not when it's
stuggling.

I also seem to remember they reduced the txop to ~2ms. turned off 802.11e.
I've recommended this for years now in the general case.

Their 100mbit ethernet devices also do flow control and are more often the
bottleneck than not, so the wifi runs empty more often.

I LIKE their gear. Despite all we've done here there are still things they
do better.


>
>
>

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #1.1.2: Type: text/html, Size: 6031 bytes --]

[-- Attachment #1.2: ns5ac_loco.png --]
[-- Type: image/png, Size: 25819 bytes --]

[-- Attachment #2: ns5ac_loco.png --]
[-- Type: image/png, Size: 25819 bytes --]

Re: [Cake] isp economics

[Dave Taht]

On Sat, Jul 28, 2018 at 11:39 AM Pete Heist <pete@heistp.net> wrote:
>
>
> On Jul 28, 2018, at 7:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>
> Exactly. Many members, including myself, are limited by our CPE links during off hours, and by the backhaul during high traffic hours.
>
>
> 3 items
>
> 1) Co-locating some essential services (like netflix) might be of help.
>
>
> That’s a good idea, will bring that up with the admins.
>
> Netflix is here now, but many folks seem to use “O2 TV Air”, which is a way to watch ordinary cable packages over any Internet connection (not just from O2). SD streams are 3Mbit and HD 6Mbit. Weirdly, they don’t pace out their data but there’s a characteristic “pulsing” of the streams every 5 seconds that when I see it on the router's throughput graph I know, yep, that’s O2 TV. I’ll find out if their on-demand stuff has such a co-location option.

thats basically how DASH traffic works. The video client probes for
extra bandwidth every 10 seconds. If it succeeds it switches over to
the higher rate flow.

this was good: https://reproducingnetworkresearch.wordpress.com/2017/06/05/cs244-17-confused-timid-and-unstable-picking-a-video-streaming-rate-is-hard/

bbr exists because of this behavior also.

> https://openconnect.netflix.com/en/
>
> 2) If the members voted for more backhaul, the costs are understandable…
>
>
> Backhaul upgrades are coming into focus gradually, and upgrading those ALIX boxes which are tanks but...tanks.
>
> I think some of the higher costs come from physical installations / placement rights. Surprising what towers can cost.
>
> 3) Philosophically I vastly prefer the concept of "everyone sharing
> the network", rather than rate plans, to create some market tension as
> to the available bandwidth at any given time of day.
>
>
> Agree wholeheartedly. Rate limiting members isn’t necessary for a non-profit (no profit motive) and especially if we can get fairness right, there’s no technical reason to do it that I know of either.
>
> ubnt did add at least airtime fairness to airos a year or two back. It
> was not on by default. Their "TDMA" qos system was this insane mess of
> sfq rules when I tore it apart.... 8 years back.
>
>
> I don’t know what they’re doing in their newer AC stuff, but I'm surprised by what appears to be ~6-8ms latency under load on the NanoStation 5 AC Loco’s I got for the camp’s backhaul. Is it really that good? This is in contrast to the 50+ms I see with rrul_be on the NanoStation M5 (without controlling the queue). This test is straight AP to AP though, with probably 1 flow up and 1 down plus ping, so I want to get 2-4 more of these and do rrul_be through the Ethernet ports, to get more flows and UDP, and see how it looks then.
>
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[Cake] 1000s of users

[Dave Taht]

If I get time, what I might try is:

one veth per user with cake bandwidth whatever
routing (no bpf) the ip address subset to each veth
10x1 oversubscription

bridging all those veths together into one interface and... just applying pie
or codel to that 'cause there are just too many flows to conceive of
fq-ing further here, and each veth has already applied a lot of
entropy. I DO have a 10gige device, so a variety of subscribers at
20,40,100mbit could be attempted to fit. Still, 1000 subscribers at
20mbit

Re: [Cake] isp economics

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]


> On Jul 28, 2018, at 9:03 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> under load on the NanoStation 5 AC Loco’s I got for the camp’s backhaul. Is it really that good? This is in contrast to the 50+ms I see with rrul_be on the NanoStation M5 (without controlling the queue). 
> 
> ubnt both cases? doubt it's the same bandwidth both cases. should be proportional to the latency you are seeing. running 2x2 incurs a latency penalty also.

Higher bandwidth on the newer AC Loco vs the M5 but not a huge difference (120Mbit one-way vs 90Mbit one-way). Both 20 MHz channels. The AC Loco defaults to 80 MHz channels, which is excessive for this application.
 
> This test is straight AP to AP though, with probably 1 flow up and 1 down plus ping, so I want to get 2-4 more of these and do rrul_be through the Ethernet ports, to get more flows and UDP, and see how it looks then.
>  
> Run more flows. SFQ is per packet fq. They have right-sized buffers when the link is running at close to the configured rate, not when it's stuggling.

I think that’s the primary reason (the way ubnt does their test), unfortunately don’t have two free to test at the moment.

> I also seem to remember they reduced the txop to ~2ms. turned off 802.11e. I've recommended this for years now in the general case. 

Re 802.11e, what’s interesting is when you run ’athstats’ on the NSM5 (older), there’s a breakdown of BK, BE, VI and VO packet stats. On the AC Loco (newer), there’s not. That does imply, but doesn’t prove, that they don’t use these queues, at least for point-to-point connections. I think they have to have 802.11e to be compliant, but I don’t know if they're mapping everything to one queue or not.

It looks like airMAX is now _not_ used at all for point-to-point connections, whereas it used to be on their older gear. This is good, as my testing showed airMAX only adds a bit of inter-flow latency for point-to-point.

> Their 100mbit ethernet devices also do flow control and are more often the bottleneck than not, so the wifi runs empty more often. 

The AC Loco has Gbit Ethernet thankfully. Looks to me like all of their AC gear does now.

Sorry for diverging too far from the ISP topic, and on the Cake list…


[-- Attachment #2: Type: text/html, Size: 5308 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 28, 2018, at 8:12 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Priority field sets tin, class sets flow. Both need the qdisc is as its major number, iirc. And both can be set from the same bpf filter which can be run in direct action mode...

This works for me. :)

I only tested so far by setting classid with u32 to map 3 macs to two members IDs, then verified fairness:

# IFACE is cake’s interface
# MAJOR_ID is from cake
while read mac id; do
        tc filter add dev $IFACE parent $MAJOR_ID: \
                u32 match ether src $mac classid $MAJOR_ID:$id
done << EOF
aa:bb:cc:11:22:33 1
bb:cc:dd:22:33:44 1
cc:dd:ee:33:44:55 2
EOF

This should use eBPF and a map lookup for performance, so I’ll see if I can make that work.

Caveats that I know of:
- Limited to 1024 members
- No fairness between flows
- Non-member traffic would have to be dealt with somehow, maybe put in its own queue or split among multiple queues, otherwise there can be hash collisions with member queues

Re: [Cake] isp economics

[Loganaden Velvindron]

[-- Attachment #1: Type: text/plain, Size: 4040 bytes --]

On Sat, Jul 28, 2018 at 11:03 PM, Dave Taht <dave.taht@gmail.com> wrote:

>
>
> On Sat, Jul 28, 2018 at 11:39 AM Pete Heist <pete@heistp.net> wrote:
>
>>
>> On Jul 28, 2018, at 7:32 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>>
>> Exactly. Many members, including myself, are limited by our CPE links
>> during off hours, and by the backhaul during high traffic hours.
>>
>>
>> 3 items
>>
>> 1) Co-locating some essential services (like netflix) might be of help.
>>
>>
>> That’s a good idea, will bring that up with the admins.
>>
>> Netflix is here now, but many folks seem to use “O2 TV Air”, which is a
>> way to watch ordinary cable packages over any Internet connection (not just
>> from O2). SD streams are 3Mbit and HD 6Mbit. Weirdly, they don’t pace out
>> their data but there’s a characteristic “pulsing” of the streams every 5
>> seconds that when I see it on the router's throughput graph I know, yep,
>> that’s O2 TV. I’ll find out if their on-demand stuff has such a co-location
>> option.
>>
>> https://openconnect.netflix.com/en/
>>
>> 2) If the members voted for more backhaul, the costs are understandable…
>>
>>
>> Backhaul upgrades are coming into focus gradually, and upgrading those
>> ALIX boxes which are tanks but...tanks.
>>
>
> apu2s are thus far being tanks for me. wndr3800s with multi-year cerowrt
> uptimes now exist, too.
>
>
>>
>> I think some of the higher costs come from physical installations /
>> placement rights. Surprising what towers can cost.
>>
>> 3) Philosophically I vastly prefer the concept of "everyone sharing
>> the network", rather than rate plans, to create some market tension as
>> to the available bandwidth at any given time of day.
>>
>>
>> Agree wholeheartedly. Rate limiting members isn’t necessary for a
>> non-profit (no profit motive) and especially if we can get fairness right,
>> there’s no technical reason to do it that I know of either.
>>
>>
> It would be a better world if more isps served their users and didn't buy
> film studios.
>
>
>> ubnt did add at least airtime fairness to airos a year or two back. It
>> was not on by default. Their "TDMA" qos system was this insane mess of
>> sfq rules when I tore it apart.... 8 years back.
>>
>>
>> I don’t know what they’re doing in their newer AC stuff, but I'm
>> surprised by what appears to be ~6-8ms latency
>>
>
> Something fq_codel-like is now in qualcomms proprietary firmware I'm told.
> No ecn
>

Which qca chipset ? Which version of the firmware ?



>
>
>> under load on the NanoStation 5 AC Loco’s I got for the camp’s backhaul.
>> Is it really that good? This is in contrast to the 50+ms I see with rrul_be
>> on the NanoStation M5 (without controlling the queue).
>>
>
> ubnt both cases? doubt it's the same bandwidth both cases. should be
> proportional to the latency you are seeing. running 2x2 incurs a latency
> penalty also.
>
>
>> This test is straight AP to AP though, with probably 1 flow up and 1 down
>> plus ping, so I want to get 2-4 more of these and do rrul_be through the
>> Ethernet ports, to get more flows and UDP, and see how it looks then.
>>
>
> Run more flows. SFQ is per packet fq. They have right-sized buffers when
> the link is running at close to the configured rate, not when it's
> stuggling.
>
> I also seem to remember they reduced the txop to ~2ms. turned off 802.11e.
> I've recommended this for years now in the general case.
>
> Their 100mbit ethernet devices also do flow control and are more often the
> bottleneck than not, so the wifi runs empty more often.
>
> I LIKE their gear. Despite all we've done here there are still things they
> do better.
>
>
>>
>>
>>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 7135 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

>> On Jul 28, 2018, at 8:12 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Priority field sets tin, class sets flow. Both need the qdisc is as its major number, iirc. And both can be set from the same bpf filter which can be run in direct action mode...
>
> This works for me. :)
>
> I only tested so far by setting classid with u32 to map 3 macs to two members IDs, then verified fairness:
>
> # IFACE is cake’s interface
> # MAJOR_ID is from cake
> while read mac id; do
>         tc filter add dev $IFACE parent $MAJOR_ID: \
>                 u32 match ether src $mac classid $MAJOR_ID:$id
> done << EOF
> aa:bb:cc:11:22:33 1
> bb:cc:dd:22:33:44 1
> cc:dd:ee:33:44:55 2
> EOF
>
> This should use eBPF and a map lookup for performance, so I’ll see if I can make that work.
>
> Caveats that I know of:
> - Limited to 1024 members
> - No fairness between flows

You could assign more than one queue per customer and hash traffic
between them in BPF...

> - Non-member traffic would have to be dealt with somehow, maybe put in
> its own queue or split among multiple queues, otherwise there can be
> hash collisions with member queues

Yeah, an "overflow queue" is definitely needed in this kind of
deployment :)

I actually wrote an eBPF classifier a few months back, that can lookup
subnets in a BPF map and map them into different classes:
https://github.com/tohojo/tc-classifier

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Dave Taht]

On Sat, Jul 28, 2018 at 10:38 AM Pete Heist <pete@heistp.net> wrote:
>
>
> On Jul 28, 2018, at 10:56 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Note that with the existing tc classifier stuff we already added to
> Cake, we basically have this already (eBPF can map traffic to tin and
> flow however it pleases).
>
>
> Sorry, this just jostled in my brain now that I may be able to implement member fairness today, based on what you wrote earlier in a thread that I entirely missed: https://lists.bufferbloat.net/pipermail/cake/2018-May/003811.html
>
> George posted an example of assigning packets to a tin: https://lists.bufferbloat.net/pipermail/cake/2018-May/003809.html
>
> How does one send packets to a specific flow / queue?
>
> This wouldn’t give both per-member and per-flow fairness, but at least per-member fairness might be possible. There are 1024(?) queues available and 800 members, so I’m just speculating that I could map members to a number from 0 to 800 (active member IDs packed and zero-based would work) and assign each member to their own flow. Thanks... :)

your typical cable modem segment is x.y.z.u/22 - in other words they
only manage 1024 subscribers per segment also.

So... the birthday problem only rears its head in the real world, when
you have small values, like 4. above 32 it increasingly doesn't
matter. We did a lot of successful ns2 tests of fq_codel with 16,32
flows settings  (in the hope it would ease a hardware design). So,
instead of trying to use up one veth with 800 subscribers, use 800/per
cpu you have. Perhaps.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2720 bytes --]


> On Jul 29, 2018, at 9:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Caveats that I know of:
>> - Limited to 1024 members
>> - No fairness between flows
> 
> You could assign more than one queue per customer and hash traffic
> between them in BPF…

True. There will always be that limit of 1024 (in my case I’ll need 800).

>> - Non-member traffic would have to be dealt with somehow, maybe put in
>> its own queue or split among multiple queues, otherwise there can be
>> hash collisions with member queues
> 
> Yeah, an "overflow queue" is definitely needed in this kind of
> deployment :)

Yep, I’d like to hash the non-member flows across the remaining queues.

> I actually wrote an eBPF classifier a few months back, that can lookup
> subnets in a BPF map and map them into different classes:
> https://github.com/tohojo/tc-classifier <https://github.com/tohojo/tc-classifier>

Nice! That along with Dave’s ack classifier will help me write one for MAC addresses. I got as far as “my first no-panic bpf”, but for starters I wasn’t sure of the right way to set classid, so I see TC_H_MAKE. The documentation one finds on BPF varies a lot in correctness, so I messed around a while.

I think regardless of whether ISP cake is a new qdisc or changes to the current one, it would be good to provide a common tool like this for mapping both MACs and IP subnets. Maybe I can just expand tc-classifier a bit for my needs and try to think of others also? Here’s how it could work:

Userspace tool:
- accepts as input from stdin or file, space or comma separated mappings of one of (MAC address, IPv4 subnet or IPv6 subnet) to both classid (flow) and priority (tin), so three fields total
- accepts as an optional argument tin to place unclassified traffic in (defaults to 0)
- returns an error if no queues available for unclassified traffic
- puts mappings into up to three global BPF maps (for MAC, IPv4 and IPv6)
- puts unclassified traffic tin, if non-zero, into a global
- should lock globals here so updates can be made without removing / re-adding qdisc

BPF filter:
- tries to classify first using MAC address map, then IPv4 or IPv6 maps for IP traffic
- spreads any unclassified traffic in unclassified traffic tin across remaining classids from max(classid)+1 to 1023

Lastly, although it's natural to use classid for flow (subscriber) and priority for tin, we have a hard maximum of 2^16 subscribers in a given tin (minor classid is 16 bits). It doesn’t matter now because we only have 1024 flows per tin, but for ISP cake, is a limit of ~2^16 subscribers in one tin enough? Otherwise we’d have to change the way we specify this.


[-- Attachment #2: Type: text/html, Size: 9860 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Sebastian Moeller]

Hi Pete

> On Jul 30, 2018, at 11:14, Pete Heist <pete@heistp.net> wrote:
> 
> 
>> On Jul 29, 2018, at 9:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> Caveats that I know of:
>>> - Limited to 1024 members
>>> - No fairness between flows
>> 
>> You could assign more than one queue per customer and hash traffic
>> between them in BPF…
> 
> True. There will always be that limit of 1024 (in my case I’ll need 800).

I believe all you needto do is change the following in scg_cake.c:
#define CAKE_QUEUES (1024)

Now I heard reports that above a certain number this breaks, but it might be enough for 32K or even 64k, or ~32 to 64 queues per customer, which might already help to spread out things a bit...


Best Regards
	Sebastian


> 
>>> - Non-member traffic would have to be dealt with somehow, maybe put in
>>> its own queue or split among multiple queues, otherwise there can be
>>> hash collisions with member queues
>> 
>> Yeah, an "overflow queue" is definitely needed in this kind of
>> deployment :)
> 
> Yep, I’d like to hash the non-member flows across the remaining queues.
> 
>> I actually wrote an eBPF classifier a few months back, that can lookup
>> subnets in a BPF map and map them into different classes:
>> https://github.com/tohojo/tc-classifier
> 
> Nice! That along with Dave’s ack classifier will help me write one for MAC addresses. I got as far as “my first no-panic bpf”, but for starters I wasn’t sure of the right way to set classid, so I see TC_H_MAKE. The documentation one finds on BPF varies a lot in correctness, so I messed around a while.
> 
> I think regardless of whether ISP cake is a new qdisc or changes to the current one, it would be good to provide a common tool like this for mapping both MACs and IP subnets. Maybe I can just expand tc-classifier a bit for my needs and try to think of others also? Here’s how it could work:
> 
> Userspace tool:
> - accepts as input from stdin or file, space or comma separated mappings of one of (MAC address, IPv4 subnet or IPv6 subnet) to both classid (flow) and priority (tin), so three fields total
> - accepts as an optional argument tin to place unclassified traffic in (defaults to 0)
> - returns an error if no queues available for unclassified traffic
> - puts mappings into up to three global BPF maps (for MAC, IPv4 and IPv6)
> - puts unclassified traffic tin, if non-zero, into a global
> - should lock globals here so updates can be made without removing / re-adding qdisc
> 
> BPF filter:
> - tries to classify first using MAC address map, then IPv4 or IPv6 maps for IP traffic
> - spreads any unclassified traffic in unclassified traffic tin across remaining classids from max(classid)+1 to 1023
> 
> Lastly, although it's natural to use classid for flow (subscriber) and priority for tin, we have a hard maximum of 2^16 subscribers in a given tin (minor classid is 16 bits). It doesn’t matter now because we only have 1024 flows per tin, but for ISP cake, is a limit of ~2^16 subscribers in one tin enough? Otherwise we’d have to change the way we specify this.
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> Hi Pete
>
>> On Jul 30, 2018, at 11:14, Pete Heist <pete@heistp.net> wrote:
>> 
>> 
>>> On Jul 29, 2018, at 9:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>> 
>>>> Caveats that I know of:
>>>> - Limited to 1024 members
>>>> - No fairness between flows
>>> 
>>> You could assign more than one queue per customer and hash traffic
>>> between them in BPF…
>> 
>> True. There will always be that limit of 1024 (in my case I’ll need 800).
>
> I believe all you needto do is change the following in scg_cake.c:
> #define CAKE_QUEUES (1024)
>
> Now I heard reports that above a certain number this breaks, but it
> might be enough for 32K or even 64k, or ~32 to 64 queues per customer,
> which might already help to spread out things a bit...

Has to fit in 16 bits, so 64k is the max; we really ought to have made
that parameter configurable...

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]



> On Jul 30, 2018, at 12:09 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> You could assign more than one queue per customer and hash traffic
>>> between them in BPF…
>> 
>> True. There will always be that limit of 1024 (in my case I’ll need 800).
> 
> I believe all you needto do is change the following in scg_cake.c:
> #define CAKE_QUEUES (1024)
> 
> Now I heard reports that above a certain number this breaks, but it might be enough for 32K or even 64k, or ~32 to 64 queues per customer, which might already help to spread out things a bit...

That’s true, I could try it. I could also add to the classifier the ability to spread one subscriber across multiple queues, as Toke suggested. You’d have to specify how many queues you’ve compiled Cake with (defaults to 1024), and it would spread subscribers across the available classids evenly. I suppose the userspace tool is right place to do this even in ISP Cake, as updates can be made pretty easily to a global BPF map without restarting the qdisc, and further customizations are possible for those writing their own eBPF without qdisc changes.

We’ll eventually want to handle changing the number of queues in ISP Cake somehow so people don’t have to change and re-compile. Should become a tc parameter instead of a define? I guess it comes back to the question of “new qdisc or not”…


[-- Attachment #2: Type: text/html, Size: 5168 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

The big thing to note here is that it needs to be a power of two, so that
the modulo operation on the hash table is really a mask operation, not a
divide.  It's easy to tell the compiler that if it's a compile time
constant.  Needs a lot more finesse if it's not.

But it could reasonably be configurable at compile time.  Just make sure to
spit the dummy out if it's a weird size.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 479 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 30, 2018, at 12:55 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> I believe all you needto do is change the following in scg_cake.c:
>> #define CAKE_QUEUES (1024)
>> 
>> Now I heard reports that above a certain number this breaks, but it
>> might be enough for 32K or even 64k, or ~32 to 64 queues per customer,
>> which might already help to spread out things a bit...
> 
> Has to fit in 16 bits, so 64k is the max; we really ought to have made
> that parameter configurable...

Couldn’t it still be made so now? Not sure of the performance impact though.

If it’s better to make it powers of 2 or there are other restrictions on it (min and max), the config interface should also handle this.

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

>> On Jul 30, 2018, at 12:55 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> I believe all you needto do is change the following in scg_cake.c:
>>> #define CAKE_QUEUES (1024)
>>> 
>>> Now I heard reports that above a certain number this breaks, but it
>>> might be enough for 32K or even 64k, or ~32 to 64 queues per customer,
>>> which might already help to spread out things a bit...
>> 
>> Has to fit in 16 bits, so 64k is the max; we really ought to have made
>> that parameter configurable...
>
> Couldn’t it still be made so now? Not sure of the performance impact
> though.

It could, but it would take some care. There's the issue of
power-of-two-ness and avoiding divides that Jonathan pointed out, and
the memory allocation would be complicated somewhat. Certainly doable,
but I'm not sure it's worth it for its own sake if the plan is to build
a new qdisc anyway...

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Pete Heist]

> On Jul 30, 2018, at 1:28 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <pete@heistp.net> writes:
> 
>> Couldn’t it still be made so now? Not sure of the performance impact
>> though.
> 
> It could, but it would take some care. There's the issue of
> power-of-two-ness and avoiding divides that Jonathan pointed out, and
> the memory allocation would be complicated somewhat. Certainly doable,
> but I'm not sure it's worth it for its own sake if the plan is to build
> a new qdisc anyway...

Fair enough, we’ll see.

The challenge now is eBPF classification doesn’t look like it will be usable on 3.16 (on FreeNet's routers). Maps weren’t introduced until 3.18 and tc support in 4.1. I’ll see what upgrades are possible.

Also, if ISP Cake is a new qdisc, I don’t see now why it would need to support kernel versions where scalable classification by IP or MAC isn’t practical to do. The hard lower limit seems like 4.1, but also direct-action mode in 4.4 would be good for performance, and without the LPM trie in 4.11, classification by subnet would be more difficult...

Pete

Re: [Cake] Using cake to shape 1000’s of users.

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

>> On Jul 30, 2018, at 1:28 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Pete Heist <pete@heistp.net> writes:
>> 
>>> Couldn’t it still be made so now? Not sure of the performance impact
>>> though.
>> 
>> It could, but it would take some care. There's the issue of
>> power-of-two-ness and avoiding divides that Jonathan pointed out, and
>> the memory allocation would be complicated somewhat. Certainly doable,
>> but I'm not sure it's worth it for its own sake if the plan is to build
>> a new qdisc anyway...
>
> Fair enough, we’ll see.
>
> The challenge now is eBPF classification doesn’t look like it will be
> usable on 3.16 (on FreeNet's routers). Maps weren’t introduced until
> 3.18 and tc support in 4.1. I’ll see what upgrades are possible.
>
> Also, if ISP Cake is a new qdisc, I don’t see now why it would need to
> support kernel versions where scalable classification by IP or MAC
> isn’t practical to do. The hard lower limit seems like 4.1, but also
> direct-action mode in 4.4 would be good for performance, and without
> the LPM trie in 4.11, classification by subnet would be more
> difficult...

Yeah, you'll probably need a newish kernel to do all the stuff you want
with eBPF...

-Toke

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

Looks like even big ISPs could potentially benefit from this sort of thing:

	https://www.youtube.com/watch?v=b3U-sSADYYY

 - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Dan Siemon]

On Fri, 2018-07-27 at 21:58 +0300, Jonathan Morton wrote:
> > 
> > We have some deployments with multiple access technologies (eg
> > DOCSIS,
> > DSL and wireless) behind the same box so per customer overhead
> > would be
> > useful.
> 
> The design I presently have in mind would allow setting the overhead
> per speed tier.  Thus you could have "10Mbit DOCSIS" and "10Mbit
> ADSL" as separate tiers.  That is, there's likely to be far fewer
> unique overhead settings than customers.

The unique overhead configurations will be much smaller than the number
of subscribers but from a provisioning standpoint, this info belongs
with the subscriber. Having to map a subscriber into an 'overhead
config' vs. just setting those values along with rate would be
inconvenient.

> > I am interested in the diffserv class ideas in Cake but need to do
> > some
> > experimentation to see if that helps in this environment. It would
> > be
> > interesting to be able to target sub-classes (not sure what the
> > proper
> > Cake terminology is) based on a eBPF classifier.
> 
> The trick with Diffserv is classifying the traffic correctly, given
> that most traffic in the wild is not properly marked, and some
> actively tries to hide its nature.  As an ISP, applying your own
> rules could be seen as questionable according to some interpretations
> of Net Neutrality.  With Cake that's not an issue by default, since
> it relies entirely on the DSCP on the packet itself, but it does mean
> that a lot of traffic which *should* probably be classified other
> than best-effort is not.
> 
> Cake does cope well with unclassified latency-sensitive traffic, due
> to DRR++, so it's not actually necessary to specifically identify
> such applications.  What it has trouble with, in common with all
> other flow-isolating qdiscs, is applications which open many parallel
> connections for bulk transfer (eg. BitTorrent, which is used by many
> software-update mechanisms as well as file-sharing).  Classifying
> this traffic as Bulk (CS1 DSCP) gets it out of the way of other
> applications while still permitting them to use available capacity.
> 
> I have some small hope that wider deployment of sensible, basic
> Diffserv at bottlenecks will encourage applications to start using it
> as intended, thereby solving a decades-long chicken-egg
> problem.  Presently Cake has such an implementation by default, but
> is not yet widely deployed enough to have any visible impact.
> 
> This is something we should probably discuss in more detail later.
> 
> > Re accounting, we currently count per-IP via eBPF not via qdisc
> > counts.
> 
> Fine, that's a feature I'm happy to leave out.

One problem I haven't solved yet is counting bytes and packets per-IP
post qdisc. This is required when multiple IPs are sent to the same
QDisc but per-IP stats are still needed.

It would be great if there were a post qdisc hook for eBPF programs.
Something like a cls_act that runs after the other qdiscs on Tx instead
of before. It would be even better if the skb could be flagged with an
ID of some kind so that the outbound eBPF program doesn't need to parse
the headers again and could instead just do a map lookup + update.

I started looking at XDP for this but at the time it was all receive
side. I'm not sure if that's still the case.

> > 
> So in summary, the logical flow of a packet should be:
> 
> 1: Map dst or src IP to subscriber (eBPF).
> 2: Map subscriber to speed/overhead tier (eBPF).
> 3: (optional) Classify Diffserv (???).
> 4: Enqueue per flow, handle global queue overflow (rare!) by dropping
> from head of longest queue (like Cake).

Since the encapsulations used in the ISP world are pretty diverse,
being able to generate the flow hash from the eBPF program is also
required. Today we do this by setting the skb->hash field and letting
the outbound FQ-CoDel classify based on that.

> --- enqueue/dequeue split ---
> 5: Wait for shaper on earliest-scheduled subscriber link with waiting
> traffic (borrow sch_fq's rbtree?).
> 6: Wait for shaper on aggregate backhaul link (congestion can happen
> here too).

Yes, multiple layers of bandwidth limiting are required - two at a
minimum. One for the subscriber's plan rate and another for the nearest
access device.

Conceptually, is it possible to build something akin to the HTB qdisc
that uses DRR++ and retains the flexible hierarchy? I don't have a good
sense of the performance impact of this type of setup vs. a single more
featureful qdisc.

> 7: Choose one of subscriber's queues, apply AQM and deliver a packet
> (mostly like Cake does).
> 
> If that seems reasonable, we can treat it as a baseline spec for
> others' input.
> 
>  - Jonathan Morton

Re: [Cake] Using cake to shape 1000’s of users.

[Michel Blais]

[-- Attachment #1: Type: text/plain, Size: 1247 bytes --]

I asked the same not long ago. It's currently limited to a global speed
shared by all IP address so you must use a veth with cake config by ip
address.

I would like to eventually see a option in cake with src, dst, src-dual and
dst-dual host mode with bandwidith by ip-address and use a veth with
different cake config by package instead of by user. That would be great
for ISP, only an ipset address list by package to manage once the initial
config done.

Currently didn't have time to test a veth by user. It's in my plan but if
you ever try it. I would gladly like to know the result.

---
Cordialement,  /  With regards,

Michel Blais
Targo communications

2018-07-16 14:39 GMT-04:00 Mike <mike@surfglobal.net>:

> Is it possible to use cake on one server to provide shaping and QOS to
> 1000’s of users through a transparent bridge with various speed plans.  I
> know one company is doing it using fq_codel but I have been unable to
> locate any resources on how to get it to work on more than one speed plan.
> Any help would be greatly appreciated.
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 2261 bytes --]

Re: [Cake] Using cake to shape 1000’s of users.

[Jonathan Morton]

> On 16 Jul, 2018, at 9:39 pm, Mike <mike@surfglobal.net> wrote:
> 
> Is it possible to use cake on one server to provide shaping and QOS to 1000’s of users through a transparent bridge with various speed plans.  I know one company is doing it using fq_codel but I have been unable to locate any resources on how to get it to work on more than one speed plan.  Any help would be greatly appreciated.

This is an application that we've had in the back of our minds for a little while, and we'd certainly like to see what we can do to help out.  It might be that a new qdisc using some of Cake's technology (arranged a little differently) would be appropriate.

 - Jonathan Morton

[Cake] Using cake to shape 1000’s of users.

[Mike]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

Is it possible to use cake on one server to provide shaping and QOS to 1000’s of users through a transparent bridge with various speed plans.  I know one company is doing it using fq_codel but I have been unable to locate any resources on how to get it to work on more than one speed plan.  Any help would be greatly appreciated.


[-- Attachment #2: Type: text/html, Size: 742 bytes --]