Re: [Cake] Cake implementations

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Fri, 22 Nov 2019 12:46:18 +0100
Toke Høiland-Jørgensen <toke@redhat.com> wrote:

> "Adam Moffett" <adam@plexicomm.net> writes:
> 
> >>>
> >>>  Are there any commercial products already using Cake?  
> >>
> >>Evenroute, eero, ubnt top that list. Evenroute's implementation is
> >>superb, the first one that used active line measurements to handle
> >>"sag". Anything derived from openwrt (somewhere between 10-30% of the
> >>home router market). I'm not sure if preseem is using it or not.
> >>dd-wrt. Most other things doing "SQM" are doing it via htb + fq_codel.
> >>
> >>  
> > An idea which was floated was to experiment with routing ISP
> > customer traffic through a Linux server using cake to improve
> > customer experience.  Basically like Preseem.  My colleague has
> > toyed with it a bit in small test cases and was impressed with the
> > outcomes.
> >
> > He's looked closer than I have, but I'm trying to picture how his
> > idea would scale.  I believe I'm seeing a CLI tool for configuring
> > policies. It seems like we'd have to create a middle layer to
> > create/update policies for customer's IP address based on
> > information obtained from our AAA and CRM systems.  I can picture
> > some shapes that might take, but I think it would ultimately have
> > to revolve around scripting the tc command.  There would be
> > thousands of policies and a policy would be created/updated
> > whenever a subscriber reconnects (e.g. when a DHCP lease renews or
> > a RADIUS auth event happens or similar).  
> 
> I know several ISPs that do this (route traffic through a Linux box and
> shape there). This deployment mode has not been the primary focus of
> CAKE, though; the "standard" way to do it is with HTB+FQ-CoDel, which
> allows you to set up arbitrarily complex configurations on a single
> interface. 

Yes, I worked for an ISP back in 2005, that route traffic through a
Linux box and does shaping.  There were a number of scalabilities
issues, that we fixed (upstream) and also Open Sources components for
others to reuse.

I did a public talk in 2008 about how we made it scale:
 http://people.netfilter.org/hawk/presentations/osd2008/osd2008_iptables_rules_JesperDangaardBrouer.pdf

This was before Codel and even-before Bufferbloat was coined/defined.
Our setup was HTB+SFQ, with a shaper per customer.  This actually
solved the bufferbloat problem in-practice for people, as SFQ gave each
end-user 128 queues.

Today I would not use iptables for this.  Instead I would use BPF or
nftables 'set' infrastructure.


> This can also be made to scale, but there's a central qdisc
> lock in Linux that you have to get around to scale to multiple cores.
> Fortunately, Jesper has already solved this particular issue:
> 
> https://github.com/netoptimizer/xdp-cpumap-tc

The central qdisc TX lock is a major scalability issue.  But I've
solved in above link, via XDP and TC.  This actually runs in production
at an ISP.

 
> > Should we even pursue this idea?  
> 
> In my own totally non-biased opinion: Yes! :)

It would be great.


> > Although most staff who would touch this will have studied programming 
> > in college, I would not qualify any of us as "programmers" per se.  My 
> > biggest concern would be hitting a service affecting problem that we 
> > can't solve.  
> 
> One way to go about this would be to open source the entire solution.
> There are already bits and pieces available as open source (such as
> Jesper's repository above, and sqm-scripts[0]) that you can build on,
> and this way you could enlist community help to solve any issues with
> the Linux side. You'd still need to get the data from your internal
> systems, of course, but you could define a simple configuration format
> that was part of the open source code; then you'll just need to write a
> script that grabs customer info from your CRM and outputs the config
> format...

Yes, this is the advantage of Open Source! :-)

If you create this as Open Source, then feel free to reach out to me
directly.  And I know of several other people (working at ISPs) that
would likely be interested in collaborating.

As Toke wrote, you can still keep your CRM system proprietary and
closed-source, we don't care anyhow ;-)

 
> > Second concern is that many of our equipment vendors already use
> > Linux. Even Cisco now in some products. Maybe we'll waste our time
> > trying to roll our own solution and then find that a software update
> > from a vendor next year gives us everything we needed anyway.  

I would not hold my breath... and if it does come as a software
upgrade, you can expect to pay plenty for it...

Maybe I'm the wrong guy to ask, but I really think it is straight
forward to implement an ISP Linux router with per customer handling.
All the components seems avail as Open Source.  (There do seem to be a
lack around a DHCP server that can handle this well).

 
> This would be great, of course, and do go and bug your vendors to
> solve this problem! Note, however, that just because a system is
> running Linux on the control plane, it may be using a
> hardware-offloaded data plane that does not have any of the
> bufferbloat mitigation features (unless the vendor specifically
> implemented them). I'm hoping that *eventually* these things will be
> ubiquitous across the industry, but thus far this has seemed to be an
> "any decade now" kind of proposition :/

I would prefer, not to waste time on creating bugs for your vendor, but
instead actually have the ability to fix the issue myself, or hire some
Linux consultant that can...

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer