[Bloat] Still seeing bloat with a DOCSIS 3.1 modem

[Aaron Wood]

On Wed, Mar 25, 2020 at 12:18 PM Dave Taht <dave.taht at gmail.com> wrote:

> On Wed, Mar 25, 2020 at 8:58 AM Aaron Wood <woody77 at gmail.com> wrote:
> >
> > One other thought I've had with this, is that the apu2 is multi-core,
> and the i210 is multi-queue.
> >
> > Cake/htb aren't, iirc, setup to run on multiple cores (as the rate
> limiters then don't talk to each other).  But with the correct tuple
> hashing in the i210, I _should_ be able to split things and do two cores at
> 500Mbps each (with lots of compute left over).
> >
> > Obviously, that puts a limit on single-connection rates, but as the
> number of connections climb, they should more or less even out (I remember
> Dave Taht showing the oddities that happen with say 4 streams and 2 cores,
> where it's common to end up with 3 streams on the same core).  But assuming
> that the hashing function results in even sharing of streams, it should be
> fairly balanced (after plotting some binomial distributions with higher "n"
> values).  Still not perfect, especially since streams aren't likely to all
> be elephants.
>
> We live with imperfect per core tcp flow behavior already.
>

Do you think this idea would make it worse, or better?  (I couldn't tell
from your comment how, exactly, you meant that)

OTOH, any gains I'd get over 500Mbps would just be gravy, as my current
router can't do more than that downstream on a single core, so even if the
sharing I have is uneven, it's better than what I have now (~400Mbps and
not pretty), so even if all fat streams landed on one core (unlikely
starting at 4+ streams), so I think I'd see overall gains (in my situation,
others might not).


> What I wanted to happen was the "list" ingress improvement to become
> more generally available ( I can't find the lwn link at the moment).
> It has. I thought that then we could express a syntax of tc qdisc add
> dev eth0 ingress cake-mq bandwidth whatever, and it would rock.
>
> I figured getting rid of the cost of the existing ifb and tc mirred,
> and having a fast path preserving each hardware queue, then using
> rcu to do a sloppy allocate atomic lock for shaped bandwidth and merge
> every ms or so might be then low-cost enough. Certainly folding
> everything into a single queue has a cost!
>

Sharing the tracked state by each cake-mq "thread" and updating it every so
often?

Or doing the rate limiting in one core, and the fq'ing on another? (I don't
think this is what you meant?)


> I was (before money ran out) prototyping adding a shared shaper to mq
> at one point (no rcu, just  There have been so many other things
> toss around (bpf?)
>
> As for load balancing better, google "RSS++", if you must.


I few years ago (before my current job ate all my brain cycles), I was
toying around with taking the ideas from OpenFlow/OpenVSwitch and RSS and
using them for parallelizing tasks like this:

- have N worker threads (say N=real cores, or real cores-1, or somesuch),
each fed by RSS / RPS / multiqueue etc.
- have a single controller thread (like the OpenFlow "controller")

Each worker publishes state/observations to the controller, as well as
forwarding "decisions to make", while the controller publishes each
worker's operating parameters to them individually.

The workers then just move packets as fast as they can, using their simple
rules, with no shared state between the workers, or need to access global
tables like connection tracking (e.g. NAT tables to map NAT'd tuples to lan
address tuples).

The controller deals with the decisions and the balancing of params (such
as dynamic configuration of the policer to keep things "fair").

I never got much farther than sketches on paper, and laying out how I'd do
it in a heavily multi-threaded userspace app (workers would use select() to
receive the control messages in-band, instead of needing to do shared
memory access).

I was also hoping that it would generalize to the hardware packet
accelerators, but I think to really take advantage of that, they would need
to be able to implement.

And, I never seem to have the time to try to stand up a rough framework for
this, to try it out...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20200328/12c130a4/attachment-0002.html>