Re: [LibreQoS] [Bloat] really lovely live plots of bandwidth, queue depth, marks and drops in libreqos

Re: [LibreQoS] [Bloat] really lovely live plots of bandwidth, queue depth, marks and drops in libreqos

[Dave Taht]

On Sat, Jan 7, 2023 at 4:57 AM Rich Brown via Bloat
<bloat@lists.bufferbloat.net> wrote:
>
>
>
> On Jan 6, 2023, at 11:14 PM, bloat-request@lists.bufferbloat.net wrote:
>
> Wearing my theorist hat... and looking at all the ISP plans
> https://payne.taht.net is now emulating.... (click on bandwidth test,
> then a plan)
>
>
> This is really cool!

I once wrote how, when I first worked on what became wifi, how hard it
was to imagine gear that once took two refrigerator-sized wireless
devices PLUS a refrigerator - to something that took up a tiny piece
of a single chip, in only 20 years. Same goes here - I had until now,
figured that we'd first see a traffic conditioner like this appear in
an FPGA for some high end offloaded ethernet card, and then move down
into logic, not be doable with a cheap off the shelf whitebox. My hat
is off to robert chacon for getting something that worked, and
herbert, for pushing it even further, and the half dozen early adoptor
ISPs (that we know of) that are willing to deploy even beta versions
on their customer bases (after having heard their support lines go
silent)

> I see the "Bandwidth Test", but don't see a place to look at a Plan...

click on it, the tests will start to run, it will pick up the
(presently) 20 plans we are are emulating... successfully show the
RTTs
are in line with codel theory (you'll see the top 10 worst RTTs end up
ordered by 10/10 25/3 and so on) - then click on the link for the plan
for those, and more detail comes to light. (feedback on what should be
plotted and how, deeply desired)

The testbed is from 3 hefty bare metal servers (temporarily) donated
by equinix (thank you equinix!), the work, mostly volunteer, with a
few dollars thrown in so far by the libreqos-users of the world (the
biggest live libreqos site we know of so far is 10k subscribers
pushing 11gbit). This is the first bufferbloat.net supported project
that does have some chance at an actual business model - the going
rate is 50 cents/subscriber/month from the commercial QoE vendors like
Preseem,  Paraqum, Cambium, etc - but as usual, to get the science
right, it's open sourced (But gplv2, and I'm considering writing my
bits of it as GPLv3). I think the QoE market is big enough for both
closed and open versions of middleboxes like these to exist, and oy!
have I now learned of all the other problems ISPs have in doing this,
and deploying IPv6, and how much (at the higher rates) the real
problems move to the customer wifi.

We haven't automated outputting the flent results of those tests, and
at the moment I'm mostly interested in validating results of the all
the new  tests like goresponsiveness, speedtest, etc. I hope to line
up the authors of those tools to help. There's also emulations of a
fifo, sfq, etc we can do fairly easily instead of just cake and
fq_codel. Adding in proper delays to emulate technologies like DSL
(20ms baseline latency), cable (10), fiber and fixed wireless (2) is
fairly straightforward, some emulation of wifi's behaviors also.

I made an attempt to correctly emulate 10k IP addresses (with those
delays AND a routing protocol) and quickly ran the sole client box out
of cpu - I'd had grand plans (see
https://github.com/LibreQoE/LibreQoS/tree/main/sim) - but it's looking
like we'll have to spin up a ton more client boxes, or adopt the same
XDP approaches we're using in the middlebox, to emulate that many
customers.

I note that despite doing these emulations, and now having such a cool
demo  - the real purpose of the work was to improve the next release
of libreqos (v1.4, targetted for mid-march) so it could have 6 sigmas
of reliability, and could hopefully crack 50Gbit on 16 cores, perhaps
even 100Gbit on 64 cores, in the real world. It is not, unfortunately,
scaling linearly presently, and we keep putting in features and seeing
performance get thwacked - I'd really wanted to sample stats at 10ms,
not 50ms (nyquist theorem), no can do without a lot of new rust that
would access the tc stats in binary rather than json. Rust eludes
me... the lead dev is the author of a great book on rust (which still
eludes me),

There's been a couple other nice diversions along the way, presently
the testbed is running with a single network port with in/out over two
vlans.

More github sponsors gladly accepted - I really wish those endpoints
and edges that would like to build a metaverse would chip in, to me,
this work is a win - win - win - or a business model, suggested. It's
not just the enormous up-front cost of perfecting this that need
covering, but six sigmas is *hard*.... comprehensive analytics,
also... It's the most fun I've had on a project in a long time.

If anyone would like to tag along the developer channel is
#libreqos:matrix.org where I have become @dtaht:matrix.org.

PS The names of the machines, btw, are based on starcraft - the client
boxes and emulations are "zergXX", the middlebox "Payne", and the test
drivers protossts.

> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC