[Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration.

Wed May 21 11:19:55 EDT 2014

On Wed, May 21, 2014 at 7:51 AM,  <dpreed at reed.com> wrote:
> Besides deployment in cerowrt and openwrt, what would really have high
> leverage is that the techniques developed in cerowrt's exploration
> (including fq_codel) get deployed where they should be deployed: in the
> access network systems: CMTS's, DSLAM's, Enterprise boundary gear, etc. from
> the major players.

+10.

>
>
>
> Cerowrt's fundamental focus has been proving that the techniques really,
> really work at scale.

That they even work on a processor designed in 1990! :)

I also have hoped that along the way we've shown what techniques don't
work...

> However, the fundamental "bloat-induced" experiences are actually occurring
> due to bloat at points where "fast meets slow".  Cerowrt can't really fix
> the problem in the download direction (currently not so bad because of high
> download speeds relative to upload speeds in the US - that's in the CMTS's
> and DSLAM's.

Well, I disagree somewhat. The downstream shaper we use works quite
well, until we run out of cpu at 50mbits. Testing on the ubnt edgerouter
has had the inbound shaper work up a little past 100mbits. So there is
no need (theoretically) to upgrade the big fat head ends if your cpe is
powerful enough to do the job. It would be better if the head ends did it,
of course....

>
>
>
> What's depressing to me is that the IETF community spends more time trying
> to convince themselves that bloat is only a theoretical problem, never
> encountered in the field.  In fact, every lab I've worked at (including the

It isn't all the IETF. Certainly google gets it and has made huge strides.
reduced RTT = money.

My own frustration comes from papers that are testing this stuff at 4mbit
or lower and not seeing the results we get above that, on everything.

https://plus.google.com/u/0/107942175615993706558/posts/AbeHRY9vzLR

ns2 and ns3 could use some improvements...

> startup accelerator where some of my current company work) has had the
> network managers complaining to me that a single heavy FTP I'm running
> causes all of the other users in the site to experience terrible web
> performance.  But when they call Cisco or F5 or whomever, they get told
> "there's nothing to do but buy complicated flow-based traffic management
> boxes to stick in line with the traffic (so they can "slow me down").

It is sad that F5, in particular, doesn't have a sane solution. Their whole
approach is to have a "load-balancer" and fq_codel is a load-balancer to
end all load balancers.

I do note nobody I know has ported BQL or fq_codel to bsd (codel is in bsd now)

>
>
>
> Bloat is the most common invisible elephant on the Internet.  Just fixing a

+10.

> few access points is a start, but even if we fix all the access points so
> that uploads interfere less, there's still more impact this one thing can
> have.

I was scared silly at the implications 2 years back, I am more sanguine
now.

>
>
>
> So, by all means get this stuff into mainstream, but it's time to start
> pushing on the access network technology companies (and there are now open
> switches from Cumulus and even Arista to hack)

Oh, cool! I keep waiting for my parallella to show up so I could start
fiddling with ethernet in the fpga....

>
>
>
>
>
>
>
> On Wednesday, May 21, 2014 7:42am, "Frits Riep" <riep at riepnet.com> said:
>
>> Thanks Dave for your responses. Based on this, it is very good that
>> qos-scripts
>> is available now through openwrt, and as I experienced, it provides a huge
>> advantage for most users. I would agree prioritizing ping is in and of
>> itself not
>> the key goal, but based on what I've read so far, fq-codel provides
>> dramatically
>> better responsiveness for any interactive application such as
>> web-browsing, voip,
>> or gaming, so it qos-scripts would be advantageous for users like your mom
>> if she
>> were in an environment where she had a slow and shared internet
>> connection. Is
>> that a valid interpretation? I am interested in further understanding the
>> differences based on the brief differences you provide. It is true that
>> few
>> devices provide DSCP marking, but if the latency is controlled for all
>> traffic,
>> latency sensitive traffic benefits tremendously even without prioritizing
>> by l7
>> (layer 7 ?). Is this interpretation also valid?
>>
>> Yes, your mom wouldn't be a candidate for setting up ceroWRT herself, but
>> if it
>> were set up for her, or if it could be incorporated into a consumer router
>> with
>> automatically determining speed parameters, she would benefit totally from
>> the
>> performance improvement. So the technology ultimately needs to be taken
>> mainstream, and yes that is a huge task.
>>
>> Frits
>>
>> -----Original Message-----
>> From: Dave Taht [mailto:dave.taht at gmail.com]
>> Sent: Tuesday, May 20, 2014 7:14 PM
>> To: Frits Riep
>> Cc: cerowrt-devel at lists.bufferbloat.net
>> Subject: Re: [Cerowrt-devel] Ideas on how to simplify and popularize
>> bufferbloat
>> control for consideration.
>>
>> On Tue, May 20, 2014 at 3:11 PM, Frits Riep <riep at riepnet.com> wrote:
>> > The concept of eliminating bufferbloat on many more routers is quite
>> > appealing. Reading some of the recent posts makes it clear there is a
>> > desire to get to a stable code, and also to find a new platform
>> > beyond the current Netgear. However, as good as some of the proposed
>> > platforms maybe for developing and for doing all of the new
>> > capabilities of CeroWRT, I also would like to propose that there also
>> > be some focus on reaching a wider and less sophisticated audience to
>> > help broaden the awareness and make control of bufferbloat more
>> > available and
>> easier to attain for more users.
>>
>> I agree that reaching more users is important. I disagree we need to reach
>> them
>> with cerowrt. More below:
>>
>> >
>> >
>> > · It appears there is a desire to merge the code into an
>> upcoming
>> > OpenWRT barrier breaker release, which is excellent as it will make it
>> > easier to fight buffer bloat on a wide range of platforms and provide
>> > users with a much easier to install firmware release. I’d like to be
>> > able to download luci-qos-scripts and sqm-scripts and have basic
>> > bufferbloat control on a much greater variety of devices and to many
>> > more users. From an awareness perspective this would be a huge win.
>> > Is the above scenario what is being planned, is it likely to happen in
>> > the
>> reasonable future?
>>
>> Yes, I'd submitted sqm for review upstream, got back a few comments.
>> Intend to
>> resubmit again when I get a chance.
>>
>> >
>> > · From my perspective, it would be ideal to have this
>> available to
>> > many users in a more affordable platform, something like an 8mb flash
>> > router like the TP-Link WDR-4300, which is otherwise a very capable
>> > router with dual channels and good performance.
>> >
>> > · (I’ve managed to set up such a WDR-4300, with OpenWRT
>> trunk,
>> > figured how to telnet and install Luci, then luci-app-qos, and
>> > qos-scripts and I thought the bufferbloat control was remarkable.)
>> > How much better would it be if I were able to use luci-qos-scripts and
>> sqm-scripts instead?
>>
>> You can easily add the .ipk files for sqm-scripts and luci-app-sqm to any
>> release
>> of openwrt. They are just scripts. They need some optional kernel modules
>> and
>> tools.
>>
>> I regard the qos-scripts as pretty good - the core differences from sqm
>> are
>>
>> qos vs sqm
>> ---------------
>> both use fq_codel. :)
>> hfsc vs htb # A wash, hfsc mostly behaves like htb ping optimized vs
>> de-optimized
>> # optimizing for ping looks good in benchmarks but it's silly in the real
>> world
>> (l7) classification vs dscp # clear win to qos here, nearly nothing uses
>> dscp no
>> framing compensation vs comprehensive framing compensation # win here for
>> sqm no
>> alternate queue models vs many alternate queue models # with fq_codel the
>> winner,
>> who cares?
>> fits in 4mb flash vs barely fits in 4mb flash
>>
>> The real killer problem for qos-scripts, for me, was that they didn't do
>> ipv6. I'd
>> like to see that fixed, at the very least.
>>
>>
>> >
>> > · For these target users, they need simplicity, good
>> performance,
>> > ease of setup and affordability. They are not able to deal with the
>> > routing between subnets on wireless, IPv6 setup, and any complexities
>> > introduced by DNSSEC. Marketing the advantages of bufferbloat alone
>> > requires lots of education and publicity (and as we have seen there
>> > are many misleading posts by seemingly persuasive nay-sayers that it is
>> > all
>> smoke and mirrors.
>>
>> Well, my intent is to make the successful bits of technology widely
>> available.
>> They are widely available. And being adopted everywhere. Win.
>>
>> As for the additional complexities, well, they will get less complex over
>> time.
>>
>> In one respect, they are a stake in the ground. I have high hopes for the
>> eventual
>> success of hnetd and mdns proxy services, although they are alpha and
>> nearly
>> unusable right now, some are making substantial investments into them.
>>
>> In another the additional complexities of cero - like routing vs bridging
>> - are
>> there to further the research into fixing wifi technologies - which we
>> haven't
>> really even started yet. I'm increasingly convinced we need to do
>> make-wifi-fast
>> as a separate, focused project, building on a stable base.
>>
>> > · Would it be possible to have a simplified release of CeroWRT
>> (in
>> > terms of setup, and features), make It available for a reliable and
>> > affordable platform, and publicize it and have it reach hopefully a
>> > much wider audience? This could potentially be through the OpenWRT
>> > channels.
>>
>> Possible yes. Affordable, no. Given that this has been a nearly full time
>> project
>> for me, for the last 38 months, with nearly zero revenue, I have no intent
>> or
>> interest in gaining anything other than knowledgable, clued users that
>> want to
>> advance the state of the art. My mom doesn't run cerowrt, nor do I want
>> her to.
>>
>> If someone dropped ~1m/year on the project, that could change, but at
>> present
>> levels of funding I'd be better off working at mcdonalds. Even if funding
>> appeared
>> from the sky I'd rather spend it on R&D than GUI...
>>
>> Certainly IF there was some cost model that made sense, awesome! let's go
>> for
>> world domination!
>>
>> I continue to pursue the grant
>> route, but the only thing that resonates even slightly with potential
>> funders is
>> not speed but security issues, which give me nightmares. Another model
>> that works
>> is actually making and selling a router, but that requires up front
>> capital and
>> entry into a very tight, profit-limited market.
>>
>> Biggest problem we have is supporting ONLY one router, even-semi-well, is
>> a PITA.
>>
>> Adding a new one costs more. I'm now on my 4th day of trying to make the
>> archer
>> work. That's 6k of my life I'll never have back. And the ath10k in it
>> sucks, and
>> working to make that work well is not something I want to be doing due to
>> the
>> binary blob wifi firmware.
>>
>> I'm all in favor of handing off future cerowrt development to a nonprofit
>> of
>> interested users, and sitting back and focusing on fixing just the bits I
>> care
>> about, if anyone is interested in forming one...
>>
>> > · Part of the reason why Tomato had been so popular is that
>> the
>> > firmware upgrade, install, configuration, and management was well
>> > within the capabilities of the average weekend hacker, and there were
>> > compelling features and reliability vs the factory firmwares at the
>> > time.
>>
>> Yep. dd-wrt is the same. And various downstream users like buffalo, meraki
>> etc.
>>
>> I'm totally happy that they exist and have a working market.
>>
>> > · Even installing OpenWRT, especially Trunk, and finding,
>> > downloading and enabling packages, while very powerful, and flexible,
>> > is still quite complex to someone who does not spend a lot of time
>> > reading wiki’s and release notes.
>>
>> Yes, CeroWrt is an improvement on OpenWrt in that regard. But it isn't
>> enough.
>> Doing serious UI improvements and simplification IS necessary, and that's
>> not my
>> bag. The EFF is making noises about doing something with the front end of
>> openwrt
>> and/or cero in the next year or so (see their owtech list for more
>> details), that
>> also goes after the security issue.
>>
>> > I’d be interested in feedback on these thoughts.
>>
>> There you go. I LOVE that we have a happy userbase, and love what we've
>> accomplished, and have loved being here to help make it happen, and love
>> that lots
>> of people want to get it more out there to more people, it's gratifying as
>> hell,
>> and there are a lot of negatives too, like chasing bugs for months on
>> end...
>>
>> ... but after we freeze, I need a vacation and to do something else for a
>> while.
>>
>> I'm presently planning on spending the summer working on something that
>> pays, and
>> on improving ns3 with the GSOC, and testing a deployment of cerowrts on a
>> modest
>> scale, and working on a new/improved rate limiter integrated with
>> fq_codel. And
>> only updating cero for CVEs or major new features.
>>
>> That's a full plate.
>>
>> If someone else wants to step up to maintain or continue to push cerowrt
>> forward
>> in some direction or another, I'm all for it.
>>
>> It's kind of my hope a clear winner on the chipset front will emerge and
>> we can
>> move to that, but even if that happens it will be months and months before
>> it
>> could be considered stable...
>>
>>
>>
>> >
>> >
>> > Frits Riep
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Cerowrt-devel mailing list
>> > Cerowrt-devel at lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >
>>
>>
>>
>> --
>> Dave Täht
>>
>> NSFW:
>>
>> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>

-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article