[Bloat] Comcast upped service levels -> WNDR3800 can't cope...

[Dave Taht]

On Fri, Aug 29, 2014 at 9:57 AM, Aaron Wood <woody77 at gmail.com> wrote:
> Comcast has upped the download rates in my area, from 50Mbps to 100Mbps.
> This morning I tried to find the limit of the WNDR3800.  And I found it.
> 50Mbps is still well within capabilities, 100Mbps isn't.
>
> And as I've seen Dave say previously, it's right around 80Mbps total
> (download + upload).
>
> http://burntchrome.blogspot.com/2014/08/new-comcast-speeds-new-cerowrt-sqm.html

Thank you very much, as always, for doing public benchmarking with a good setup!

Yes we hit kind of an unexpected wall on everything shipped with a processor
originally designed in 1989, and the prevalance of hardware offloads to bridge
the gap and lower costs between 100mbit and a gige is a real PITA.

I had only had data on x86 (good to 100s or 1000s of mbits) and a few
mips platforms to go on until recently.

> I tried disabling downstream shaping to see what the result was, and it
> wasn't pretty.

Well, I'll argue that only seeing an increase of 20ms or so with the upstream
only, fq_codeled, (vs 120ms not) is not bad and within tolerances of most
applications, even voip. Secondly the characteristics of normal
traffic, as opposed
to the benchmark, make it pretty hard to hit that 100mbit download limit,
so a mere outbound rate limiter will suffice.

As for a comment in that blog, nothing is running pie yet. Some enhancements
to how buffering is configured on the modem may well have finally been put
into play in your default configuration. The best short term option the cablecos
have had has been to increase bandwidth as it's the only easy option they
can tweak, no matter how much it adds to the cost of their cable plant, etc.,
lacking further co-operation with the CMTS vendors and cablemodem makers.

> I also tried using the "simplest.qos" script, and that
> didn't really gain me anything, so I went back to the simple.qos script
> (those results aren't included above).
>
> It looks like it's definitely time for a new router platform (for me).

At the moment the rangeley platforms look like a good bet for future work,
but they are terribly pricey. A plus is that openwrt already supports them.
I got one earlier this week but didn't get the right case and power supply
for it...

Less pricy is the edgerouter pro and some
of the new arm based routers with 1ghz dual cores. In the first case
that involves switching to debian which is a headache, and the htb
implementation appears to be inaccurate at higher rates so far, why
I don't know.

In the second
there are tons of binary blobs and beta code to deal with.

I lean, personally, towards taking a vacation.

>
> Or, we need to find a way to implement the system such that it doesn't max
> out a 680MHz mips core just to push 100Mbps of data.

An option is to explore conventional policing, rather than rate shaping, the
downstream. Policing is much lighter weight, but has really drastic effects
when limits are hit, and you have to tune the burst parameter sanely.

> That's roughly 10K cpu
> cycles per packet, which seems like an awful lot.  Unless the other problem
> is that the memory bus just can't keep up.  My experience of a lot of these
> processors is that the low-level offload engines have great DMA capabilities
> for "wire-speed" operation, but that the processor core itself can't move
> data to save it's life.

The cpu caches are 32k/32k, the memory interface 16 bit. The rate limiter
(the thing eating all the cycles, not the fq_codel algorithm!) is
single threaded and has global locks,
and is at least partially interrupt bound at 100Mbits/sec.

One thing to look at tuning might be the htb burst parameter to the sqm system.
we tune the quantum presently, but not the burst.

I would very much like to find a better inbound software rate limiting
algorithm and/or
to improve what exists today. In the future, finding something that
could be easily
implemented in hardware would be good.

I have some thoughts
towards adding tbf to bql directly which might be a win, but that is
outbound only.

>
> What's the limit of the EdgeRouter Lite?

It's a little higher but not good. It is a dual core 680mhz mips
platform, but has
a lot of single-threadedness and possible other problems. Work on it continues
over on the relevant ubnt forums, but I haven't had time to profile the kernel
in any case to look hardware at where the limits are coming from. Hope someone
else does with more time and chops.

> Or should I start looking for something like this:
>
> http://www.gateworks.com/product/item/ventana-gw5310-network-processor
>
> (although that's an expensive board, given the very low production volume,
> for the same cost I could probably build a small passively-cooled
> mini/micro-atx setup running x86 and dual NICs).

There is that option as well. I would certainly like to find a low end x86 box
that could rate limit + fq_codel at up to 300Mbits/sec. Toke's x86 boxes
have proven out to do 100Mbit/10Mbit correctly, but I don't remember their
specs, nor has he tried to push them past that, yet.

> -Aaron
>
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article