From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id F085621F56F; Sat, 30 Aug 2014 10:19:17 -0700 (PDT) Received: by mail-ig0-f179.google.com with SMTP id r2so4039516igi.12 for ; Sat, 30 Aug 2014 10:19:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1AlnuI47FCUSB/ez7C8FOGFkSamiX93v+jR3mEI+2mg=; b=HjGSFLALJKdeF0BAijvDSzJuCc5P1D7g5s7p78e45c6Gy0Oac9CJlaU/yk0+T9J4BX qY6SIFa1QsFxjeNVIQDoDDRp8JTm8WfM59ZRrF/bPktFmCELpzJdK7CjJZKgep8xG3ZH Up9txMCF8qmq/THGZm6nMZuOPAkGYYcWcZ/K6yAcrPh2bws8nuaB108tDnRhcMio1Khy DO/HQ1JZlEPORsPdSK8I5CoHfjlz4nC9Ouz5rO6CxnVTgfr2zT4Onz6fxKS/bnKo7JeB yerXe5ShGmRZmLhTIus+U8DUADEppGWh8USPPo/IybFvqHRgqSwdUYfv3EVvipAkmv1F AJtQ== MIME-Version: 1.0 X-Received: by 10.42.107.145 with SMTP id d17mr1821151icp.61.1409419157157; Sat, 30 Aug 2014 10:19:17 -0700 (PDT) Received: by 10.64.243.196 with HTTP; Sat, 30 Aug 2014 10:19:17 -0700 (PDT) In-Reply-To: References: Date: Sat, 30 Aug 2014 10:19:17 -0700 Message-ID: From: Aaron Wood To: Dave Taht Content-Type: multipart/alternative; boundary=20cf301fb60d33bd580501dbf897 Cc: cerowrt-devel , bloat Subject: Re: [Bloat] Comcast upped service levels -> WNDR3800 can't cope... X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Aug 2014 17:19:18 -0000 --20cf301fb60d33bd580501dbf897 Content-Type: text/plain; charset=UTF-8 On Fri, Aug 29, 2014 at 11:06 AM, Dave Taht wrote: > On Fri, Aug 29, 2014 at 9:57 AM, Aaron Wood wrote: > > Comcast has upped the download rates in my area, from 50Mbps to 100Mbps. > > This morning I tried to find the limit of the WNDR3800. And I found it. > > 50Mbps is still well within capabilities, 100Mbps isn't. > > > > And as I've seen Dave say previously, it's right around 80Mbps total > > (download + upload). > > > > > http://burntchrome.blogspot.com/2014/08/new-comcast-speeds-new-cerowrt-sqm.html > > Thank you very much, as always, for doing public benchmarking with a good > setup! > No problem, I find this sort of investigation a lot of fun. Even if it is somewhat maddening at times. > Yes we hit kind of an unexpected wall on everything shipped with a > processor > originally designed in 1989, and the prevalance of hardware offloads to > bridge > the gap and lower costs between 100mbit and a gige is a real PITA. > Do you think this is a limitation of MIPS as a whole, or just the particular MIPS cores in use on these platforms? OTOH, I have noticed that MIPS is losing ground to ARM as bandwidths go up. The router SoCs that I'm seeing from the usual suspects have been switching from MIPS to ARM over the last year or two. The WNDR is in the top-tier for SOHO SoCs, but at a product family is getting long in the tooth. > > I tried disabling downstream shaping to see what the result was, and it > > wasn't pretty. > > Well, I'll argue that only seeing an increase of 20ms or so with the > upstream > only, fq_codeled, (vs 120ms not) is not bad and within tolerances of most > applications, even voip. Secondly the characteristics of normal > traffic, as opposed > to the benchmark, make it pretty hard to hit that 100mbit download limit, > so a mere outbound rate limiter will suffice. > Well, yes... I have considered just turning it off entirely, as the the extra latency isn't awful. And frankly, the laptops (individually) never see that sort of bandwidth, but the AppleTV might when downloading video (I need to go see what the downloads are capped at by Apple). The cpu caches are 32k/32k, the memory interface 16 bit. The rate limiter > (the thing eating all the cycles, not the fq_codel algorithm!) is > single threaded and has global locks, > and is at least partially interrupt bound at 100Mbits/sec. > This is interesting, and lines up with Sebastian's idea about perhaps using ethtool to lock the upstream interface to 100Mbps. Except that moves the bottleneck to the next upstream device... (modem), with it's buffer mgmt, so maybe that's not a great idea, either. Upstream is certainly where the biggest issues are. > > Or should I start looking for something like this: > > > > http://www.gateworks.com/product/item/ventana-gw5310-network-processor > > > > (although that's an expensive board, given the very low production > volume, > > for the same cost I could probably build a small passively-cooled > > mini/micro-atx setup running x86 and dual NICs). > > There is that option as well. I would certainly like to find a low end x86 > box > that could rate limit + fq_codel at up to 300Mbits/sec. Toke's x86 boxes > have proven out to do 100Mbit/10Mbit correctly, but I don't remember their > specs, nor has he tried to push them past that, yet. > If I do get my hands on a Ventana board (I may still for work purposes), I'll certain set it up and see what it does in this scenario, too. -Aaron --20cf301fb60d33bd580501dbf897 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Fri, Aug 29, 2014 at 11:06 AM, Dave Taht <<= a href=3D"mailto:dave.taht@gmail.com" target=3D"_blank">dave.taht@gmail.com= > wrote:
On Fri, Aug 29, 2014 at 9:57= AM, Aaron Wood <woody77@gmail.com<= /a>> wrote:
> Comcast has upped the download rates in my area, from 50Mbps to 100Mbp= s.
> This morning I tried to find the limit of the WNDR3800.=C2=A0 And I fo= und it.
> 50Mbps is still well within capabilities, 100Mbps isn't.
>
> And as I've seen Dave say previously, it's right around 80Mbps= total
> (download + upload).
>
> http://burntchrome.blogspot.com/201= 4/08/new-comcast-speeds-new-cerowrt-sqm.html

Thank you very much, as always, for doing public benchmarking with a = good setup!

No problem, I find this sor= t of investigation a lot of fun. =C2=A0Even if it is somewhat maddening at = times.
=C2=A0
Yes we hit kind of an unexpected wall on everything shipped with a processo= r
originally designed in 1989, and the prevalance of hardware offloads to bri= dge
the gap and lower costs between 100mbit and a gige is a real PITA.

Do you think this is a limitation of MIPS as a = whole, or just the particular MIPS cores in use on these platforms? =C2=A0<= /div>

OTOH, I have noticed that MIPS is losing ground to ARM = as bandwidths go up. =C2=A0The router SoCs that I'm seeing from the usu= al suspects have been switching from MIPS to ARM over the last year or two.= =C2=A0The WNDR is in the top-tier for SOHO SoCs, but at a product family i= s getting long in the tooth.
=C2=A0
> I tried disabling downstream shaping to see what the result was, and i= t
> wasn't pretty.

Well, I'll argue that only seeing an increase of 20ms or so with = the upstream
only, fq_codeled, (vs 120ms not) is not bad and within tolerances of most applications, even voip. Secondly the characteristics of normal
traffic, as opposed
to the benchmark, make it pretty hard to hit that 100mbit download limit, so a mere outbound rate limiter will suffice.

Well, yes... =C2=A0I have considered just turning it off entirely, a= s the the extra latency isn't awful. =C2=A0And frankly, the laptops (in= dividually) never see that sort of bandwidth, but the AppleTV might when do= wnloading video (I need to go see what the downloads are capped at by Apple= ).
=C2=A0

The cpu cach= es are 32k/32k, the memory interface 16 bit. The rate limiter
(the thing eating all the cycles, not the fq_codel algorithm!) is
single threaded and has global locks,
and is at least partially interrupt bound at 100Mbits/sec.
=

This is interesting, and lines up with Sebastian's = idea about perhaps using ethtool to lock the upstream interface to 100Mbps.= =C2=A0Except that moves the bottleneck to the next upstream device... (mod= em), with it's buffer mgmt, so maybe that's not a great idea, eithe= r. =C2=A0Upstream is certainly where the biggest issues are.
=C2=A0
> Or sho= uld I start looking for something like this:
>
> http://www.gateworks.com/product/item/ventan= a-gw5310-network-processor
>
> (although that's an expensive board, given the very low production= volume,
> for the same cost I could probably build a small passively-cooled
> mini/micro-atx setup running x86 and dual NICs).

There is that option as well. I would certainly like to find a low en= d x86 box
that could rate limit + fq_codel at up to 300Mbits/sec. Toke's x86 boxe= s
have proven out to do 100Mbit/10Mbit correctly, but I don't remember th= eir
specs, nor has he tried to push them past that, yet.
<= br>
If I do get my hands on a Ventana board (I may still for work= purposes), I'll certain set it up and see what it does in this scenari= o, too.

-Aaron
--20cf301fb60d33bd580501dbf897--