From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-x233.google.com (mail-oa0-x233.google.com [IPv6:2607:f8b0:4003:c02::233]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id F05AB21F73E; Fri, 29 Aug 2014 11:06:32 -0700 (PDT) Received: by mail-oa0-f51.google.com with SMTP id n16so2035379oag.38 for ; Fri, 29 Aug 2014 11:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=OgR34XmH3PWpQQOaZatpGv73K2M7FLkyAhHBQwuBU8U=; b=Bb+XykfoLTnxCncLpBZMB1QKt2mhOCofa+cQeME2V3rum+vzE9L7xt844a3PqUgEPm JFgOXv3ssOIbONAF8s1Crr85Tk1YDMJkg49I9qxTuzVFzGtGQ/nVBjxfwZxxxXq0l1Ci TCKKTJmeTBfl7+Zj/odlF2P0I+IY08rneMCp6ocw1rSVvVm8n77pz5QjeC8lDvdJ7Ly1 Kfp9lRbMp7gxV1WrNlLkissb1za3vnAGFpTaMUH2J+NcwNdDW/JhJ4pz6aVK+Z0ZKqsn QiyBsfdf/KkDA9AIHP2Peyd8g15znyREMwSqwP1YFtLIx5l5UKMP+6ELQsE5YH+V1+Hb jB2Q== MIME-Version: 1.0 X-Received: by 10.60.135.233 with SMTP id pv9mr3866801oeb.75.1409335592171; Fri, 29 Aug 2014 11:06:32 -0700 (PDT) Received: by 10.202.227.76 with HTTP; Fri, 29 Aug 2014 11:06:32 -0700 (PDT) In-Reply-To: References: Date: Fri, 29 Aug 2014 11:06:32 -0700 Message-ID: From: Dave Taht To: Aaron Wood Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: cerowrt-devel , bloat Subject: Re: [Bloat] Comcast upped service levels -> WNDR3800 can't cope... X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Aug 2014 18:06:33 -0000 On Fri, Aug 29, 2014 at 9:57 AM, Aaron Wood wrote: > Comcast has upped the download rates in my area, from 50Mbps to 100Mbps. > This morning I tried to find the limit of the WNDR3800. And I found it. > 50Mbps is still well within capabilities, 100Mbps isn't. > > And as I've seen Dave say previously, it's right around 80Mbps total > (download + upload). > > http://burntchrome.blogspot.com/2014/08/new-comcast-speeds-new-cerowrt-sq= m.html Thank you very much, as always, for doing public benchmarking with a good s= etup! Yes we hit kind of an unexpected wall on everything shipped with a processo= r originally designed in 1989, and the prevalance of hardware offloads to bri= dge the gap and lower costs between 100mbit and a gige is a real PITA. I had only had data on x86 (good to 100s or 1000s of mbits) and a few mips platforms to go on until recently. > I tried disabling downstream shaping to see what the result was, and it > wasn't pretty. Well, I'll argue that only seeing an increase of 20ms or so with the upstre= am only, fq_codeled, (vs 120ms not) is not bad and within tolerances of most applications, even voip. Secondly the characteristics of normal traffic, as opposed to the benchmark, make it pretty hard to hit that 100mbit download limit, so a mere outbound rate limiter will suffice. As for a comment in that blog, nothing is running pie yet. Some enhancement= s to how buffering is configured on the modem may well have finally been put into play in your default configuration. The best short term option the cab= lecos have had has been to increase bandwidth as it's the only easy option they can tweak, no matter how much it adds to the cost of their cable plant, etc= ., lacking further co-operation with the CMTS vendors and cablemodem makers. > I also tried using the "simplest.qos" script, and that > didn't really gain me anything, so I went back to the simple.qos script > (those results aren't included above). > > It looks like it's definitely time for a new router platform (for me). At the moment the rangeley platforms look like a good bet for future work, but they are terribly pricey. A plus is that openwrt already supports them. I got one earlier this week but didn't get the right case and power supply for it... Less pricy is the edgerouter pro and some of the new arm based routers with 1ghz dual cores. In the first case that involves switching to debian which is a headache, and the htb implementation appears to be inaccurate at higher rates so far, why I don't know. In the second there are tons of binary blobs and beta code to deal with. I lean, personally, towards taking a vacation. > > Or, we need to find a way to implement the system such that it doesn't ma= x > out a 680MHz mips core just to push 100Mbps of data. An option is to explore conventional policing, rather than rate shaping, th= e downstream. Policing is much lighter weight, but has really drastic effects when limits are hit, and you have to tune the burst parameter sanely. > That's roughly 10K cpu > cycles per packet, which seems like an awful lot. Unless the other probl= em > is that the memory bus just can't keep up. My experience of a lot of the= se > processors is that the low-level offload engines have great DMA capabilit= ies > for "wire-speed" operation, but that the processor core itself can't move > data to save it's life. The cpu caches are 32k/32k, the memory interface 16 bit. The rate limiter (the thing eating all the cycles, not the fq_codel algorithm!) is single threaded and has global locks, and is at least partially interrupt bound at 100Mbits/sec. One thing to look at tuning might be the htb burst parameter to the sqm sys= tem. we tune the quantum presently, but not the burst. I would very much like to find a better inbound software rate limiting algorithm and/or to improve what exists today. In the future, finding something that could be easily implemented in hardware would be good. I have some thoughts towards adding tbf to bql directly which might be a win, but that is outbound only. > > What's the limit of the EdgeRouter Lite? It's a little higher but not good. It is a dual core 680mhz mips platform, but has a lot of single-threadedness and possible other problems. Work on it contin= ues over on the relevant ubnt forums, but I haven't had time to profile the ker= nel in any case to look hardware at where the limits are coming from. Hope some= one else does with more time and chops. > Or should I start looking for something like this: > > http://www.gateworks.com/product/item/ventana-gw5310-network-processor > > (although that's an expensive board, given the very low production volume= , > for the same cost I could probably build a small passively-cooled > mini/micro-atx setup running x86 and dual NICs). There is that option as well. I would certainly like to find a low end x86 = box that could rate limit + fq_codel at up to 300Mbits/sec. Toke's x86 boxes have proven out to do 100Mbit/10Mbit correctly, but I don't remember their specs, nor has he tried to push them past that, yet. > -Aaron > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --=20 Dave T=C3=A4ht NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_= indecent.article