From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp201.iad.emailsrvr.com (smtp201.iad.emailsrvr.com [207.97.245.201]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 8FBAA202295 for ; Wed, 22 Aug 2012 13:44:54 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp50.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 2A22D37131D; Wed, 22 Aug 2012 16:44:53 -0400 (EDT) X-Virus-Scanned: OK Received: from legacy3.wa-web.iad1a (legacy3.wa-web.iad1a.rsapps.net [192.168.2.219]) by smtp50.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 0CF093710A4; Wed, 22 Aug 2012 16:44:53 -0400 (EDT) Received: from reed.com (localhost [127.0.0.1]) by legacy3.wa-web.iad1a (Postfix) with ESMTP id E0A2521680AC; Wed, 22 Aug 2012 16:44:52 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Wed, 22 Aug 2012 16:44:52 -0400 (EDT) Date: Wed, 22 Aug 2012 16:44:52 -0400 (EDT) From: dpreed@reed.com To: "Dave Taht" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20120822164452000000_31452" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: Message-ID: <1345668292.918911017@apps.rackspace.com> X-Mailer: webmail7.0 Cc: cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] =?utf-8?q?Coping_with_router_memory_limitations_i?= =?utf-8?q?n_fq=5Fcodel?= X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 20:44:55 -0000 ------=_20120822164452000000_31452 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0ASince there seems to be some confusion about what aspect of bufferbloat = fq_codel controls, perhaps somewhere in a HOWTO somebody should talk about = how to combine tbf (for the upstream "cable modem" case) with fq_codel on= cerowrt today - though I think the contemplated mfq_codel idea mentioned b= elow would be nice, but it would have to be parameterized, which codel need= not be.=0A =0AFor even better user experience, perhaps Luci could have a "= wizard" that asks a simple question like:=0A =0Ais your upstream a cable mo= dem with bufferbloat? ( ) yes ( ) no=0A =0Awhat is its rated "uplink" bitra= te? _____ Mb/sec=0A =0AAnd set up the tbf+fq_codel properly for people wh= o want simple setup. Reasonable tbf parameters should be easy to calculat= e given the rate.=0A =0A-----Original Message-----=0AFrom: "Dave Taht" =0ASent: Monday, August 20, 2012 3:12pm=0ATo: "Sebastian M= oeller" =0ACc: cerowrt-devel@lists.bufferbloat.net, "Felix= Fietkau" =0ASubject: [Cerowrt-devel] Coping with router memo= ry limitations in fq_codel=0A=0A=0A=0ADear Sebastian:=0A=0AIn addition to y= our udp flooding DoS attack, I attacked cero also by=0Ausing diffserv marki= ng in netperf (-Y codepoint,codepoint) to saturate=0Aall 4 wifi fq_codel qu= eues, and also would get the router to have=0Amemory allocation failures an= d ultimately crash in the same way you=0Aare crashing it. I can similarly d= o what you just did with rtp=0Aflooding. You are correct that codel is tune= d for tcp, and that=0Afq_codel by maintaining many queues is even more susc= eptible to a=0Atuned udp flooding attack on a memory limited device such as= this.=0A=0AI tried to cope with this in 3.3.8-10/11 by reducing the packet= =0Alimits, which helped a lot. Unfortunately the settings I used then=0Awer= e below codel's reaction time, which invoked "interesting" tail=0Adrop beha= vior, so I arbitrarily doubled them in -17. To invoke more of=0Athe kind of= problems you are encountering...=0A=0A0) Since then I have been looking in= to ways to improve codel's=0Areaction time that are in the ns2 model presen= tly, also fixing an=0Aassumption about newton's method that didn't hold in = reverse, and also=0Ameans to incorporate more aggressive codel behavior whe= n queue limits=0Aare near to being exceeded.=0A=0AUnfortunately as the memo= ry pressure problem starts in the driver,=0Ait's not communicated up the st= ack to where it could be controlled=0Abetter...=0A=0A0) I would like avoid = having to determining if a queue is tcp or=0A"other", and then having diffe= rent kinds of drop strategies for each.=0AThat said, it seems possible to i= mplement that...=0A=0A1) A workaround of sorts for the 64MB 3700v2 has been= to give up on=0Anamed and get some memory back that way.=0A=0A2) I believe= , but am not sure, that Linux 3.6(5?) has some stuff in it=0Ato get skb mem= ory allocations done more efficiently. Eric and I and=0Afelix had talked ab= out it, I don't know what was implemented.=0A=0A3) It may be possible to im= prove how the memory allocations from the=0A2048 slab work in general. I im= agine that half of memory is being=0Awasted on big packets otherwise.=0A=0A= 4) some options for improving fq_codel for more memory constrained=0Ahome e= nvironments better.=0A=0A4a) On the wifi front (as well as other devices wi= th multiple hardware=0Aqueues), I envision something like "mfq_codel", whic= h would have an=0Aoverall similar packet limit to a single fq_codel, but be= able to=0Adeliver (and fair queue) packets to the underlying hardware queu= es=0Aindependently.=0A=0A4b) On the home to-ISP gateway qos front, a rate l= imited (tbf)=0Amfq_codel with 2-4 queues would replace the complexity of hf= sc or htb=0Awith a default qdisc that "just worked" without any scripting. = It=0Acould be mildly more responsive (htb buffers up some data and has it's= =0Aown notion of time and quantums), thus cpu and memory usage would be=0Al= ower than htb + multiple fq_codel queues.=0A=0AGetting something that scale= d down to 10s of kbits and up to gigabits=0Awould be hard, tho. HTB needs t= o be tuned when running lower or higher=0Athan it's original operating rang= e, presently, and that is where, in=0Apart, the simple_qos.sh effort is "st= uck".=0A=0A4c) Another thought would be to have a weighted packet (to handl= e=0Aclassification) oriented sfq codel or qfq_codel rather than separate=0A= fq_codel queues that are each byte-aware... we have CPU to burn, but=0Anot = memory...=0A=0AOn Mon, Aug 20, 2012 at 11:24 AM, Sebastian Moeller wrote:=0A> Hi Dave,=0A>=0A> so I went to play around with this a = bit more. I turned to UDP flooding my cable modem through the router and th= is surely allows me to create enough load on the wndr3700v2 to cause the al= location errors and as a "bonus" also to drive the router to reboot (driven= by the watchdog timer?). Here is the script I used over 5G wireless (from = http://blog.ioshints.info/2008/03/udp-flood-in-perl.html)=0A>=0A> #!/usr/bi= n/perl=0A=0AIt would be nice to have a C or lua version of this sort of tes= t.=0A=0A> ##############=0A>=0A> # udp flood.=0A> ##############=0A>=0A> us= e Socket;=0A> use strict;=0A>=0A> if ($#ARGV !=3D 3) {=0A> print "flood.p= l