From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 631DD3B25E; Fri, 6 May 2016 08:47:47 -0400 (EDT) Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5DBF63DD47; Fri, 6 May 2016 12:47:46 +0000 (UTC) Received: from localhost (ovpn-200-47.brq.redhat.com [10.40.200.47]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u46Clf6D029946; Fri, 6 May 2016 08:47:42 -0400 Date: Fri, 6 May 2016 14:47:40 +0200 From: Jesper Dangaard Brouer To: Felix Fietkau , Dave Taht Cc: Roman Yeryomin , Jonathan Morton , "codel@lists.bufferbloat.net" , ath10k , make-wifi-fast@lists.bufferbloat.net, zajec5@gmail.com, "netdev@vger.kernel.org" , brouer@redhat.com, openwrt-devel@lists.openwrt.org Message-ID: <20160506144740.210901f5@redhat.com> In-Reply-To: <20160506114243.4eb4f95e@redhat.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 Subject: Re: [Make-wifi-fast] OpenWRT wrong adjustment of fq_codel defaults (Was: [Codel] fq_codel_drop vs a udp flood) X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2016 12:47:47 -0000 I've created a OpenWRT ticket[1] on this issue, as it seems that someone[2] closed Felix'es OpenWRT email account (bad choice! emails bouncing). Sounds like OpenWRT and the LEDE https://www.lede-project.org/ project is in some kind of conflict. OpenWRT ticket [1] https://dev.openwrt.org/ticket/22349 [2] http://thread.gmane.org/gmane.comp.embedded.openwrt.devel/40298/focus=40335 On Fri, 6 May 2016 11:42:43 +0200 Jesper Dangaard Brouer wrote: > Hi Felix, > > This is an important fix for OpenWRT, please read! > > OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, > without also adjusting q->flows_cnt. Eric explains below that you must > also adjust the buckets (q->flows_cnt) for this not to break. (Just > adjust it to 128) > > Problematic OpenWRT commit in question: > http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e > 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") > > > I also highly recommend you cherry-pick this very recent commit: > net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") > https://git.kernel.org/davem/net-next/c/9d18562a227 > > This should fix very high CPU usage in-case fq_codel goes into drop mode. > The problem is that drop mode was considered rare, and implementation > wise it was chosen to be more expensive (to save cycles on normal mode). > Unfortunately is it easy to trigger with an UDP flood. Drop mode is > especially expensive for smaller devices, as it scans a 4K big array, > thus 64 cache misses for small devices! > > The fix is to allow drop-mode to bulk-drop more packets when entering > drop-mode (default 64 bulk drop). That way we don't suddenly > experience a significantly higher processing cost per packet, but > instead can amortize this. > > To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk > drop, given we also recommend bucket size to be 128 ? (thus the amount > of memory to scan is less, but their CPU is also much smaller). > > --Jesper > > > On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: > > > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: > > > On 5 May 2016 at 19:12, Eric Dumazet wrote: > > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: > > > > > > > >> > > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 > > > >> quantum 1514 target 5.0ms interval 100.0ms ecn > > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) > > > >> backlog 0b 0p requeues 0 > > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > > > >> new_flows_len 0 old_flows_len 0 > > > > > > > > > > > > Limit of 1024 packets and 1024 flows is not wise I think. > > > > > > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, > > > > which is almost the same than having no queue at all) > > > > > > > > I suggest to have at least 8 packets per bucket, to let Codel have a > > > > chance to trigger. > > > > > > > > So you could either reduce number of buckets to 128 (if memory is > > > > tight), or increase limit to 8192. > > > > > > Will try, but what I've posted is default, I didn't change/configure that. > > > > fq_codel has a default of 10240 packets and 1024 buckets. > > > > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 > > > > If someone changed that in the linux variant you use, he probably should > > explain the rationale. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer