From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x241.google.com (mail-oi0-x241.google.com [IPv6:2607:f8b0:4003:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 0B0013B25E; Fri, 6 May 2016 14:56:58 -0400 (EDT) Received: by mail-oi0-x241.google.com with SMTP id d139so19439550oig.1; Fri, 06 May 2016 11:56:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=+qzzVnlgOZ0rLzRP+XAQ49h/INh6yEfmttjAXSl5nPY=; b=n7+RGzmgCWLRG+XDj4dIrCvCBPi9zflNerW6Ke+OsGcxvcyhrZMxSo+Gyqncyo5lQS 4eZMFHroMbSK/OZM11XrdlpViugq12rBHLpk6WZkfc4sgPsMXb9P9SSbw6X5hmXUvknd C1lBLfQ64BTdQYWxiflaM2bevozF7yNwLiBySEja2PWikyOaJn6WgkOWkjqKI5tZ6xuH 4ROkvPHe65m3wbuswoSWVZfjeZEGVthurkOVl8Ho2/1H7t0NELBkYv2GCbbqjxbDtbQO iI+e7wlZXFlSUUjDRQCryK7AQHZ3YHrrfFJoPX2CyssxLQQFs4fYfM8ALsCbbjebk1xt cGzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=+qzzVnlgOZ0rLzRP+XAQ49h/INh6yEfmttjAXSl5nPY=; b=JeyeRbgYd9AOfNvldE7UGCPxKrR0cnTHwARrNDCdRqJ+VY67P5TFFwSJ8joAtE+MCH 1JjUSPWqA0kVRhJIexJgaj51ZQGFey+LVDatiaOVr6G1H21FL7twMTKHoq68lkcf0XB7 VUgX1F2YYiOisX3QF5pl/pqfwiHQRenbc82Aeg40gholC8vZ8q3aQM5c5YamCimV63Vw F3qIys6wLkGZaBKiUtEecP44JAys2RlUgIQeVr27QXldnjVZDA5C3BWznd3StecnUmRA +GpsVWJKv9hze6C3uNw4cw3kfP/C3CGMZi6YRUyYJScSOu7S9HBCVqMksLtqHv4k9RiV qydQ== X-Gm-Message-State: AOPr4FXUW5F6RThus5eJQk/fyn+xhKSbPyvv9aD+7D44YKWtQZ/PUSi7lcWcpeeqtskWGlpaS47tydiOPfOgPA== MIME-Version: 1.0 X-Received: by 10.202.186.132 with SMTP id k126mr9074196oif.113.1462561018545; Fri, 06 May 2016 11:56:58 -0700 (PDT) Received: by 10.202.252.9 with HTTP; Fri, 6 May 2016 11:56:58 -0700 (PDT) In-Reply-To: References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> <20160506144740.210901f5@redhat.com> Date: Fri, 6 May 2016 21:56:58 +0300 Message-ID: From: Roman Yeryomin To: Jesper Dangaard Brouer Cc: Felix Fietkau , Dave Taht , Jonathan Morton , "codel@lists.bufferbloat.net" , ath10k , make-wifi-fast@lists.bufferbloat.net, =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , "netdev@vger.kernel.org" , OpenWrt Development List Content-Type: text/plain; charset=UTF-8 Subject: Re: [Codel] OpenWRT wrong adjustment of fq_codel defaults (Was: fq_codel_drop vs a udp flood) X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2016 18:56:59 -0000 On 6 May 2016 at 21:43, Roman Yeryomin wrote: > On 6 May 2016 at 15:47, Jesper Dangaard Brouer wrote: >> >> I've created a OpenWRT ticket[1] on this issue, as it seems that someone[2] >> closed Felix'es OpenWRT email account (bad choice! emails bouncing). >> Sounds like OpenWRT and the LEDE https://www.lede-project.org/ project >> is in some kind of conflict. >> >> OpenWRT ticket [1] https://dev.openwrt.org/ticket/22349 >> >> [2] http://thread.gmane.org/gmane.comp.embedded.openwrt.devel/40298/focus=40335 > > OK, so, after porting the patch to 4.1 openwrt kernel and playing a > bit with fq_codel limits I was able to get 420Mbps UDP like this: > tc qdisc replace dev wlan0 parent :1 fq_codel flows 16 limit 256 Forgot to mention, I've reduced drop_batch_size down to 32 > This is certainly better than 30Mbps but still more than two times > less than before (900). > TCP also improved a little (550 to ~590). > > Felix, others, do you want to see the ported patch, maybe I did something wrong? > Doesn't look like it will save ath10k from performance regression. > >> >> On Fri, 6 May 2016 11:42:43 +0200 >> Jesper Dangaard Brouer wrote: >> >>> Hi Felix, >>> >>> This is an important fix for OpenWRT, please read! >>> >>> OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, >>> without also adjusting q->flows_cnt. Eric explains below that you must >>> also adjust the buckets (q->flows_cnt) for this not to break. (Just >>> adjust it to 128) >>> >>> Problematic OpenWRT commit in question: >>> http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e >>> 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") >>> >>> >>> I also highly recommend you cherry-pick this very recent commit: >>> net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") >>> https://git.kernel.org/davem/net-next/c/9d18562a227 >>> >>> This should fix very high CPU usage in-case fq_codel goes into drop mode. >>> The problem is that drop mode was considered rare, and implementation >>> wise it was chosen to be more expensive (to save cycles on normal mode). >>> Unfortunately is it easy to trigger with an UDP flood. Drop mode is >>> especially expensive for smaller devices, as it scans a 4K big array, >>> thus 64 cache misses for small devices! >>> >>> The fix is to allow drop-mode to bulk-drop more packets when entering >>> drop-mode (default 64 bulk drop). That way we don't suddenly >>> experience a significantly higher processing cost per packet, but >>> instead can amortize this. >>> >>> To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk >>> drop, given we also recommend bucket size to be 128 ? (thus the amount >>> of memory to scan is less, but their CPU is also much smaller). >>> >>> --Jesper >>> >>> >>> On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: >>> >>> > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: >>> > > On 5 May 2016 at 19:12, Eric Dumazet wrote: >>> > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: >>> > > > >>> > > >> >>> > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 >>> > > >> quantum 1514 target 5.0ms interval 100.0ms ecn >>> > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) >>> > > >> backlog 0b 0p requeues 0 >>> > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 >>> > > >> new_flows_len 0 old_flows_len 0 >>> > > > >>> > > > >>> > > > Limit of 1024 packets and 1024 flows is not wise I think. >>> > > > >>> > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, >>> > > > which is almost the same than having no queue at all) >>> > > > >>> > > > I suggest to have at least 8 packets per bucket, to let Codel have a >>> > > > chance to trigger. >>> > > > >>> > > > So you could either reduce number of buckets to 128 (if memory is >>> > > > tight), or increase limit to 8192. >>> > > >>> > > Will try, but what I've posted is default, I didn't change/configure that. >>> > >>> > fq_codel has a default of 10240 packets and 1024 buckets. >>> > >>> > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 >>> > >>> > If someone changed that in the linux variant you use, he probably should >>> > explain the rationale. >> >> -- >> Best regards, >> Jesper Dangaard Brouer >> MSc.CS, Principal Kernel Engineer at Red Hat >> Author of http://www.iptv-analyzer.org >> LinkedIn: http://www.linkedin.com/in/brouer