From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.taht.net (mail.taht.net [IPv6:2a01:7e00::f03c:91ff:feae:7028]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5A30E3B260 for ; Wed, 10 Aug 2016 05:45:00 -0400 (EDT) Received: from dair-2430.local (p2003004607349D01DCDE725396A1B33D.dip0.t-ipconnect.de [IPv6:2003:46:734:9d01:dcde:7253:96a1:b33d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.taht.net (Postfix) with ESMTPSA id 34A7621312; Wed, 10 Aug 2016 09:44:59 +0000 (UTC) To: bloat@lists.bufferbloat.net, Stephen Hemminger References: From: =?UTF-8?Q?Dave_T=c3=a4ht?= Message-ID: <80a7042c-7028-b202-20c7-044a672bb9e6@taht.net> Date: Wed, 10 Aug 2016 11:44:57 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Subject: Re: [Bloat] Intel recommends against LRO in routing and bridging. X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Aug 2016 09:45:00 -0000 On 8/9/16 12:13 AM, Aaron Wood wrote: > Just came across this at the top of the README for the ixgbe driver: > (http://downloadmirror.intel.com/22919/eng/README.txt) > > WARNING: The ixgbe driver compiles by default with the LRO (Large Receive > Offload) feature enabled. This option offers the lowest CPU utilization for > receives, but is completely incompatible with *routing/ip forwarding* and > *bridging*. If enabling ip forwarding or bridging is a requirement, it is > necessary to disable LRO using compile time options as noted in the LRO > section later in this document. The result of not disabling LRO when combined > with ip forwarding or bridging can be low throughput or even a kernel panic. I am not sure how true this remains. (?) We certainly have gone through hell with TSO and GRO, but it was my hope that most of those issues were fixed over the past 2 years, in the drivers, and htb/cake/hfsc, etc. I don't think that the existing pie in linux head is handling 64k superpackets[1] well. tbf is doing more of the right things here now,, but wasn't till about a year back. Haven't tracked htb, and certainly hfsc has shown some problems that I don't know are fixed or not (there was a kernel bug filed on it some time back with stephen taking the lead on it) Cake's "peeling" mechanism for superpackets is also overly aggressive, but we have not put much effort into tuning it yet. We have a variety of potential solutions for that... BQL tends to get a very big number for its estimator when superpackets are in use, usually 4x what happens without superpackets turned on via ethtool - but NOT having superpackets turned on can really hurt throughput. The code we tested a while back on the bsd implementations (which generally lack superpacket support) had trouble hitting fifo speeds at high rates on the hardware under test (>400mbits). [1] For want of a word, I'll use superpackets to refer to TSO/GSO/GRO/LSO universally. > > -Aaron > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat >