From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-02-iad.dyndns.com (mxout-025-iad.mailhop.org [216.146.32.25]) by lists.bufferbloat.net (Postfix) with ESMTP id 5CBA32E032F for ; Fri, 25 Feb 2011 03:54:48 -0800 (PST) Received: from scan-02-iad.mailhop.org (scan-02-iad.local [10.150.0.207]) by mail-02-iad.dyndns.com (Postfix) with ESMTP id 4072C833441 for ; Fri, 25 Feb 2011 11:54:34 +0000 (UTC) X-Spam-Score: 1.2 (+) X-Mail-Handler: MailHop by DynDNS X-Originating-IP: 209.85.161.43 Received: from mail-fx0-f43.google.com (mail-fx0-f43.google.com [209.85.161.43]) by mail-02-iad.dyndns.com (Postfix) with ESMTP id 9D2968335C9; Fri, 25 Feb 2011 11:54:33 +0000 (UTC) Received: by fxm9 with SMTP id 9so1987979fxm.16 for ; Fri, 25 Feb 2011 03:54:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:from:to:cc:in-reply-to:references :content-type:date:message-id:mime-version:x-mailer :content-transfer-encoding; bh=maTT/5IZL9MMGBk9KBgn3n0uNlO/weolRaLuELw1q+A=; b=JC79CpfEX1EE5Yww8+BBl4iUf9A3+Y1dR3N9q/jOH4g57EMtcm1IqeBI5ZvFG37oR6 QoXm8yFwVG1rNZNREcEu1m2FIfnaL0LHcWWk8S4+wh+2wHYEQ/NyW2g0AhDYFiLtZLNm WW3Dy54Tw9SoeraSjtHqxqIzp6HOjH+pNDXfo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:from:to:cc:in-reply-to:references:content-type:date :message-id:mime-version:x-mailer:content-transfer-encoding; b=IPq/0CFGUf0hhIywkiauXl9RFrg/LsZkhXnn/QqPVB5EV5+YMruVNYJI6w9mRtR35Y YW00uzzOEak1W8W13qJH85KT4HHTnrZ39Mflulw/YZZg5RKZbg+HI0svYXJBFL7yvUrt /Qb0kXffCxsWxaZRGegzZEk5tbLQsvVpkCcjA= Received: by 10.223.53.68 with SMTP id l4mr2560626fag.44.1298634865574; Fri, 25 Feb 2011 03:54:25 -0800 (PST) Received: from [10.150.51.210] (gw0.net.jmsp.net [212.23.165.14]) by mx.google.com with ESMTPS id c11sm236867fav.2.2011.02.25.03.54.22 (version=SSLv3 cipher=OTHER); Fri, 25 Feb 2011 03:54:24 -0800 (PST) From: Eric Dumazet To: Jesper Dangaard Brouer In-Reply-To: <1298632912.21810.33.camel@traveldev.cxnet.dk> References: <4D6668F4.5010705@freedesktop.org> <4D668827.8060508@freedesktop.org> <1298567313.2814.7.camel@edumazet-laptop> <87sjvds2r7.fsf@cruithne.co.teklibre.org> <1298575769.2659.10.camel@edumazet-laptop> <1298632912.21810.33.camel@traveldev.cxnet.dk> Content-Type: text/plain; charset="UTF-8" Date: Fri, 25 Feb 2011 12:54:19 +0100 Message-ID: <1298634859.2659.44.camel@edumazet-laptop> Mime-Version: 1.0 X-Mailer: Evolution 2.30.3 Content-Transfer-Encoding: 8bit Cc: Van Jacobson , bloat-devel@lists.bufferbloat.net, herbert@gondor.apana.org.au, bloat@lists.bufferbloat.net Subject: Re: [Bloat] GSO (was: Please enter issues into the issue tracker - Issue system organisation needed) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Feb 2011 11:54:48 -0000 Le vendredi 25 février 2011 à 12:21 +0100, Jesper Dangaard Brouer a écrit : > On Thu, 2011-02-24 at 20:29 +0100, Eric Dumazet wrote: > > - Its important to set TSO off (ethtool -K eth0 tso off), or else we > > send big packets (up to 64Kbytes) and this used to break SFQ fairness. > > This can really hurt latencies of interactive flows. > > Don't you mean "GSO" Generic-Segmentation-Offload (ethtool -K eth0 gso > off) as this happens in the stack. While TSO Tcp-Segmentation-Offload > happens in hardware, and you will not see it in the SFQ qdisc? > I definitly see big packets if TSO is enabled, for localy generated trafic. (You probably are concerned by routers, where all trafic is forwarded, so TSO is not used, even if enabled) > I recommend that both is turned off, on small bandwidth links where > latency matters. > Sure. > I'm wondering if LRO (Large-Receive-Offload) affect you, when you are > using SFQ on ingress? > > GRO/LRO can have an impact, for sure. But most 'current' kernels dont have GRO/LRO by default. I mean, kernels in use by 2-3 years old distros. > Recently had some "funny" issues with GRO, where a 100 Mbit/s customer > could "only" get approx 90 Mbit/s throughput to our speed test server > (other customers, in another appartment building could get approx 96 > Mbit/s). The issue was resolved by disabling GSO on the speed test > server. The theory is that some switch on the path cannot handle the > bursts generated by GSO, which is max 64K (I think, correct me if I'm > wrong). > Thats right. One 64K packet with standard MTU means some spikes on wire, but if your switches cant resist to this... Is TCP SACK active on the customer side (and speed test server) ? > When adjusting buffer sizes, its important to take this bursty TCP > behavior into account, which is created by both GSO and TSO. I'm not > saying that the queue size needs to be above 64K. For smaller links, it > might make sense to set it, significantly below 64K, to avoid a GSO > enabled Linux machine to ramp up its window size, which makes it capable > of bursting. > TSO basically hurts SFQ or other AQM, unless you use big/fast pipes. For a router workload anyway, I would say its better to not try to coalesce frames in software level, just handle them one by one.