From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from g6t0184.atlanta.hp.com (g6t0184.atlanta.hp.com [15.193.32.61]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "smtp.hp.com", Issuer "VeriSign Class 3 Secure Server CA - G3" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id A6BBF2002DE for ; Mon, 16 Apr 2012 10:05:11 -0700 (PDT) Received: from g5t0030.atlanta.hp.com (g5t0030.atlanta.hp.com [16.228.8.142]) by g6t0184.atlanta.hp.com (Postfix) with ESMTP id 8A231C058; Mon, 16 Apr 2012 17:05:09 +0000 (UTC) Received: from [16.89.64.213] (tardy.cup.hp.com [16.89.64.213]) by g5t0030.atlanta.hp.com (Postfix) with ESMTP id 4400714F3B; Mon, 16 Apr 2012 17:05:09 +0000 (UTC) Message-ID: <4F8C5144.60404@hp.com> Date: Mon, 16 Apr 2012 10:05:08 -0700 From: Rick Jones User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.28) Gecko/20120313 Lightning/1.0b2 Thunderbird/3.1.20 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Roger_J=F8rgensen?= References: <20120406213725.GA12641@uio.no> <20120406222138.GB12641@uio.no> <4F88C65C.80209@hp.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] Best practices for paced TCP on Linux? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Apr 2012 17:05:12 -0000 On 04/14/2012 02:06 PM, Roger Jørgensen wrote: > On Sat, Apr 14, 2012 at 2:35 AM, Rick Jones wrote: >> On 04/06/2012 03:21 PM, Steinar H. Gunderson wrote: >>> On Fri, Apr 06, 2012 at 02:49:38PM -0700, Dave Taht wrote: >>>> However in your environment you will need the beefed up SFQ that is in >>>> 3.3. >>>> and BQL. If you are not saturating that 10GigE card, you can turn off >>>> TSO/GSO >>>> as well. >>> We're not anywhere near saturating our 10GigE card, and even if we did, we >>> could add at least one 10GigE card more. >> >> TSO/GSO isn't so much about saturating the 10 GbE NIC as it is avoiding >> saturating the CPU(s) driving the 10 GbE NIC. That is, they save trips down >> the protocol stack, saving CPU cycles. So, if you are not saturating one or >> more of the CPUs in the system, disabling TSO/GSO should not affect your >> ability to drive bits out the NIC. > > What will happen in a virtual only environment when all the VM's got > more than one 10Gbps and you push close to 10Gbps through each VM? > like heavy iperf between lots of the VM's? I don't know, I run netperf :) > Unless the platform does something that should start to saturate some > of the CPU core's in the entire playform. If the VMs are all on the same system, or there are enough 10 GbEs yes, that would probably start to saturate the CPUs, perhaps even with TSO on (if the VMs have that on their emulated interfaces). Probably lots of time spent moving data around. If though all the VMs are talking out the one 10 Gbps pipe, even with bloat the TCP connections (I'm assuming TCP) will get backed-off and the CPUs won't be at saturation. But the eash with which things like TSO/GSO and LRO/GRO can hide a multitude of path-length sins is why I prefer to use aggregate, burst-mode TCP_RR to measure scalability - lots and lots of trips up and down the protocol stack. (I should probably switch the example to TCP_RR - http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-_002d_002denable_002ddemo ) rick jones