From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-24-ewr.dyndns.com (mxout-044-ewr.mailhop.org [216.146.33.44]) by lists.bufferbloat.net (Postfix) with ESMTP id E30932E0117 for ; Fri, 25 Feb 2011 13:39:10 -0800 (PST) Received: from scan-21-ewr.mailhop.org (scan-21-ewr.local [10.0.141.243]) by mail-24-ewr.dyndns.com (Postfix) with ESMTP id C76425CD452 for ; Fri, 25 Feb 2011 21:38:51 +0000 (UTC) X-Spam-Score: 0.0 () X-Mail-Handler: MailHop by DynDNS X-Originating-IP: 192.6.19.124 Received: from madara.hpl.hp.com (madara.hpl.hp.com [192.6.19.124]) by mail-24-ewr.dyndns.com (Postfix) with ESMTP id 45C535CC8E5 for ; Fri, 25 Feb 2011 21:38:47 +0000 (UTC) Received: from masterns.hpl.hp.com (masterns.hpl.hp.com [15.0.48.4]) by madara.hpl.hp.com (8.14.4/8.14.4/HPL-PA Relay) with ESMTP id p1PLci7w024139 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 25 Feb 2011 13:38:44 -0800 Received: from bougret.hpl.hp.com (bougret.hpl.hp.com [15.9.72.130]) by masterns.hpl.hp.com (8.14.3/8.14.3/HPL-PA Hub) with ESMTP id p1PLchKu021387 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); Fri, 25 Feb 2011 13:38:43 -0800 Received: from jt by bougret.hpl.hp.com with local (Exim 4.69) (envelope-from ) id 1Pt5N5-0001CS-GO; Fri, 25 Feb 2011 13:38:43 -0800 Date: Fri, 25 Feb 2011 13:38:43 -0800 From: Jean Tourrilhes To: Dave =?iso-8859-1?Q?T=E4ht?= Message-ID: <20110225213843.GA4139@bougret.hpl.hp.com> References: <8739nuy4w1.fsf@cruithne.co.teklibre.org> <20110211162320.GA32058@bougret.hpl.hp.com> <87wrkodv3p.fsf@cruithne.co.teklibre.org> <20110225163956.GB4050@bougret.hpl.hp.com> <87ei6wcb50.fsf@cruithne.co.teklibre.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87ei6wcb50.fsf@cruithne.co.teklibre.org> Organisation: HP Labs Palo Alto Address: HP Labs, 1U-17, 1501 Page Mill road, Palo Alto, CA 94304, USA. E-mail: jt@hpl.hp.com User-Agent: Mutt/1.5.18 (2008-05-17) X-Scanned-By: MIMEDefang 2.71 on 15.0.152.124 Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] do you still exist? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list Reply-To: jt@hpl.hp.com List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Feb 2011 21:39:11 -0000 On Fri, Feb 25, 2011 at 09:49:31AM -0700, Dave Täht wrote: > > I'm deeply concerned about the 10GigE switches coming out, too. The only > one I looked at (the intel) seemed to be using absurd amounts of > buffering. How is the HP gear in this respect? Actually, the switches themselves have very low latency, and many are even cut-through. The ProCurve 5406zl switch, which I've implemented OpenFlow on, is specified with a 2.1us latency for 64B packets at 10GB/s (which is noise compared to the ping RTT measured Linux->Linux). This switch implements WRED and should keep buffer queues fairly small in presence of congestion. To me, the issue most often is in the NIC. Most NIC use large transmit offload and large receive offload to cover the latency of the NIC<->CPU channel. I had to tweak the NIC to hell to get acceptable 10GB/s performance with 1500 B packets (of course, much easier with jumbo). Because the NIC is becoming so bursty, the switch need some buffers to smooth out the bursts. If you look at the time picture, at 10 Gb/s, those buffers don't represent a huge amount of time, especially compared to end to end RTT and TCP timeout. Lower cost 10Gb/s switches will likely have less buffers (for cost saving), and remember that you also have to divide the buffer by the number of ports to get an meaningful comparison. In my opinion, I think you should not be concerned about switches, and more about residential gateways, and I'll explain why. There is a very healthy market for entreprise switches, with many expensive and high end models. Because those high end entreprise switches are expensive, a lot of R&D can be spent on optimising their performance with TCP/IP traffic. Entreprise customers are knowledgeable and demand great performance, and performance is one of the main criteria, and the purchase cost of the device is somewhat trivial compared to operational cost. Then, the technology developped for high end entreprise switches naturally percolate to smaller switches and residential switches. For example, the NetGear GS608 residential switch is specified with 15us max latency, and 128k buffer (16k per port), but I don't know if it implements RED or not. Residential gateway are a totally different market, there is no money, it's all volume with razor thin margin, and the products are renewed very frequently. There is no high end or entreprise version driving the R&D. Consequently, the amount of engineering going into those is limited, and this is highlighted by the fact that most advanced users turn to third party firmware. Also, for those gateway, the number one priority is low cost, followed by cheap price, and then maybe UI, security features and power consumption. Nobody cares about performance and everybody will blame to broadband provider anyway. Another issue is that those devices are implemented using a CPU, and there are latency in the coupling of the NIC and the CPU, due to the way the CPU and OS respond to I/Os. That's something I alluded in an earlier e-mail. Have fun... Jean