From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from oproxy1-pub.bluehost.com (oproxy1-pub.bluehost.com [66.147.249.253]) by huchra.bufferbloat.net (Postfix) with SMTP id D13CB201745 for ; Fri, 13 May 2011 15:28:19 -0700 (PDT) Received: (qmail 9707 invoked by uid 0); 13 May 2011 22:36:43 -0000 Received: from unknown (HELO host291.hostmonster.com) (74.220.215.91) by cpoproxy1.bluehost.com with SMTP; 13 May 2011 22:36:43 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=avanw.com; h=MIME-Version:In-Reply-To:References:Date:Message-ID:Subject:From:To:Content-Type:X-Identified-User; b=VDd36bPJaE5Ua14GcYT+kpdxZO2cnYs1Bg7u2EE2mpW3gagkdA++uIx/GP3j5gs0Ps9I1D1tMAwq/ztInLHadniEA04FY1Y28RJtk6O2hwL3gTO8yob7XATwn5UD4G+B; Received: from mail-fx0-f43.google.com ([209.85.161.43]) by host291.hostmonster.com with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.69) (envelope-from ) id 1QL0yQ-0007xI-P3 for bloat@lists.bufferbloat.net; Fri, 13 May 2011 16:36:43 -0600 Received: by fxm3 with SMTP id 3so3353067fxm.16 for ; Fri, 13 May 2011 15:36:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.101.72 with SMTP id b8mr2412497fao.15.1305326201255; Fri, 13 May 2011 15:36:41 -0700 (PDT) Received: by 10.223.117.209 with HTTP; Fri, 13 May 2011 15:36:41 -0700 (PDT) In-Reply-To: References: <4DB70FDA.6000507@mti-systems.com> <4DC2C9D2.8040703@freedesktop.org> <20110505091046.3c73e067@nehalam> <6E25D2CF-D0F0-4C41-BABC-4AB0C00862A6@pnsol.com> <35D8AC71C7BF46E29CC3118AACD97FA6@srichardlxp2> <1304964368.8149.202.camel@tardy> <4DD9A464-8845-49AA-ADC4-A0D36D91AAEC@cisco.com> <1305297321.8149.549.camel@tardy> <-4629065256951087821@unknownmsgid> Date: Fri, 13 May 2011 16:36:41 -0600 Message-ID: From: Kevin Gross To: bloat@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=00151743f9188ab78b04a32ff022 X-Identified-User: {1416:host291.hostmonster.com:avanwcom:avanw.com} {sentby:smtp auth 209.85.161.43 authed with kevin.gross@avanw.com} Subject: Re: [Bloat] Jumbo frames and LAN buffers (was: RE: Burst Loss) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 May 2011 22:28:20 -0000 --00151743f9188ab78b04a32ff022 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Even through jumbo frames are not standardized, most new network equipment supports them (though generally support is disabled by default). If you IPv= 4 route jumbo packets to a network that doesn't support them, the router will fragment for you. Under IPv6, it is the sender's responsibility to choose a= n MTU that is supported by all networks between source and destination. IPv6 routers do no fragmentation. Although consumer products are often dumbed down, it is not difficult to find switches with comprehensive QoS configurability. Weighted fair queuing is a popular scheme. Strict priority is a bit dangerous but useful for latency-critical applications. The IEEE has just ratified a credit-based algorithm called 802.1av. What I find is missing from all but the high-end equipment is configurability of buffering capacity and behavior. Bad buffering can burn an otherwise competent QoS implementation. In his talks, Jim Gettys claims that these QoS features do not fix bufferbloat - they just move the problem elsewhere. I generally agree with this though I find that moving the problem elsewhere is sometimes a perfectly acceptable solution. Kevin Gross On Fri, May 13, 2011 at 2:21 PM, Dave Taht wrote: > > > On Fri, May 13, 2011 at 2:03 PM, Kevin Gross wrote= : > >> Do we think that bufferbloat is just a WAN problem? I work on live media >> applications for LANs and campus networks. I'm seeing what I think could= be >> characterized as bufferbloat in LAN equipment. The timescales on 1 Gb >> Ethernet are orders of magnitude shorter and the performance problems ca= used >> are in many cases a bit different but root cause and potential solutions >> are, I'm hoping, very similar. >> >> >> >> Keeping the frame byte size small while the frame time has shrunk >> maintains the overhead at the same level. Again, this has been a conscio= us >> decision not a stubborn relic. Ethernet improvements have increased >> bandwidth by orders of magnitude. Do we really need to increase it by a >> couple percentage points more by reducing overhead for large payloads? >> >> >> >> The cost of that improved marginal bandwidth efficiency is a 6x increase >> in latency. Many applications would not notice an increase from 12 us to= 72 >> us for a Gigabit switch hop. But on a large network it adds up, some >> applications are absolutely that sensitive (transaction processing, clus= ter >> computing, SANs) and (I thought I'd be preaching to the choir here) ther= e's >> no way to ever recover the lost performance. >> >> >> > > You are preaching to the choir here, but I note several things: > > Large frame sizes on 10GigE networks to other 10GigE networks is less of = a > problem than 10GigE to 10Mbit networks. I would hope/expect that frame wo= uld > fragment in that case. > > Getting to where latencies are less than 10ms in the general case makes > voip feasible again. I'm still at well over 300ms on bismark. > > Enabling higher speed stock market trades and live music exchange over a > lan would be next on my list after getting below 10ms on the local > switch/wireless interface! > > A lot of research points to widely enabling some form of fair queuing at > the servers and switches to distribute the load at sane levels. (nagle, 8= 9) > I think few gig+e vendors are doing that in hardware, and it would be goo= d > to know who is and who isn't. > > For example, the switch I'm using on bismark has all sorts of wonderful Q= oS > features such as fair queuing, but as best as I can tell they are not > enabled, and I'm seeing buffering in the switch at well above 20ms.... > > It is astonishing that a switch chip this capable has reached the consume= r > marketplace... > > http://realtek.info/pdf/rtl8366s_8366sr_datasheet_vpre-1.4_20071022.pdf > > And depressing that so few of it's capabilities have software to configur= e > them. > >> Kevin Gross >> >> >> >> *From:* Dave Taht [mailto:dave.taht@gmail.com] >> *Sent:* Friday, May 13, 2011 8:54 AM >> *To:* rick.jones2@hp.com >> *Cc:* Kevin Gross; bloat@lists.bufferbloat.net >> *Subject:* Re: [Bloat] Burst Loss >> >> >> >> >> >> On Fri, May 13, 2011 at 8:35 AM, Rick Jones wrote: >> >> On Thu, 2011-05-12 at 23:00 -0600, Kevin Gross wrote: >> > One of the principal reasons jumbo frames have not been standardized >> > is due to latency concerns. I assume this group can appreciate the >> > IEEE holding ground on this. >> >> Thusfar at least, bloaters are fighting to eliminate 10s of milliseconds >> of queuing delay. I don't think this list is worrying about the tens of >> microseconds difference between the transmission time of a 9000 byte >> frame at 1 GbE vs a 1500 byte frame, or the single digit microseconds >> difference at 10 GbE. >> >> >> Heh. With the first iteration of the bismark project I'm trying to get = to >> where I have less than 30ms latency under load and have far larger probl= ems >> to worry about than jumbo frames. I'll be lucky to manage 1/10th that >> (300ms) at this point. >> >> Not, incidentally that I mind the idea of jumbo frames. It seems silly t= o >> be saddled with default frame sizes that made sense in the 70s, and in a= n >> age where we will be seeing ever more packet encapsulation, reducing the >> header size as a ratio to data size strikes me as a very worthy goal. >> >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> >> > > > -- > Dave T=E4ht > SKYPE: davetaht > US Tel: 1-239-829-5608 > http://the-edge.blogspot.com > --00151743f9188ab78b04a32ff022 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Even through jumbo frames are not standardized, most new network equipment = supports them (though generally support is disabled by default). If you IPv= 4 route jumbo packets to a network that doesn't support them, the route= r will fragment for you. Under IPv6, it is the sender's responsibility = to choose an MTU that is supported by all networks between source and desti= nation. IPv6 routers do no fragmentation.

Although consumer products are often dumbed down, it is not = difficult to find switches with comprehensive QoS configurability.=A0Weight= ed=A0fair=A0queuing=A0is a popular scheme. Strict=A0priority=A0is a bit dan= gerous but useful for latency-critical applications. The IEEE has just rati= fied a credit-based algorithm called 802.1av. What I find is missing from a= ll but the high-end equipment is configurability of buffering capacity and = behavior. Bad buffering can burn an otherwise competent QoS implementation.=

In his talks, Jim Gettys claims that these QoS features= do not fix bufferbloat - they just move the problem elsewhere. I generally= agree with this though I find that moving the problem elsewhere is=A0somet= imes=A0a perfectly acceptable solution.

Kevin Gross


On Fri, May 13, 2011 at 2:21 PM, Dave Taht <dave.taht@gmail.com> wrote:


On Fri, May 13, 2011 at 2:03 PM, Kevin G= ross <kevin.gross@avanw.com> wrote:

Do we think that = bufferbloat is just a WAN problem? I work on live media applications for LA= Ns and campus networks. I'm seeing what I think could be characterized = as bufferbloat in LAN equipment. The timescales on 1 Gb Ethernet are orders= of magnitude shorter and the performance problems caused are in many cases= a bit different but root cause and potential solutions are, I'm hoping= , very similar.

Keeping the frame byte size small while the frame time ha= s shrunk maintains the overhead at the same level. Again, this has been a c= onscious decision not a stubborn relic. Ethernet improvements have increase= d bandwidth by orders of magnitude. Do we really need to increase it by a c= ouple percentage points more by reducing overhead for large payloads?

=A0

The cost of that improved mar= ginal bandwidth efficiency is a 6x increase in latency. Many applications w= ould not notice an increase from 12 us to 72 us for a Gigabit switch hop. B= ut on a large network it adds up, some applications are absolutely that sen= sitive (transaction processing, cluster computing, SANs) and (I thought I&#= 39;d be preaching to the choir here) there's no way to ever recover the= lost performance.

http://realtek.info/pd= f/rtl8366s_8366sr_datasheet_vpre-1.4_20071022.pdf

And depressing that so few of it's capabilities have software to co= nfigure them.

Kevin Gross

=A0

F= rom: Dave Taht [mailto:dave.taht@gmail.com] Sent: Friday, May 13, 2011 8:54 AM
To: rick.jones2@hp.com
Cc: K= evin Gross; bloat@lists.bufferbloat.net
Subject: Re: [Bloat] Burst Loss

=A0

=A0

<= p class=3D"MsoNormal">On Fri, May 13, 2011 at 8:35 AM, Rick Jones <rick.jones2@hp.com&g= t; wrote:

On Thu, 2011-05-12= at 23:00 -0600, Kevin Gross wrote:
> One of the principal reasons ju= mbo frames have not been standardized
> is due to latency concerns. I= assume this group can appreciate the
> IEEE holding ground on this.

Thusfar a= t least, bloaters are fighting to eliminate 10s of milliseconds
of queui= ng delay. =A0I don't think this list is worrying about the tens of
microseconds difference between the transmission time of a 9000 byte
frame at 1 GbE vs a 1500 byte frame, or the single digit microseconds
di= fference at 10 GbE.


Heh.=A0 With the first iteration of the bismark project I'm tr= ying to get to where I have less than 30ms latency under load and have far = larger problems to worry about than jumbo frames. I'll be lucky to mana= ge 1/10th that (300ms) at this point.

Not, incidentally that I mind the idea of jumbo frames. It seems silly = to be saddled with default frame sizes that made sense in the 70s, and in a= n age where we will be seeing ever more packet encapsulation, reducing the = header size as a ratio to data size strikes me as a very worthy goal.


_______________________________________________=
Bloat mailing list
Bloat@list= s.bufferbloat.net
= https://lists.bufferbloat.net/listinfo/bloat




--=
Dave T=E4ht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspo= t.com

--00151743f9188ab78b04a32ff022--