From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sj-iport-2.cisco.com (sj-iport-2.cisco.com [171.71.176.71]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "sj-iport-2.cisco.com", Issuer "Cisco SSCA" (not verified)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 623CD2012EB for ; Sat, 14 May 2011 13:39:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=fred@cisco.com; l=22169; q=dns/txt; s=iport; t=1305406118; x=1306615718; h=subject:mime-version:from:in-reply-to:date:cc:message-id: references:to; bh=L/u3hI9wQsUnLYAmD001SlJlXcIIEu2Tqqj+jlc0+Aw=; b=kLe7QYLWZ7D1fbtbi0pS0GGxahACnSDkK7GXB8sFFKQSlWK0c6ihaoTz vQZMabx5Cgxwxvtn09iJ6lv1zs/8dPErVfGNAoLjHF3b310+CcSV1jZ3D piqPcOQJ4ZXjCjXKtDMx0vNev5PMACcS1x+MxLWEQj6QC64VJdCXP6CWh s=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhQBAHDpzk2rRDoJ/2dsb2JhbACCY5RtjkV3iHChC50UhhkEhlCHdIFNhC+EHIZK X-IronPort-AV: E=Sophos;i="4.64,368,1301875200"; d="scan'208,217";a="357118584" Received: from mtv-core-4.cisco.com ([171.68.58.9]) by sj-iport-2.cisco.com with ESMTP; 14 May 2011 20:48:20 +0000 Received: from stealth-10-32-244-222.cisco.com (stealth-10-32-244-222.cisco.com [10.32.244.222]) by mtv-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id p4EKmF1P025681; Sat, 14 May 2011 20:48:20 GMT Received: from [127.0.0.1] by stealth-10-32-244-222.cisco.com (PGP Universal service); Sat, 14 May 2011 13:48:20 -0700 X-PGP-Universal: processed; by stealth-10-32-244-222.cisco.com on Sat, 14 May 2011 13:48:20 -0700 Mime-Version: 1.0 (Apple Message framework v1084) From: Fred Baker In-Reply-To: <014c01cc11a8$de78ac10$9b6a0430$@gross@avanw.com> Date: Sat, 14 May 2011 13:48:04 -0700 Message-Id: <8A928839-1D91-4F18-8252-F06BD004E37D@cisco.com> References: <4DB70FDA.6000507@mti-systems.com> <4DC2C9D2.8040703@freedesktop.org> <20110505091046.3c73e067@nehalam> <6E25D2CF-D0F0-4C41-BABC-4AB0C00862A6@pnsol.com> <35D8AC71C7BF46E29CC3118AACD97FA6@srichardlxp2> <1304964368.8149.202.camel@tardy> <4DD9A464-8845-49AA-ADC4-A0D36D91AAEC@cisco.com> <1305297321.8149.549.camel@tardy> <014c01cc11a8$de78ac10$9b6a0430$@gross@avanw.com> To: Kevin Gross X-Mailer: Apple Mail (2.1084) Content-Type: multipart/alternative; boundary=Apple-Mail-943-681370044 Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] Jumbo frames and LAN buffers (was: RE: Burst Loss) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 May 2011 20:39:47 -0000 --Apple-Mail-943-681370044 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On May 13, 2011, at 1:03 PM, Kevin Gross wrote: > Do we think that bufferbloat is just a WAN problem? I work on live = media applications for LANs and campus networks. I'm seeing what I think = could be characterized as bufferbloat in LAN equipment. The timescales = on 1 Gb Ethernet are orders of magnitude shorter and the performance = problems caused are in many cases a bit different but root cause and = potential solutions are, I'm hoping, very similar. Bufferbloat is most noticeable on WANs, because they have longer delays, = but yes LAN equipment does the same thing. It shows up as extended delay = or as an increase in loss rates. A lot of LAN equipment has very shallow = buffers due to cost (LAN markets are very cost-sensitive). One myth with = bufferbloat is that a reasonable solution is to make the buffer shallow; = no, because when the queue fills you now have an increased loss rate, = which shows up in timeout-driven retransmissions - you really want a = deep buffer (for bursts and temporary surges) that you keep shallow = using AQM techniques. > Keeping the frame byte size small while the frame time has shrunk = maintains the overhead at the same level. Again, this has been a = conscious decision not a stubborn relic. Ethernet improvements have = increased bandwidth by orders of magnitude. Do we really need to = increase it by a couple percentage points more by reducing overhead for = large payloads? You might talk with folks who do the LAN Speed records. They generally = view end to end jumboframes as material to the achievement. It's not = about changing the serialization delay, it's about changing the amount = of processing at the endpoints. > The cost of that improved marginal bandwidth efficiency is a 6x = increase in latency. Many applications would not notice an increase from = 12 us to 72 us for a Gigabit switch hop. But on a large network it adds = up, some applications are absolutely that sensitive (transaction = processing, cluster computing, SANs) and (I thought I'd be preaching to = the choir here) there's no way to ever recover the lost performance. Well, the extra delay is solvable in the transport. The question isn't = really what the impact on the network is; it's what the requirements of = the application are. For voice, if a voice sample is delayed 50 ms the = jitter buffer in the codec resolves that - microseconds are irrelevant. = Video codecs generally keep at least three video frames in their jitter = buffer; at 30 fps, that's 100 milliseconds of acceptable variation in = delay. milliseconds.=20 Where it gets dicey is in elastic applications (applications using = transports with the characteristics of TCP) that are retransmitting or = otherwise reacting in timeframes comparable to the RTT and the RTT is = small, or in elastic applications in which the timeout-retransmission = interval is on the order of hundreds of milliseconds to seconds (true of = most TCPs) but the RTT is on the order of microseconds to milliseconds. = In the former, a deep queue buildup and trigger a transmission that = further builds the queue; in the latter, a hiccup can have dramatic side = effects. There is ongoing research on how best to do such things in data = centers. My suspicion is that the right approach is something akin to = 802.2 at the link layer, but with NACK retransmission - system A = enumerates the data it sends to system B, and if system B sees a number = skip it asks A to retransmit the indicated datagram. You might take a = look at RFC 5401/5740/5776 for implementation suggestions.=20 > Kevin Gross > =20 > From: Dave Taht [mailto:dave.taht@gmail.com]=20 > Sent: Friday, May 13, 2011 8:54 AM > To: rick.jones2@hp.com > Cc: Kevin Gross; bloat@lists.bufferbloat.net > Subject: Re: [Bloat] Burst Loss > =20 > =20 >=20 > On Fri, May 13, 2011 at 8:35 AM, Rick Jones = wrote: > On Thu, 2011-05-12 at 23:00 -0600, Kevin Gross wrote: > > One of the principal reasons jumbo frames have not been standardized > > is due to latency concerns. I assume this group can appreciate the > > IEEE holding ground on this. >=20 > Thusfar at least, bloaters are fighting to eliminate 10s of = milliseconds > of queuing delay. I don't think this list is worrying about the tens = of > microseconds difference between the transmission time of a 9000 byte > frame at 1 GbE vs a 1500 byte frame, or the single digit microseconds > difference at 10 GbE. >=20 > Heh. With the first iteration of the bismark project I'm trying to = get to where I have less than 30ms latency under load and have far = larger problems to worry about than jumbo frames. I'll be lucky to = manage 1/10th that (300ms) at this point.=20 >=20 > Not, incidentally that I mind the idea of jumbo frames. It seems silly = to be saddled with default frame sizes that made sense in the 70s, and = in an age where we will be seeing ever more packet encapsulation, = reducing the header size as a ratio to data size strikes me as a very = worthy goal. >=20 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --Apple-Mail-943-681370044 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On May 13, 2011, at 1:03 PM, Kevin = Gross wrote:

Do we = think that bufferbloat is just a WAN problem? I work on live media = applications for LANs and campus networks. I'm seeing what I think could = be characterized as bufferbloat in LAN equipment. The timescales on 1 Gb = Ethernet are orders of magnitude shorter and the performance problems = caused are in many cases a bit different but root cause and potential = solutions are, I'm hoping, very = similar.

Bufferbloat is most noticeable on WANs, = because they have longer delays, but yes LAN equipment does the same = thing. It shows up as extended delay or as an increase in loss rates. A = lot of LAN equipment has very shallow buffers due to cost (LAN markets = are very cost-sensitive). One myth with bufferbloat is that a reasonable = solution is to make the buffer shallow; no, because when the queue fills = you now have an increased loss rate, which shows up in timeout-driven = retransmissions - you really want a deep buffer (for bursts and = temporary surges) that you keep shallow using AQM = techniques.

The = cost of that improved marginal bandwidth efficiency is a 6x increase in = latency. Many applications would not notice an increase from 12 us to 72 = us for a Gigabit switch hop. But on a large network it adds up, some = applications are absolutely that sensitive (transaction processing, = cluster computing, SANs) and (I thought I'd be preaching to the choir = here) there's no way to ever recover the lost = performance.
<= span class=3D"Apple-style-span" style=3D"border-collapse: separate; = font-family: Helvetica; font-style: normal; font-variant: normal; = font-weight: normal; letter-spacing: normal; line-height: normal; = orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; = widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; = -webkit-border-vertical-spacing: 0px; = -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: = auto; -webkit-text-stroke-width: 0px; font-size: medium; ">

Well, the extra delay is solvable in the = transport. The question isn't really what the impact on the network is; = it's what the requirements of the application are. For voice, if a voice = sample is delayed 50 ms the jitter buffer in the codec resolves that - = microseconds are irrelevant. Video codecs generally keep at least three = video frames in their jitter buffer; at 30 fps, that's 100 milliseconds = of acceptable variation in delay. = milliseconds. 

Where it gets dicey is in elastic = applications (applications using transports with the characteristics of = TCP) that are retransmitting or otherwise reacting in timeframes = comparable to the RTT and the RTT is small, or in elastic applications = in which the timeout-retransmission interval is on the order of hundreds = of milliseconds to seconds (true of most TCPs) but the RTT is on the = order of microseconds to milliseconds. In the former, a deep queue = buildup and trigger a transmission that further builds the queue; in the = latter, a hiccup can have dramatic side effects. There is ongoing = research on how best to do such things in data centers. My suspicion is = that the right approach is something akin to 802.2 at the link layer, = but with NACK retransmission - system A enumerates the data it sends to = system B, and if system B sees a number skip it asks A to retransmit the = indicated datagram. You might take a look at RFC 5401/5740/5776 for = implementation suggestions. 

Kevin = Gross
From: Dave Taht = [mailto:dave.taht@gmail.com] 
Sent: Friday, May 13, 2011 8:54 = AM
To: rick.jones2@hp.com
Cc: Kevin Gross;  Re: [Bloat] Burst = Loss

 

On Thu, = 2011-05-12 at 23:00 -0600, Kevin Gross wrote:
> One of the = principal reasons jumbo frames have not been standardized
> is due = to latency concerns. I assume this group can appreciate the
> IEEE = holding ground on this.

Thusfar at = least, bloaters are fighting to eliminate 10s of milliseconds
of = queuing delay.  I don't think this list is worrying about the tens = of
microseconds difference between the transmission time of a 9000 = byte
frame at 1 GbE vs a 1500 byte frame, or the single digit = microseconds
difference at 10 GbE.


Heh.  With the first iteration of = the bismark project I'm trying to get to where I have less than 30ms = latency under load and have far larger problems to worry about than = jumbo frames. I'll be lucky to manage 1/10th that (300ms) at this = point. 

Not, = incidentally that I mind the idea of jumbo frames. It seems silly to be = saddled with default frame sizes that made sense in the 70s, and in an = age where we will be seeing ever more packet encapsulation, reducing the = header size as a ratio to data size strikes me as a very worthy = goal.

____________________________________= ___________
Bloat mailing list
Bloat@lists.bufferbloat.net