<html><head><base href="x-msg://1611/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On May 13, 2011, at 1:03 PM, Kevin Gross wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Do we think that bufferbloat is just a WAN problem? I work on live media applications for LANs and campus networks. I'm seeing what I think could be characterized as bufferbloat in LAN equipment. The timescales on 1 Gb Ethernet are orders of magnitude shorter and the performance problems caused are in many cases a bit different but root cause and potential solutions are, I'm hoping, very similar.<o:p></o:p></span></div></div></div></span></blockquote><div><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;">Bufferbloat is most noticeable on WANs, because they have longer delays, but yes LAN equipment does the same thing. It shows up as extended delay or as an increase in loss rates. A lot of LAN equipment has very shallow buffers due to cost (LAN markets are very cost-sensitive). One myth with bufferbloat is that a reasonable solution is to make the buffer shallow; no, because when the queue fills you now have an increased loss rate, which shows up in timeout-driven retransmissions - you really want a deep buffer (for bursts and temporary surges) that you keep shallow using AQM techniques.</span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div></div></div></span></div><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Keeping the frame byte size small while the frame time has shrunk maintains the overhead at the same level. Again, this has been a conscious decision not a stubborn relic. Ethernet improvements have increased bandwidth by orders of magnitude. Do we really need to increase it by a couple percentage points more by reducing overhead for large payloads?</span></div></div></div></span></blockquote><div><br></div>You might talk with folks who do the LAN Speed records. They generally view end to end jumboframes as material to the achievement. It's not about changing the serialization delay, it's about changing the amount of processing at the endpoints.</div><div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The cost of that improved marginal bandwidth efficiency is a 6x increase in latency. Many applications would not notice an increase from 12 us to 72 us for a Gigabit switch hop. But on a large network it adds up, some applications are absolutely that sensitive (transaction processing, cluster computing, SANs) and (I thought I'd be preaching to the choir here) there's no way to ever recover the lost performance.<o:p></o:p></span></div></div></div></span></blockquote><div><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;">Well, the extra delay is solvable in the transport. The question isn't really what the impact on the network is; it's what the requirements of the application are. For voice, if a voice sample is delayed 50 ms the jitter buffer in the codec resolves that - microseconds are irrelevant. Video codecs generally keep at least three video frames in their jitter buffer; at 30 fps, that's 100 milliseconds of acceptable variation in delay. milliseconds. </span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;">Where it gets dicey is in elastic applications (applications using transports with the characteristics of TCP) that are retransmitting or otherwise reacting in timeframes comparable to the RTT and the RTT is small, or in elastic applications in which the timeout-retransmission interval is on the order of hundreds of milliseconds to seconds (true of most TCPs) but the RTT is on the order of microseconds to milliseconds. In the former, a deep queue buildup and trigger a transmission that further builds the queue; in the latter, a hiccup can have dramatic side effects. There is ongoing research on how best to do such things in data centers. My suspicion is that the right approach is something akin to 802.2 at the link layer, but with NACK retransmission - system A enumerates the data it sends to system B, and if system B sees a number skip it asks A to retransmit the indicated datagram. You might take a look at RFC 5401/5740/5776 for implementation suggestions. </span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><font class="Apple-style-span" color="#1F497D" face="Calibri, sans-serif" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div></div></div></span></div><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Kevin Gross<o:p></o:p></span></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; "><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span>Dave Taht [mailto:dave.taht@gmail.com]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, May 13, 2011 8:54 AM<br><b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:rick.jones2@hp.com" style="color: blue; text-decoration: underline; ">rick.jones2@hp.com</a><br><b>Cc:</b><span class="Apple-converted-space"> </span>Kevin Gross;<span class="Apple-converted-space"> </span><a href="mailto:bloat@lists.bufferbloat.net" style="color: blue; text-decoration: underline; ">bloat@lists.bufferbloat.net</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Bloat] Burst Loss<o:p></o:p></span></div></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><p class="MsoNormal" style="margin-top: 0in; margin-right: 0in; margin-bottom: 12pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></p><div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">On Fri, May 13, 2011 at 8:35 AM, Rick Jones <<a href="mailto:rick.jones2@hp.com" style="color: blue; text-decoration: underline; ">rick.jones2@hp.com</a>> wrote:<o:p></o:p></div><div><p class="MsoNormal" style="margin-top: 0in; margin-right: 0in; margin-bottom: 12pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">On Thu, 2011-05-12 at 23:00 -0600, Kevin Gross wrote:<br>> One of the principal reasons jumbo frames have not been standardized<br>> is due to latency concerns. I assume this group can appreciate the<br>> IEEE holding ground on this.<o:p></o:p></p></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">Thusfar at least, bloaters are fighting to eliminate 10s of milliseconds<br>of queuing delay. I don't think this list is worrying about the tens of<br>microseconds difference between the transmission time of a 9000 byte<br>frame at 1 GbE vs a 1500 byte frame, or the single digit microseconds<br>difference at 10 GbE.<o:p></o:p></div><div><p class="MsoNormal" style="margin-top: 0in; margin-right: 0in; margin-bottom: 12pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><br>Heh. With the first iteration of the bismark project I'm trying to get to where I have less than 30ms latency under load and have far larger problems to worry about than jumbo frames. I'll be lucky to manage 1/10th that (300ms) at this point.<span class="Apple-converted-space"> </span><br><br>Not, incidentally that I mind the idea of jumbo frames. It seems silly to be saddled with default frame sizes that made sense in the 70s, and in an age where we will be seeing ever more packet encapsulation, reducing the header size as a ratio to data size strikes me as a very worthy goal.<o:p></o:p></p></div></div></div>_______________________________________________<br>Bloat mailing list<br><a href="mailto:Bloat@lists.bufferbloat.net" style="color: blue; text-decoration: underline; ">Bloat@lists.bufferbloat.net</a><br><a href="https://lists.bufferbloat.net/listinfo/bloat" style="color: blue; text-decoration: underline; ">https://lists.bufferbloat.net/listinfo/bloat</a><br></div></span></blockquote></div><br></body></html>