<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regarding AQM in North American HFC deployments-<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I also can’t speak for individual Service Providers, but Greg was being modest and the following may be interesting.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The most recent DOCSIS 3.1 specs calls for AQM in the CMTS. It specifically calls for a specific variant of PIE that is designed with the DOCSIS MAC layer
in mind. The DOCSIS 3.0 spec is also being amended to require AQM. Both specs also have recommendations to include AQM in the Cable Modems that can be turned on in the HFC network.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">See
<a href="http://tools.ietf.org/html/draft-white-aqm-docsis-pie-00">http://tools.ietf.org/html/draft-white-aqm-docsis-pie-00</a> for more details.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">bvs<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="543" style="width:407.25pt">
<tbody>
<tr>
<td colspan="3" style="padding:0in 0in 0in 0in">
<p class="MsoNormal"><span style="color:#1F497D"><img border="0" width="110" height="73" id="Picture_x0020_4" src="cid:image001.jpg@01CFC2BE.FB8EFAA0" alt="http://www.cisco.com/web/europe/images/email/signature/logo05.jpg"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td nowrap="" valign="top" style="padding:0in 0in 0in .25in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#666666">Bill Ver Steeg</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#666666"><br>
Distinguished Engineer <br>
Cisco Systems<o:p></o:p></span></p>
</td>
<td width="50%" valign="top" style="width:50.0%;padding:11.25pt 0in 7.5pt 15.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#666666"><br>
<br>
<o:p></o:p></span></p>
</td>
<td style="padding:0in 0in 0in 0in"></td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net]
<b>On Behalf Of </b>Greg White<br>
<b>Sent:</b> Thursday, August 28, 2014 12:36 PM<br>
<b>To:</b> Jerry Jongerius; 'Rich Brown'<br>
<b>Cc:</b> bloat@lists.bufferbloat.net<br>
<b>Subject:</b> Re: [Bloat] The Dark Problem with AQM in the Internet?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">And again, AQM is not causing the problem that you observed. As Jonathan indicated, it would almost certainly make your performance better. I can't speak
for Comcast, but AFAIK they are on a path to deploy AQM. If their customers start raising FUD that could change.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">TCP requires congestion signals. In the vast majority of cases today (and for the foreseeable future) those signals are dropped packets. Going on a witch hunt
to find the evildoer that dropped your packet is counter productive. I think you should instead be asking "why didn't you drop my packet earlier, before the buffer got so bloated and power boost cut the BDP by 60%?"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">-Greg<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Jerry Jongerius <<a href="mailto:jerryj@duckwae.com">jerryj@duckwae.com</a>><br>
<b>Date: </b>Thursday, August 28, 2014 at 10:20 AM<br>
<b>To: </b>'Rich Brown' <<a href="mailto:richb.hanover@gmail.com">richb.hanover@gmail.com</a>><br>
<b>Cc: </b>"<a href="mailto:bloat@lists.bufferbloat.net">bloat@lists.bufferbloat.net</a>" <<a href="mailto:bloat@lists.bufferbloat.net">bloat@lists.bufferbloat.net</a>><br>
<b>Subject: </b>Re: [Bloat] The Dark Problem with AQM in the Internet?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It add accountability. Everyone in the path right now denies that they could possibly be the one dropping the packet.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If I want (or need!) to address the problem, I can’t now. I would have to make a change and just hope that it fixed the problem.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">With accountability, I can address the problem. I then have a choice. If the problem is the ISP, I can switch ISP’s. If the problem is the mid-level peer
or the hosting provider, I can test out new hosting providers.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">- Jerry</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"> Rich Brown [<a href="mailto:richb.hanover@gmail.com">mailto:richb.hanover@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, August 28, 2014 10:39 AM<br>
<b>To:</b> Jerry Jongerius<br>
<b>Cc:</b> Greg White; Sebastian Moeller; <a href="mailto:bloat@lists.bufferbloat.net">
bloat@lists.bufferbloat.net</a><br>
<b>Subject:</b> Re: [Bloat] The Dark Problem with AQM in the Internet?</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Hi Jerry,<o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">AQM<span class="apple-converted-space"> </span><b>is</b><span class="apple-converted-space"> </span>a great solution for bufferbloat. End of story. But if you
want to track down which device in the network intentionally dropped a packet (when many devices in the network path will be running AQM), how are<span class="apple-converted-space"> </span><i>you</i><span class="apple-converted-space"> </span>going to do
that? Or how do<span class="apple-converted-space"> </span><i>you</i>propose to do that?</span><span style="color:black"><o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">Yes, but... I want to understand why you are looking to know which device dropped the packet. What would you do with the information?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The great beauty of fq_codel is that it discards packets that have dwelt too long in a queue by actually *measuring* how long they've been in the queue. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">If the drops happen in your local gateway/home router, then it's interesting to you as the "operator" of that device. If the drops happen elsewhere (perhaps some enlightened ISP has installed fq_codel, PIE, or
some other zoomy queue discipline) then they're doing the right thing as well - they're managing their traffic as well as they can. But once the data leaves your gateway router, you can't make any further predictions.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The SQM/AQM efforts of CeroWrt/fq_codel are designed to give near optimal performance of the *local* gateway, to make it adapt to the remainder of the (black box) network. It might make sense to instrument the
CeroWrt/OpenWrt code to track the number of fq_codel drops to come up with a sense of what's 'normal'. And if you need to know exactly what's happening, then tcpdump/wireshark are your friends. <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Maybe I'm missing the point of your note, but I'm not sure there's anything you can do beyond your gateway. In the broader network, operators are continually watching their traffic and drop rates, and adjusting/reconfiguring
their networks to adapt. But in general, it's impossible for you to have any sway/influence on their operations, so I'm not sure what you would do if you could know that the third router in traceroute was dropping...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Best regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Rich<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>