<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
font-variant:normal !important;
text-transform:none;
text-decoration:none none;
vertical-align:baseline;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
font-variant:normal !important;
color:windowtext;
text-transform:none;
text-decoration:none none;
vertical-align:baseline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thanks Greg,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But wouldn’t this potentially violate at least one MUST from section 5.2 of l4s-id?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"> The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"> be roughly proportional to the square of the likelihood that it would<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black"> have marked it if it had been an L4S packet (p_L)<o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5.2">https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5.2</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Maybe it depends on how far you stretch “roughly”, I guess... I’m not sure, but I’d imagine some realistic conditions could provide counterexamples, unless there’s some reason I’m missing that prevents it?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Jake<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Greg White <g.white@CableLabs.com><br>
<b>Date: </b>2019-03-20 at 14:49<br>
<b>To: </b>"Holland, Jake" <jholland@akamai.com>, Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com><br>
<b>Cc: </b>tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net><br>
<b>Subject: </b>Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">I responded to point 2 separately. In response to point 1, the dual queue mechanism is not the only way to support L4S and TCP Prague. As we’ve mentioned a time or two, an fq_codel system can also support L4S
and TCP Prague. I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">At dequeue:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">If packet indicates ECT(1):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> If sojourn_time > L4S_threshold:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Set CE*, and forward packet<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Else:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Forward packet<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Else:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> Do all the normal CoDel stuff<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will. But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including
the ECN field in the flow hash would keep those packets separate. <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">*a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "Holland, Jake" <jholland@akamai.com><br>
<b>Date: </b>Wednesday, March 20, 2019 at 1:06 PM<br>
<b>To: </b>Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com><br>
<b>Cc: </b>tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net><br>
<b>Subject: </b>Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:1.0in">Hi Bob & Greg,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I agree there has been a reasonably open conversation about the L4S<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">work, and thanks for all your efforts to make it so.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">However, I think there's 2 senses in which "private" might be fair that<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I didn't see covered in your rebuttals (merging forks and including<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">Greg's rebuttal by reference from here:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">Please note:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I don't consider these senses of "private" a disqualifying argument<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">against the use of L4S, though I do consider them costs that should be<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">taken into account (and of course opinions may differ here).<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">With that said, I wondered whether either of you have any responses that<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">speak to these points:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">1. the L4S use of the ECT(1) codepoint can't be marked CE except by a<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">patent-protected AQM scheduler.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I understand that l4s-id suggests the possibility of an alternate<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">scheme. However, comparing the MUSTs of the section 5 requirements<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">with the claims made by the patent seems to leave no room for an<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">alternate that would not infringe the patent claims, unless I'm missing<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">something?<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">https://patents.google.com/patent/US20170019343A1/en<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">2. the L4S use of the ECT(1) codepoint privileges the low latency use<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">case.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">As Jonathan Morton pointed out, low latency is only one of several<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">known use cases that would be worthwhile to identify to an AQM<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">scheduler, which the L4S approach cannot be extended to handle:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">- Minimize Latency<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">- Minimize Loss<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">- Minimize Cost<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">- Maximize Throughput<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I understand that "Minimize Cost" is perhaps covered by LEPHB, and that<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">operator tuning parameters for a dualq node can possibly allow for<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">tuning between minimizing loss and latency, as mentioned in section<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">4.1 of aqm-dualq, but since the signaling is bundled together, it can<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">only happen for one at a time, if I'm reading it right.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">But more importantly, the L4S usage couples the minimized latency use<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">case to any possibility of getting a high fidelity explicit congestion<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">signal, so the "maximize throughput" use case can't ever get it.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">Jake<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">PS:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">If you get a chance, I'm still interested in seeing answers to my<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">questions about deployment mitigations on the tsvwg list:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">I'm not surprised if it slipped by unnoticed, there have been a lot of<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">emails on this. But good answers to those questions would go a long way<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">toward easing my concerns about the urgency on this discussion.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">PPS:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">These issues are a bit sideways to my technical reasons for preferring<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">the SCE formulation of ECT(1), which have more to do with the confusing<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">semantics and proliferation of corner cases it creates for CE and ECE.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">But I thought I’d ask about them since it seemed like maybe there was a<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in">disconnect here.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Bob Briscoe <ietf@bobbriscoe.net><br>
<b>Date: </b>2019-03-18 at 18:07<br>
<b>To: </b>"David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com><br>
<b>Cc: </b>tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net><br>
<b>Subject: </b>Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:1.5in">
David,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.5in">On 17/03/2019 18:07, David P. Reed wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">Vint -</span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.</span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">It depends on getting reliable current congestion information via packet drops and/or ECN.</span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">What do you mean 'not the cable guys'?<br>
This thread was reasonably civil until this intervention.<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">THe cable guys are trying to get a "private" field in the IP header for their own use.</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-left:1.5in"><br>
There is nothing private about this codepoint, and there never has been. Here's some data points:<br>
<br>
* The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
<br>
* A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
<br>
* Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
<br>
* That was when it was decided that ECT(1) was most appropriate. <br>
* The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
<br>
* David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
<br>
* This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
<br>
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.<br>
<br>
Some history:<br>
* I had been researching the idea since 2012. <br>
* In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.<br>
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper<br>
* Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
<br>
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still
doing her PhD part-time with Simula<br>
o By that time, Al-Lu and Simula had a cool prototype running. <br>
o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=">
https://riteproject.eu/dctth/#1511dispatchwg</a><br>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):<br>
o cloud-rendered remote presence in a racing car within a VR headset<br>
o the interactive cloud-rendered video already linked above<br>
o an online gaming benchmark<br>
o HAS video streaming<br>
o a number of bulk file downloads<br>
o a heavy synthetic load of web browsing<br>
<br>
L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:<br>
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
<br>
* BT was obviously interested in DSL, <br>
* but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
<br>
* In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
<br>
Now at: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=">
https://github.com/L4STeam/sch_dualpi2_upstream</a><br>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows<br>
* In Jul 2016, the main IETF BoF on L4S was held:<br>
o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE<br>
o along with Kevin Smith from Vodafone and <br>
o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).<br>
o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media:
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=">
https://github.com/EricssonResearch/scream/</a><br>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=">
https://github.com/mirjak/linux-accecn</a><br>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S<br>
o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.<br>
o You can find the primary companies involved at the end of the White Paper. <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=">
https://cablela.bs/low-latency-docsis-technology-overview-february-2019</a><br>
o Operators:<br>
Liberty Global<br>
Charter<br>
Rogers<br>
Comcast<br>
Shaw<br>
Cox Communications<br>
o Equipment vendors<br>
ARRIS<br>
Huawei<br>
Broadcom<br>
Intel<br>
Casa<br>
Nokia<br>
Cisco<br>
Videotron<br>
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.<br>
* Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC<br>
* L4S hardware is also being implemented for hi-speed switches at the moment <br>
(the developer wants to announce it themselves, so I have been asked not to identify them).
<br>
<br>
There's a lot more stuff been going on, but I've tried to pick out highlights.<br>
<br>
All this is good healthy development of much lower latency for Internet technology.<br>
<br>
<br>
I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
<br>
<br>
L4S has been open-sourced since 2016 so that everyone can develop it and make it better...<br>
<br>
If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they
have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless
proven otherwise.<br>
<br>
Best Regards<br>
<br>
<br>
<br>
Bob<br>
<br>
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between
the spikes either.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:1.5in;margin-bottom:.0001pt;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:1.5in;overflow-wrap: break-word">
<span style="font-size:12.0pt;font-family:"Arial",sans-serif">-----Original Message-----<br>
From: "Vint Cerf" <a href="mailto:vint@google.com"><vint@google.com></a><br>
Sent: Saturday, March 16, 2019 5:57pm<br>
To: "Holland, Jake" <a href="mailto:jholland@akamai.com"><jholland@akamai.com></a><br>
Cc: "Mikael Abrahamsson" <a href="mailto:swmike@swm.pp.se"><swmike@swm.pp.se></a>, "David P. Reed"
<a href="mailto:dpreed@deepplum.com"><dpreed@deepplum.com></a>, <a href="mailto:ecn-sane@lists.bufferbloat.net">
"ecn-sane@lists.bufferbloat.net"</a> <a href="mailto:ecn-sane@lists.bufferbloat.net">
<ecn-sane@lists.bufferbloat.net></a>, "bloat" <a href="mailto:bloat@lists.bufferbloat.net">
<bloat@lists.bufferbloat.net></a><br>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104</span><o:p></o:p></p>
<div id="SafeStyles1552845686">
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">where does BBR fit into all this?
</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">v</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <<a href="mailto:jholland@akamai.com">jholland@akamai.com</a>> wrote:</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">On 2019-03-15, 11:37, "Mikael Abrahamsson" <<a href="mailto:swmike@swm.pp.se" target="_blank">swmike@swm.pp.se</a>> wrote:<br>
L4S has a much better possibility of actually getting deployment into the <br>
wider Internet packet-moving equipment than anything being talked about <br>
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as <br>
good, but it fits better into actual silicon and it's being proposed by <br>
people who actually have better channels into the people setting hard <br>
requirements.<br>
<br>
I suggest you consider joining them instead of opposing them.<br>
<br>
<br>
Hi Mikael,<br>
<br>
I agree it makes sense that fq_anything has issues when you're talking<br>
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE<br>
makes better sense there.<br>
<br>
But fq_x makes great sense and provides real value for the uplink in a<br>
home, small office, coffee shop, etc. (if you run the final rate limit<br>
on the home side of the access link.) I'm thinking maybe there's a<br>
disconnect here driven by the different use cases for where AQMs can go.<br>
<br>
The thing is, each of these is the most likely congestion point at<br>
different times, and it's worthwhile for each of them to be able to<br>
AQM (and mark packets) under congestion.<br>
<br>
One of the several things that bothers me with L4S is that I've seen<br>
precious little concern over interfering with the ability for another<br>
different AQM in-path to mark packets, and because it changes the<br>
semantics of CE, you can't have both working at the same time unless<br>
they both do L4S.<br>
<br>
SCE needs a lot of details filled in, but it's so much cleaner that it<br>
seems to me there's reasonably obvious answers to all (or almost all) of<br>
those detail questions, and because the semantics are so much cleaner,<br>
it's much easier to tell it's non-harmful.<br>
<br>
<aside regarding="non-harmful"><br>
The point you raised in another thread about reordering is mostly<br>
well-taken, and a good counterpoint to the claim "non-harmful relative<br>
to L4S".<br>
<br>
To me it seems sad and dumb that switches ended up trying to make<br>
ordering guarantees at cost of switching performance, because if it's<br>
useful to put ordering in the switch, then it must be equally useful to<br>
put it in the receiver's NIC or OS.<br>
<br>
So why isn't it in all the receivers' NIC or OS (where it would render<br>
the switch's ordering efforts moot) instead of in all the switches?<br>
<br>
I'm guessing the answer is a competition trap for the switch vendors,<br>
plus "with ordering goes faster than without, when you benchmark the<br>
switch with typical load and current (non-RACK) receivers".<br>
<br>
If that's the case, it seems like the drive for a competitive advantage<br>
caused deployment of a packet ordering workaround in the wrong network<br>
location(s), out of a pure misalignment of incentives.<br>
<br>
RACK rates to fix that in the end, but a lot of damage is already done,<br>
and the L4S approach gives switches a flag that can double as proof that<br>
RACK is there on the receiver, so they can stop trying to order those<br>
packets.<br>
<br>
So point granted, I understand and agree there's a cost to abandoning<br>
that advantage.<br>
</aside><br>
<br>
But as you also said so well in another thread, this is important. ("The<br>
last unicorn", IIRC.) How much does it matter if there's a feature that<br>
has value today, but only until RACK is widely deployed? If you were<br>
convinced RACK would roll out everywhere within 3 years and SCE would<br>
produce better results than L4S over the following 15 years, would that<br>
change your mind?<br>
<br>
It would for me, and that's why I'd like to see SCE explored before<br>
making a call. I think at its core, it provides the same thing L4S does<br>
(a high-fidelity explicit congestion signal for the sender), but with<br>
much cleaner semantics that can be incrementally added to congestion<br>
controls that people are already using.<br>
<br>
Granted, it still remains to be seen whether SCE in practice can match<br>
the results of L4S, and L4S was here first. But it seems to me L4S comes<br>
with some problems that have not yet been examined, and that are nicely<br>
dodged by a SCE-based approach.<br>
<br>
If L4S really is as good as they seem to think, I could imagine getting<br>
behind it, but I don't think that's proven yet. I'm not certain, but<br>
all the comparative analyses I remember seeing have been from more or<br>
less the same team, and I'm not convinced they don't have some<br>
misaligned incentives of their own.<br>
<br>
I understand a lot of work has gone into L4S, but this move to jump it<br>
from interesting experiment to de-facto standard without a more critical<br>
review that digs deeper into some of the potential deployment problems<br>
has me concerned.<br>
<br>
If it really does turn out to be good enough to be permanent, I'm not<br>
opposed to it, but I'm just not convinced that it's non-harmful, and my<br>
default position is that the cleaner solution is going to be better in<br>
the long run, if they can do the same job.<br>
<br>
It's not that I want it to be a fight, but I do want to end up with the<br>
best solution we can get. We only have the one internet.<br>
<br>
Just my 2c. <br>
<br>
-Jake<br>
<br>
<br>
_______________________________________________<br>
Ecn-sane mailing list<br>
<a href="mailto:Ecn-sane@lists.bufferbloat.net" target="_blank">Ecn-sane@lists.bufferbloat.net</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=" target="_blank">https://lists.bufferbloat.net/listinfo/ecn-sane</a></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif"><br>
-- </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">New postal address:
</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Google</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">1875 Explorer Street, 10th Floor</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:12.0pt;font-family:"Arial",sans-serif">Reston, VA 20190</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:1.5in"><br>
<br>
<br>
<br>
<o:p></o:p></p>
<pre style="margin-left:1.5in">_______________________________________________<o:p></o:p></pre>
<pre style="margin-left:1.5in">Bloat mailing list<o:p></o:p></pre>
<pre style="margin-left:1.5in"><a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><o:p></o:p></pre>
<pre style="margin-left:1.5in"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=">https://lists.bufferbloat.net/listinfo/bloat</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-left:1.5in"><br>
<br>
<br>
<br>
<o:p></o:p></p>
<pre style="margin-left:1.5in">-- <o:p></o:p></pre>
<pre style="margin-left:1.5in">________________________________________________________________<o:p></o:p></pre>
<pre style="margin-left:1.5in">Bob Briscoe <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=">http://bobbriscoe.net/</a><o:p></o:p></pre>
</div>
</body>
</html>