<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Jake,<br>
<br>
<div class="moz-cite-prefix">On 20/03/2019 19:04, Holland, Jake
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:AC14ACBB-A7CC-40E0-882C-2519D05ADC05@akamai.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
font-variant:normal !important;
color:windowtext;
text-transform:none;
text-decoration:none none;
vertical-align:baseline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">Hi Bob & Greg,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I agree there has been a reasonably open
conversation about the L4S<o:p></o:p></p>
<p class="MsoNormal">work, and thanks for all your efforts to
make it so.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, I think there's 2 senses in which
"private" might be fair that<o:p></o:p></p>
<p class="MsoNormal">I didn't see covered in your rebuttals
(merging forks and including<o:p></o:p></p>
<p class="MsoNormal">Greg's rebuttal by reference from here:<o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html">https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html</a>
)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please note:<o:p></o:p></p>
<p class="MsoNormal">I don't consider these senses of "private"
a disqualifying argument<o:p></o:p></p>
<p class="MsoNormal">against the use of L4S, though I do
consider them costs that should be<o:p></o:p></p>
<p class="MsoNormal">taken into account (and of course opinions
may differ here).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
Thanks for engaging in this. <br>
I do also intend to address one or two other recurring questions on
the bloat list, but I have a lot on at the mo.<br>
<br>
<blockquote type="cite"
cite="mid:AC14ACBB-A7CC-40E0-882C-2519D05ADC05@akamai.com">
<div class="WordSection1">
<p class="MsoNormal">With that said, I wondered whether either
of you have any responses that<o:p></o:p></p>
<p class="MsoNormal">speak to these points:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1. the L4S use of the ECT(1) codepoint
can't be marked CE except by a<o:p></o:p></p>
<p class="MsoNormal">patent-protected AQM scheduler.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I understand that l4s-id suggests the
possibility of an alternate<o:p></o:p></p>
<p class="MsoNormal">scheme. However, comparing the MUSTs of
the section 5 requirements<o:p></o:p></p>
<p class="MsoNormal">with the claims made by the patent seems to
leave no room for an<o:p></o:p></p>
<p class="MsoNormal">alternate that would not infringe the
patent claims, unless I'm missing<o:p></o:p></p>
<p class="MsoNormal">something?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5">https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://patents.google.com/patent/US20170019343A1/en">https://patents.google.com/patent/US20170019343A1/en</a><o:p></o:p></p>
</div>
</blockquote>
1/ In 2016, I arranged for the hire of a patent attorney to
undertake the unusual step of filing a third party observation with
the European Patent Office. This went through Al-Lu's patent
application claim by claim pointing to prior art and giving the
patent examiner all the arguments to reject each claim. However, the
examiner chose to take very little note of it, which was
disappointing and costly for us. The main prior art is:<br>
Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority
Queues," IEEE Transactions on Automatic Control 47(6):1016--1020
(June 2002)<br>
The guys named as inventors in AL-Lu's filing published a paper on
PI2 with me, in which we included a citation of this Gibbens paper
as inspiration for the coupling. The Gibbens paper was already cited
as background by other patents, so the EPO has it in their citation
index.<br>
<br>
The coupling was also based on my prior research with Mirja before I
started working with the guys from Al-Lu in the RITE European
Collaborative project. we had to go through a few rejections, but
Mirja and I finally got this work published in 2014 - still before
the priority date of the Al-Lu patent application:<br>
Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B.,
"Using Data Center TCP (DCTCP) in the Internet," In: Proc. Third
IEEE Globecom Workshop on Telecommunications Standards: From
Research to Standards pp.583-588 (December 2014)<br>
<br>
2/ The only claim that I could not find prior art for (in the
original EU filing) was a very specific claim about using a square
root for the coupling. The Linux implementation runs this the other
way round so that it only has to do a squaring. So I figured we were
safe from that.<br>
<br>
However, until just now, I had not noticed that Al-Lu has
retrospectively re-written the claims in the US patent and in the EU
patent application to claim this the other way round - as a
squaring. And to claim the two random number trick. Both
restructuring to use a squaring and the two random number trick were
definitely my ideas (while working for BT in a collaboration with
Al-Lu). I have emails to prove this (from memory they were actually
both in the same email). This is important, because a patent has to
be about mechanism, not algorithm.<br>
<br>
3/ This is a positive development. It means this patent is on very
shaky legal ground. I have been trying to put pressure on Nokia to
license this royalty free. But now I see what they have done, I am
going to have to get a different type of legal advice. <br>
<br>
<br>
<blockquote type="cite"
cite="mid:AC14ACBB-A7CC-40E0-882C-2519D05ADC05@akamai.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2. the L4S use of the ECT(1) codepoint
privileges the low latency use<o:p></o:p></p>
<p class="MsoNormal">case.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As Jonathan Morton pointed out, low latency
is only one of several<o:p></o:p></p>
<p class="MsoNormal">known use cases that would be worthwhile to
identify to an AQM<o:p></o:p></p>
<p class="MsoNormal">scheduler, which the L4S approach cannot be
extended to handle:<o:p></o:p></p>
<p class="MsoNormal">- Minimize Latency<o:p></o:p></p>
<p class="MsoNormal">- Minimize Loss<o:p></o:p></p>
<p class="MsoNormal">- Minimize Cost<o:p></o:p></p>
<p class="MsoNormal">- Maximize Throughput<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html">https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I understand that "Minimize Cost" is
perhaps covered by LEPHB, and that<o:p></o:p></p>
<p class="MsoNormal">operator tuning parameters for a dualq node
can possibly allow for<o:p></o:p></p>
<p class="MsoNormal">tuning between minimizing loss and latency,
as mentioned in section<o:p></o:p></p>
<p class="MsoNormal">4.1 of aqm-dualq, but since the signaling
is bundled together, it can<o:p></o:p></p>
<p class="MsoNormal">only happen for one at a time, if I'm
reading it right.</p>
</div>
</blockquote>
Not at all. There is a reason why it's called Low Latency, Low Loss,
Scalable throughput (L4S).<br>
<br>
The L4S definition of ECN marking addresses all four of these at
once. Strictly it directly addresses all except minimize cost, but
minimize cost can be built around the system. This was the subject
of my PhD. I haven't described L4S in these terms, because most
people are only interested in the latency. But this is the
underlying reason for my obsession with ECN.<br>
<br>
Frank Kelly predicted that queuing delay would be removed from the
optimization as it was minimized. With L4S we've got very close to
that.<br>
<br>
ECN removes all congestion loss.<br>
<br>
And the use of a inverse linear congestion controller gives the
scalable throughput. I shall be touching on this in my talk for
netdev tomorrow, but it's not really a subject for an implementation
conference.<br>
<br>
Minimize cost is something you do by combining the congestion
signals across a network. So any AQM is part of that. And congestion
controllers are the other part - they implicitly optimize cost,
using the congestion signals as shadow prices. The square root in
classic TCP distorts this, but DCTCP's inverse linear controller
gives proportional fairness directly. Without a weight term in the
congestion controller, there is not really an economic optimization,
but that can be built onto a proportionally fair system and
competition will gradually cause that to happen (or regulation as a
proxy for competition). These are very long term processes though.<br>
<br>
<blockquote type="cite"
cite="mid:AC14ACBB-A7CC-40E0-882C-2519D05ADC05@akamai.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But more importantly, the L4S usage couples
the minimized latency use<o:p></o:p></p>
<p class="MsoNormal">case to any possibility of getting a high
fidelity explicit congestion<o:p></o:p></p>
<p class="MsoNormal">signal, so the "maximize throughput" use
case can't ever get it.
</p>
</div>
</blockquote>
Eh? There's definitely a misunderstanding or a difference in
terminology between us here. The whole point of using a congestion
controller like DCTCP is so that flow rate can scale indefinitely
with capacity. Van Jacobson actually noted that the original TCP was
unscalable in a footnote to the tech report version of the SIGCOMM
paper.<br>
<br>
The high fidelity congestion signal of what we call scalable
congestion controllers (like DCTCP) is inversely proportional to the
window. So as window scales up, the congestion signal scales down,
so that their product remains constant. That means the number of ECN
marks per RTT is scale-invariant. So the control signal remains just
as tight at any scale. <br>
<br>
Cheers<br>
<br>
<br>
<br>
Bob<br>
<br>
<blockquote type="cite"
cite="mid:AC14ACBB-A7CC-40E0-882C-2519D05ADC05@akamai.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Jake<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PS:<o:p></o:p></p>
<p class="MsoNormal">If you get a chance, I'm still interested
in seeing answers to my<o:p></o:p></p>
<p class="MsoNormal">questions about deployment mitigations on
the tsvwg list:<o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A">https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I'm not surprised if it slipped by
unnoticed, there have been a lot of<o:p></o:p></p>
<p class="MsoNormal">emails on this. But good answers to those
questions would go a long way<o:p></o:p></p>
<p class="MsoNormal">toward easing my concerns about the urgency
on this discussion.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PPS:<o:p></o:p></p>
<p class="MsoNormal">These issues are a bit sideways to my
technical reasons for preferring<o:p></o:p></p>
<p class="MsoNormal">the SCE formulation of ECT(1), which have
more to do with the confusing<o:p></o:p></p>
<p class="MsoNormal">semantics and proliferation of corner cases
it creates for CE and ECE.<o:p></o:p></p>
<p class="MsoNormal">But I thought I’d ask about them since it
seemed like maybe there was a<o:p></o:p></p>
<p class="MsoNormal">disconnect here.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span
style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Bob
Briscoe <a class="moz-txt-link-rfc2396E" href="mailto:ietf@bobbriscoe.net"><ietf@bobbriscoe.net></a><br>
<b>Date: </b>2019-03-18 at 18:07<br>
<b>To: </b>"David P. Reed" <a class="moz-txt-link-rfc2396E" href="mailto:dpreed@deepplum.com"><dpreed@deepplum.com></a>,
Vint Cerf <a class="moz-txt-link-rfc2396E" href="mailto:vint@google.com"><vint@google.com></a><br>
<b>Cc: </b>tsvwg IETF list <a class="moz-txt-link-rfc2396E" href="mailto:tsvwg@ietf.org"><tsvwg@ietf.org></a>, bloat
<a class="moz-txt-link-rfc2396E" href="mailto:bloat@lists.bufferbloat.net"><bloat@lists.bufferbloat.net></a><br>
<b>Subject: </b>Re: [Bloat] [Ecn-sane] [iccrg] Fwd:
[tcpPrague] Implementation and experimentation of TCP
Prague/L4S hackaton at IETF104<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">David,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">On 17/03/2019
18:07, David P. Reed wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">Vint
-<o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">BBR
is the end-to-end control logic that adjusts the source
rate to match the share of the bolttleneck link it should
use.<o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">It
depends on getting reliable current congestion information
via packet drops and/or ECN.<o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">So
the proposal by these guys (not the cable guys) is an
attempt to improve the quality of the congestion signal
inserted by the router with the bottleneck outbound link.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">What
do you mean 'not the cable guys'?<br>
This thread was reasonably civil until this intervention.<br>
<br>
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">THe
cable guys are trying to get a "private" field in the IP
header for their own use.<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><br>
There is nothing private about this codepoint, and there never
has been. Here's some data points:<br>
<br>
* The IP header codepoint in question (ECT(1) in the ECN
field) was proposed for use as an alternative ECN behaviour in
July 2105 in the IETF AQM WG and the IETF's transport area WG
(which handles all ECN matters).
<br>
* A year later there followed a packed IETF BoF on the subject
(after 2 open Bar BoFs).
<br>
* Long discussion ensued on the merits of different IP header
field combinations, on both these IETF lists, involving people
active on this list (bloat), including Dave Taht, who is
acknowledged for his contributions in the IETF draft.
<br>
* That was when it was decided that ECT(1) was most
appropriate. <br>
* The logic of the decision is written up in an appendix of
draft-ietf-ecn-l4s-id.
<br>
* David Black, one of the co-chairs of the IETF's transport
area WG and co-author of both the original ECN and Diffserv
RFCs, wrote RFC8311 to lay out the process for reclaiming and
reusing the necessary codepoints.
<br>
* This work and the process of freeing up codepoints has been
very visible at every IETF ever since, with multiple drafts to
fix other aspects of the protocols working their way through
the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
<br>
* L4S has also been mentioned in IETF liaisons with the IEEE
and 3GPP.<br>
<br>
Some history:<br>
* I had been researching the idea since 2012. <br>
* In fact my first presentation on it was scheduled directly
after Van Jacobson's first presentation of CoDel at the IETF
in July 2012. VJ overran by nearly 20mins leaving just 3 mins
for my presentation.<br>
* Mirja Kuehlewind and I did early groundwork in 2013 and
published a paper<br>
* Then I (working for BT) brought the work into the EU RITE
project (Reducing Internet Transport Latency) with Simula,
Alcatel-Lucent, etc.
<br>
* By 2015 the two main L4S proponents were Koen De Schepper
from Alcatel Lucent and myself (I had just switched from BT to
Simula), along with Olga Bondarenko (now Albisser), a PhD
student at Simula who now works for Microsoft (on something
else) and is still doing her PhD part-time with Simula<br>
o By that time, Al-Lu and Simula had a cool prototype
running. <br>
o This was demonstrated publicly for the first time in the
IETF AQM WG over DC+core+backhaul+DSL+home networks.
<br>
<a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e="
moz-do-not-send="true">
https://riteproject.eu/dctth/#1511dispatchwg</a><br>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16
with /every/ packet from all the following simultaneous
applications achieving ~1ms queuing delay or less over a
40Mb/s Internet access link (7ms base RTT):<br>
o cloud-rendered remote presence in a racing car within a VR
headset<br>
o the interactive cloud-rendered video already linked above<br>
o an online gaming benchmark<br>
o HAS video streaming<br>
o a number of bulk file downloads<br>
o a heavy synthetic load of web browsing<br>
<br>
L4S has never been access-technology-specific. Indeed the
cable industry has been funding my work at the IETF to help
encourage a wider L4S ecosystem. There is nothing private to
the cable industry in this:<br>
* Al-Lu used DSL as a use-case, but L4S was relevant to all
the access technologies they supplied.
<br>
* BT was obviously interested in DSL, <br>
* but BT's initial motivating use-case was to incrementally
deploy the low queuing delay of DCTCP over BT's data centre
interconnect networks.
<br>
* In Jul 2016 the open-source Linux repo for the Coupled AQM
was published, with a fully working version to be used and
abused.
<br>
Now at: <a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e="
moz-do-not-send="true">
https://github.com/L4STeam/sch_dualpi2_upstream</a><br>
* Of course, DCTCP was already open-sourced in Linux and
FreeBSD, as well as available in Windows<br>
* In Jul 2016, the main IETF BoF on L4S was held:<br>
o Ingemar Johansson from Ericsson was one of the proponents,
focused on using L4S in LTE<br>
o along with Kevin Smith from Vodafone and <br>
o Praveen Balasubramanian from Microsoft (who maintains the
Windows TCP stack, including DCTCP).<br>
o Ingemar has since written an open-source L4S variant of
the SCReAM congestion controller for real-time media:
<a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e="
moz-do-not-send="true">
https://github.com/EricssonResearch/scream/</a><br>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented
the necessary feedback in TCP
<a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e="
moz-do-not-send="true">
https://github.com/mirjak/linux-accecn</a><br>
* In summer 2017 CableLabs started work on Low Latency DOCSIS,
and hired me later in the year to help develop and specify it,
along with support for L4S<br>
o In Jan 2019 the Low Latency DOCSIS spec was published and
is now being implemented.<br>
o You can find the primary companies involved at the end of
the White Paper. <a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e="
moz-do-not-send="true">
https://cablela.bs/low-latency-docsis-technology-overview-february-2019</a><br>
o Operators:<br>
Liberty Global<br>
Charter<br>
Rogers<br>
Comcast<br>
Shaw<br>
Cox Communications<br>
o Equipment vendors<br>
ARRIS<br>
Huawei<br>
Broadcom<br>
Intel<br>
Casa<br>
Nokia<br>
Cisco<br>
Videotron<br>
* Nicolas Kuhn of CNES has been assessing the use of L4S for
satellite.<br>
* Magnus Westerlund of Ericsson with a team of others has
written the necessary ECN feedback into QUIC<br>
* L4S hardware is also being implemented for hi-speed switches
at the moment <br>
(the developer wants to announce it themselves, so I have
been asked not to identify them).
<br>
<br>
There's a lot more stuff been going on, but I've tried to pick
out highlights.<br>
<br>
All this is good healthy development of much lower latency for
Internet technology.<br>
<br>
<br>
I find it extremely disappointing that some people on this
list are taking such a negative attitude to the major
development in their own field that they seem not to have
noticed since it first hit the limelight in 2015.
<br>
<br>
L4S has been open-sourced since 2016 so that everyone can
develop it and make it better...<br>
<br>
If I was in this position, having overlooked something
important for multiple years, I would certainly not condemn it
while I was trying to understand what it was and how it
worked. Can I suggest everyone takes a step back, and suspends
judgement until they have understood the potential, the goals
and the depth of what they have missed. People who know me,
know that I am very careful with Internet architecture, and
particularly with balancing public policy against commercial
issues. Please presume respect unless proven otherwise.<br>
<br>
Best Regards<br>
<br>
<br>
<br>
Bob<br>
<br>
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to
get into the L4S queue. But only if it can keep latency down
below around 1ms. Currently those ~15-25ms delay spikes would
not pass muster. Indeed, its delay is not consistently low
enough between the spikes either.<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p
style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in;overflow-wrap:
break-word">
<span
style="font-size:12.0pt;font-family:"Arial",sans-serif">-----Original
Message-----<br>
From: "Vint Cerf" <a href="mailto:vint@google.com"
moz-do-not-send="true"><vint@google.com></a><br>
Sent: Saturday, March 16, 2019 5:57pm<br>
To: "Holland, Jake" <a href="mailto:jholland@akamai.com"
moz-do-not-send="true"><jholland@akamai.com></a><br>
Cc: "Mikael Abrahamsson" <a
href="mailto:swmike@swm.pp.se" moz-do-not-send="true"><swmike@swm.pp.se></a>,
"David P. Reed"
<a href="mailto:dpreed@deepplum.com"
moz-do-not-send="true"><dpreed@deepplum.com></a>,
<a href="mailto:ecn-sane@lists.bufferbloat.net"
moz-do-not-send="true">
"ecn-sane@lists.bufferbloat.net"</a> <a
href="mailto:ecn-sane@lists.bufferbloat.net"
moz-do-not-send="true">
<ecn-sane@lists.bufferbloat.net></a>, "bloat" <a
href="mailto:bloat@lists.bufferbloat.net"
moz-do-not-send="true">
<bloat@lists.bufferbloat.net></a><br>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague]
Implementation and experimentation of TCP Prague/L4S
hackaton at IETF104<o:p></o:p></span></p>
<div id="SafeStyles1552845686">
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">where
does BBR fit into all this?
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">v<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">On
Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <<a
href="mailto:jholland@akamai.com"
moz-do-not-send="true">jholland@akamai.com</a>>
wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">On
2019-03-15, 11:37, "Mikael Abrahamsson" <<a
href="mailto:swmike@swm.pp.se" target="_blank"
moz-do-not-send="true">swmike@swm.pp.se</a>>
wrote:<br>
L4S has a much better possibility of actually
getting deployment into the <br>
wider Internet packet-moving equipment than
anything being talked about <br>
here. Same with PIE as opposed to FQ_CODEL. I
know it's might not be as <br>
good, but it fits better into actual silicon and
it's being proposed by <br>
people who actually have better channels into
the people setting hard <br>
requirements.<br>
<br>
I suggest you consider joining them instead of
opposing them.<br>
<br>
<br>
Hi Mikael,<br>
<br>
I agree it makes sense that fq_anything has issues
when you're talking<br>
about the OLT/CMTS/BNG/etc., and I believe it when
you tell me PIE<br>
makes better sense there.<br>
<br>
But fq_x makes great sense and provides real value
for the uplink in a<br>
home, small office, coffee shop, etc. (if you run
the final rate limit<br>
on the home side of the access link.) I'm thinking
maybe there's a<br>
disconnect here driven by the different use cases
for where AQMs can go.<br>
<br>
The thing is, each of these is the most likely
congestion point at<br>
different times, and it's worthwhile for each of
them to be able to<br>
AQM (and mark packets) under congestion.<br>
<br>
One of the several things that bothers me with L4S
is that I've seen<br>
precious little concern over interfering with the
ability for another<br>
different AQM in-path to mark packets, and because
it changes the<br>
semantics of CE, you can't have both working at the
same time unless<br>
they both do L4S.<br>
<br>
SCE needs a lot of details filled in, but it's so
much cleaner that it<br>
seems to me there's reasonably obvious answers to
all (or almost all) of<br>
those detail questions, and because the semantics
are so much cleaner,<br>
it's much easier to tell it's non-harmful.<br>
<br>
<aside regarding="non-harmful"><br>
The point you raised in another thread about
reordering is mostly<br>
well-taken, and a good counterpoint to the claim
"non-harmful relative<br>
to L4S".<br>
<br>
To me it seems sad and dumb that switches ended up
trying to make<br>
ordering guarantees at cost of switching
performance, because if it's<br>
useful to put ordering in the switch, then it must
be equally useful to<br>
put it in the receiver's NIC or OS.<br>
<br>
So why isn't it in all the receivers' NIC or OS
(where it would render<br>
the switch's ordering efforts moot) instead of in
all the switches?<br>
<br>
I'm guessing the answer is a competition trap for
the switch vendors,<br>
plus "with ordering goes faster than without, when
you benchmark the<br>
switch with typical load and current (non-RACK)
receivers".<br>
<br>
If that's the case, it seems like the drive for a
competitive advantage<br>
caused deployment of a packet ordering workaround in
the wrong network<br>
location(s), out of a pure misalignment of
incentives.<br>
<br>
RACK rates to fix that in the end, but a lot of
damage is already done,<br>
and the L4S approach gives switches a flag that can
double as proof that<br>
RACK is there on the receiver, so they can stop
trying to order those<br>
packets.<br>
<br>
So point granted, I understand and agree there's a
cost to abandoning<br>
that advantage.<br>
</aside><br>
<br>
But as you also said so well in another thread, this
is important. ("The<br>
last unicorn", IIRC.) How much does it matter if
there's a feature that<br>
has value today, but only until RACK is widely
deployed? If you were<br>
convinced RACK would roll out everywhere within 3
years and SCE would<br>
produce better results than L4S over the following
15 years, would that<br>
change your mind?<br>
<br>
It would for me, and that's why I'd like to see SCE
explored before<br>
making a call. I think at its core, it provides the
same thing L4S does<br>
(a high-fidelity explicit congestion signal for the
sender), but with<br>
much cleaner semantics that can be incrementally
added to congestion<br>
controls that people are already using.<br>
<br>
Granted, it still remains to be seen whether SCE in
practice can match<br>
the results of L4S, and L4S was here first. But it
seems to me L4S comes<br>
with some problems that have not yet been examined,
and that are nicely<br>
dodged by a SCE-based approach.<br>
<br>
If L4S really is as good as they seem to think, I
could imagine getting<br>
behind it, but I don't think that's proven yet. I'm
not certain, but<br>
all the comparative analyses I remember seeing have
been from more or<br>
less the same team, and I'm not convinced they don't
have some<br>
misaligned incentives of their own.<br>
<br>
I understand a lot of work has gone into L4S, but
this move to jump it<br>
from interesting experiment to de-facto standard
without a more critical<br>
review that digs deeper into some of the potential
deployment problems<br>
has me concerned.<br>
<br>
If it really does turn out to be good enough to be
permanent, I'm not<br>
opposed to it, but I'm just not convinced that it's
non-harmful, and my<br>
default position is that the cleaner solution is
going to be better in<br>
the long run, if they can do the same job.<br>
<br>
It's not that I want it to be a fight, but I do want
to end up with the<br>
best solution we can get. We only have the one
internet.<br>
<br>
Just my 2c. <br>
<br>
-Jake<br>
<br>
<br>
_______________________________________________<br>
Ecn-sane mailing list<br>
<a href="mailto:Ecn-sane@lists.bufferbloat.net"
target="_blank" moz-do-not-send="true">Ecn-sane@lists.bufferbloat.net</a><br>
<a
href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e="
target="_blank" moz-do-not-send="true">https://lists.bufferbloat.net/listinfo/ecn-sane</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif"><br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">New
postal address:
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">Google<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">1875
Explorer Street, 10th Floor<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:12.0pt;font-family:"Arial",sans-serif">Reston,
VA 20190<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<o:p></o:p></p>
<pre style="margin-left:.5in">_______________________________________________<o:p></o:p></pre>
<pre style="margin-left:.5in">Bloat mailing list<o:p></o:p></pre>
<pre style="margin-left:.5in"><a href="mailto:Bloat@lists.bufferbloat.net" moz-do-not-send="true">Bloat@lists.bufferbloat.net</a><o:p></o:p></pre>
<pre style="margin-left:.5in"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=" moz-do-not-send="true">https://lists.bufferbloat.net/listinfo/bloat</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<o:p></o:p></p>
<pre style="margin-left:.5in">-- <o:p></o:p></pre>
<pre style="margin-left:.5in">________________________________________________________________<o:p></o:p></pre>
<pre style="margin-left:.5in">Bob Briscoe <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=" moz-do-not-send="true">http://bobbriscoe.net/</a><o:p></o:p></pre>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
________________________________________________________________
Bob Briscoe <a class="moz-txt-link-freetext" href="http://bobbriscoe.net/">http://bobbriscoe.net/</a></pre>
</body>
</html>