* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
[not found] ` <d91a6a71-5898-9571-2a02-0d9d83839615@bobbriscoe.net>
@ 2019-03-15 10:46 ` Dave Taht
2019-03-15 13:01 ` Sebastian Moeller
2019-03-15 21:34 ` Wesley Eddy
0 siblings, 2 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-15 10:46 UTC (permalink / raw)
To: Bob Briscoe; +Cc: tcpm IETF list, iccrg IRTF list, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 8673 bytes --]
Bufferbloat.net's ecn-sane working group members have a co-ordinated
response to your efforts brewing but it's not ready yet. We have a
worldwide team of linux and freebsd developers co-ordinating on landing
code for our competing proposal "Some Congestion Experienced", which we
submitted to tsvwg last sunday.
That draft is under continuous revision, here:
https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
Our Linux and FreeBSD team is also flying into prague for SCE presentations
at netdevconf and ietf.
Some background to this: after the L4S/TCP Prague/and dualpi experiments
appeared stalled out indefinitely in the IETF, and with our own frustration
with IETF processes, bufferbloat.net project members publicly formed our
own working group to look into the problems with ecn, back in august of
last year.
Its charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/
We were unaware, until last month, that the cable industry had 16 months
back gone and formed its own private working group also, and was intending
to turn the tcp prague/l4s/dualpi IETF "experiments" into an actual DOCSIS
standard.
Our SCE proposal appears to be backward compatible with the existing
10s-100s of millions of ecn-enabled fq_codel[1] and sch_cake[2]
deployments, and doesn't require any changes to any RFC3168 tcps (or any
tcp-friendly congestion control) at all in order to basically work.
tcp-prague is subtly incompatible with that, and dualpi, more so. Our
proposal is different also, it proposes some receiver side changes in order
to get the full benefit of SCE while remaining backward compatible with the
existing meaning of the CE codepoint.
In either case, either approach essentially permanently redefines the ECT_1
codepoint incompatibly, once and for all, and for all time. This is a final
battle over the meaning of a single bit in IP, and I expect the debates to
be as difficult as the ones described in
https://www.ietf.org/rfc/ien/ien137.txt - I would really, really, really
prefer that they stay technical and not veer into politics, but I have
little hope for that.
The members of the ecn-sane working group are delighted to finally hear
that running code for tcp-prague might land this ietf, and look forward to
finally testing the whole l4s/tcpprague/dualpi architecture in conjunction
with the flent.org 's and irtt's exhaustive suite of tests and servers in
the cloud in the coming months, both against our existing, deployed,
fq_codel, fq_pie, cake and pie derived solutions and our new SCE proposal.
We hope to finally be able to write new tests for flent in particular, that
can show tcpprague off in the ways that are important to those developing
it. Flent has some basic dctcp tests, but nothing that can get down below a
20ms sample rate on modern hardware. (currently. It's easy to add tests,
flent is written in python)
We also hope that more test tools and implementations in ns2 and ns3 show
up for tcpprauge and dualpi show up soon also, from members of those
projects.
Note: sunday's dual-pi linux submission was kicked back from the linux
networking developers due to some technical and legal problems with linux
net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/ ) , and I do
hope that a corrected version lands soon, so we can safely test it with
current versions of OpenWrt, etc.
Finally, running code. Will we find consensus?
Thx!
[1] https://tools.ietf.org/html/rfc8290
[2] sch_cake was available for 3 years out of tree and was mainlined last
august, in linux 4.19. It is partially described by "Piece of CAKE: A
Comprehensive Queue Management Solution for Home Gateways" "
https://arxiv.org/pdf/1804.07617.pdf
A second paper describing its fq_codel-derived "cobalt" AQM algorithm is
awaiting publication in a peer reviewed journal. It has been part of
openwrt and the related sqm-scripts for many years and is widely deployed
on multiple commercial products, such as those from eero and evenroute.
Cake has a docsis specific mode which we longed for cablelabs to evaluate.
On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
> Forwarding to tcpm & iccrg - apologies if you were already on one of the
> lists that received this.
>
> Olivier has been working hard on integrating the pieces of a Linux
> implementation of TCP Prague, and is close to having a version ported
> against the tip of the Linux mainline tree. This is his request for more
> people to get involved.
>
>
> Bob
>
>
> -------- Forwarded Message --------
> Subject: [tcpPrague] Implementation and experimentation of TCP Prague/L4S
> hackaton at IETF104
> Date: Wed, 6 Mar 2019 10:26:05 +0000
> From: Tilmans, Olivier (Nokia - BE/Antwerp)
> <olivier.tilmans@nokia-bell-labs.com>
> <olivier.tilmans@nokia-bell-labs.com>
> To: hackathon@ietf.org <hackathon@ietf.org> <hackathon@ietf.org>,
> tcpprague@ietf.org <tcpprague@ietf.org> <tcpprague@ietf.org>
> CC: dlebrun@google.com <dlebrun@google.com> <dlebrun@google.com>, Joakim
> Misund <joakim.misund@gmail.com> <joakim.misund@gmail.com>, Bob Briscoe
> <research@bobbriscoe.net> <research@bobbriscoe.net>, Quentin De Coninck
> <quentin.deconinck@uclouvain.be> <quentin.deconinck@uclouvain.be>,
> François Michel <francois.michel@uclouvain.be>
> <francois.michel@uclouvain.be>, Mirja Kuehlewind
> <mirja.kuehlewind@tik.ee.ethz.ch> <mirja.kuehlewind@tik.ee.ethz.ch>,
> Maxime Piraux <maxime.piraux@uclouvain.be> <maxime.piraux@uclouvain.be>,
> Olga Albisser <olga@albisser.org> <olga@albisser.org>, Fabien Duchêne
> <fabien.duchene@uclouvain.be> <fabien.duchene@uclouvain.be>, De Schepper,
> Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
> <koen.de_schepper@nokia-bell-labs.com>
>
> Hi all,
>
> We'll be working on the "TCP Prague" congestion control/L4S architecture
> during the IETF-104 hackaton.
> This topics aims at accelerating the work that started during the IETF93
> (coincidentally also in Prague), in order to get TCP Prague to an 'usable'
> state—i.e., meet the safety requirements and have supporting materials
> (e.g., VMs, labs) to let people experiment with it. Depending on people's
> interest, prototyping something similar for QUIC is another possible output.
>
> Details and links to resources/supporting drafts are available at
> https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague and
> copied below.
> Additionally, few topics will presented during netdev 0x13 the week before.
>
> See you in Prague.
>
> Best,
> Olivier
>
>
> Implementation and experimentation of TCP Prague/L4S
>
> * Champion
> * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
> * Projects
> * Prototype the "TCP Prague" congestion control on Linux
> * Finalize the implementation of accurate ECN (draft conformance), and
> port it on Linux v5.x * Build tooling around L4S to let people experiment
> with the technology (e.g., virtual machine, or mininet labs)
> * Work towards "QUIC Prague"
> * Resources
> * TCP Prague
> * Repository — https://github.com/L4STeam/tcp-prague
> * Requirements —
> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
> * Upcoming netdev talk —
> https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
> * Accurate ECN
> * Specs — https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
> * Implementation for Linux v4.17 — https://github.com/mirjak/linux-accecn
> * Past netdev talk —
> https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
> * Paced Chirping * Repository —
> https://github.com/JoakimMisund/PacedChirping
> * Upcoming netdev talk —
> https://netdevconf.org/0x13/session.html?talk-chirp
> * L4S architecture
> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
> * DualPI2 AQM
> * Specs —
> https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
> * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
> * Upcoming netdev talk —
> https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
> * RITE Project — https://riteproject.eu/dctth/#code
> _______________________________________________
> tcpPrague mailing list
> tcpPrague@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpprague
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg
>
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
[-- Attachment #2: Type: text/html, Size: 16938 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 10:46 ` [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104 Dave Taht
@ 2019-03-15 13:01 ` Sebastian Moeller
2019-03-15 14:06 ` Dave Taht
2019-03-15 14:27 ` [Bloat] " Jonathan Morton
2019-03-15 21:34 ` Wesley Eddy
1 sibling, 2 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-15 13:01 UTC (permalink / raw)
To: Dave Täht; +Cc: bloat
Hi Dave,
I pruned the CC list as I am out of my league here and want to restrict the radius of my embarrassment to those that already know my level of incompetence before hand.
That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1) for a purpose outside of its nominal purpose namely explicit congestion notification, why not think bolder? If the plan is to change everything the feasibility can not hinge upon the ability to re-using one old legacy bit, can it...
Conceptually it seems much cleaner to use ECT(1) for congestion notification directly (there are already proposals out there and SCE just added another fully back-ward compatible one) and find another way to mark l4s traffic, sure that is going to be hard and inconvenient, but if you set out to change the internet that is par for the course.
IMHO they would do more good if they actually fought for a better use of the 6 DSCP bits instead. (say by splitting into two groups of 3 bits, one group for reduced diffserv and one group for new features, that would even allow for concurrent use of the inevitable L5S and L6S ;) ). Especially since as far as I can understand l4s actually would like to have a more gradual congestion information stream than classic ECN, and since they need to modify DCTCP anyway to make it save for the wider internet, replacing its ECN response should be well inside the scope of work they already have on their list.
If I sound a bit miffed, it is because after reading https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03 I do not have the feeling they are trying to build abetter internet, but rather that they are building an internet where I can be a better "product" and customer of newfangled services, and I do not look forward to such a facebooky future with much enthusiasm.
I hope that the discussion in Prague go well and a compromise/consense can be hashed out as I see different implementations duking it out here, but the overall goal of the competitors seems quite compatible, improving the internet by focussing on latency.
Best Regards
Sebastian
> On Mar 15, 2019, at 11:46, Dave Taht <dave.taht@gmail.com> wrote:
>
> Bufferbloat.net's ecn-sane working group members have a co-ordinated response to your efforts brewing but it's not ready yet. We have a worldwide team of linux and freebsd developers co-ordinating on landing code for our competing proposal "Some Congestion Experienced", which we submitted to tsvwg last sunday.
>
> That draft is under continuous revision, here:
>
> https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
>
> Our Linux and FreeBSD team is also flying into prague for SCE presentations at netdevconf and ietf.
>
> Some background to this: after the L4S/TCP Prague/and dualpi experiments appeared stalled out indefinitely in the IETF, and with our own frustration with IETF processes, bufferbloat.net project members publicly formed our own working group to look into the problems with ecn, back in august of last year.
>
> Its charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/
>
> We were unaware, until last month, that the cable industry had 16 months back gone and formed its own private working group also, and was intending to turn the tcp prague/l4s/dualpi IETF "experiments" into an actual DOCSIS standard.
>
> Our SCE proposal appears to be backward compatible with the existing 10s-100s of millions of ecn-enabled fq_codel[1] and sch_cake[2] deployments, and doesn't require any changes to any RFC3168 tcps (or any tcp-friendly congestion control) at all in order to basically work. tcp-prague is subtly incompatible with that, and dualpi, more so. Our proposal is different also, it proposes some receiver side changes in order to get the full benefit of SCE while remaining backward compatible with the existing meaning of the CE codepoint.
>
> In either case, either approach essentially permanently redefines the ECT_1 codepoint incompatibly, once and for all, and for all time. This is a final battle over the meaning of a single bit in IP, and I expect the debates to be as difficult as the ones described in https://www.ietf.org/rfc/ien/ien137.txt - I would really, really, really prefer that they stay technical and not veer into politics, but I have little hope for that.
>
> The members of the ecn-sane working group are delighted to finally hear that running code for tcp-prague might land this ietf, and look forward to finally testing the whole l4s/tcpprague/dualpi architecture in conjunction with the flent.org 's and irtt's exhaustive suite of tests and servers in the cloud in the coming months, both against our existing, deployed, fq_codel, fq_pie, cake and pie derived solutions and our new SCE proposal. We hope to finally be able to write new tests for flent in particular, that can show tcpprague off in the ways that are important to those developing it. Flent has some basic dctcp tests, but nothing that can get down below a 20ms sample rate on modern hardware. (currently. It's easy to add tests, flent is written in python)
>
> We also hope that more test tools and implementations in ns2 and ns3 show up for tcpprauge and dualpi show up soon also, from members of those projects.
>
> Note: sunday's dual-pi linux submission was kicked back from the linux networking developers due to some technical and legal problems with linux net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/ ) , and I do hope that a corrected version lands soon, so we can safely test it with current versions of OpenWrt, etc.
>
> Finally, running code. Will we find consensus?
>
> Thx!
>
>
> [1] https://tools.ietf.org/html/rfc8290
> [2] sch_cake was available for 3 years out of tree and was mainlined last august, in linux 4.19. It is partially described by "Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways" "https://arxiv.org/pdf/1804.07617.pdf
>
> A second paper describing its fq_codel-derived "cobalt" AQM algorithm is awaiting publication in a peer reviewed journal. It has been part of openwrt and the related sqm-scripts for many years and is widely deployed on multiple commercial products, such as those from eero and evenroute.
>
> Cake has a docsis specific mode which we longed for cablelabs to evaluate.
>
>
> On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
> Forwarding to tcpm & iccrg - apologies if you were already on one of the lists that received this.
>
> Olivier has been working hard on integrating the pieces of a Linux implementation of TCP Prague, and is close to having a version ported against the tip of the Linux mainline tree. This is his request for more people to get involved.
>
>
> Bob
>
>
> -------- Forwarded Message --------
> Subject: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
> Date: Wed, 6 Mar 2019 10:26:05 +0000
> From: Tilmans, Olivier (Nokia - BE/Antwerp) <olivier.tilmans@nokia-bell-labs.com>
> To: hackathon@ietf.org <hackathon@ietf.org>, tcpprague@ietf.org <tcpprague@ietf.org>
> CC: dlebrun@google.com <dlebrun@google.com>, Joakim Misund <joakim.misund@gmail.com>, Bob Briscoe <research@bobbriscoe.net>, Quentin De Coninck <quentin.deconinck@uclouvain.be>, François Michel <francois.michel@uclouvain.be>, Mirja Kuehlewind <mirja.kuehlewind@tik.ee.ethz.ch>, Maxime Piraux <maxime.piraux@uclouvain.be>, Olga Albisser <olga@albisser.org>, Fabien Duchêne <fabien.duchene@uclouvain.be>, De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
>
> Hi all,
>
> We'll be working on the "TCP Prague" congestion control/L4S architecture during the IETF-104 hackaton.
> This topics aims at accelerating the work that started during the IETF93 (coincidentally also in Prague), in order to get TCP Prague to an 'usable' state—i.e., meet the safety requirements and have supporting materials (e.g., VMs, labs) to let people experiment with it. Depending on people's interest, prototyping something similar for QUIC is another possible output.
>
> Details and links to resources/supporting drafts are available at https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague and copied below.
> Additionally, few topics will presented during netdev 0x13 the week before.
>
> See you in Prague.
>
> Best,
> Olivier
>
>
> Implementation and experimentation of TCP Prague/L4S
>
> * Champion
> * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
> * Projects
> * Prototype the "TCP Prague" congestion control on Linux
> * Finalize the implementation of accurate ECN (draft conformance), and port it on Linux v5.x * Build tooling around L4S to let people experiment with the technology (e.g., virtual machine, or mininet labs)
> * Work towards "QUIC Prague"
> * Resources
> * TCP Prague
> * Repository — https://github.com/L4STeam/tcp-prague
> * Requirements — https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
> * Accurate ECN
> * Specs — https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
> * Implementation for Linux v4.17 — https://github.com/mirjak/linux-accecn
> * Past netdev talk — https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
> * Paced Chirping * Repository — https://github.com/JoakimMisund/PacedChirping
> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-chirp
> * L4S architecture
> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
> * DualPI2 AQM
> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
> * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
> * RITE Project — https://riteproject.eu/dctth/#code
> _______________________________________________
> tcpPrague mailing list
> tcpPrague@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpprague
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 13:01 ` Sebastian Moeller
@ 2019-03-15 14:06 ` Dave Taht
2019-03-15 15:52 ` Sebastian Moeller
2019-03-15 18:07 ` Mikael Abrahamsson
2019-03-15 14:27 ` [Bloat] " Jonathan Morton
1 sibling, 2 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-15 14:06 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: bloat, ecn-sane
I would really prefer to move this discussion to the ecn-sane mailing
list, as IMHO, ecn is generally such a tiny thing needed for good
congestion control compared to better transports like pacing + bbr,
and things like bql, fq, and aqm using drop.
I'm going to keep cc-ing that list in the hope that we can keep better
track of the discussion here.
On Fri, Mar 15, 2019 at 6:01 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> I pruned the CC list as I am out of my league here and want to restrict the radius of my embarrassment to those that already know my level of incompetence before hand.
IMHO, your work on educating the OpenWrt community over the years on
how to use sqm, makes you much more than "only a grasshopper". You
have a firm grip on what can be achieved in the real world.
>
> That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
I am so glad someone other than I has now read it.
> The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1) for a purpose outside of its nominal purpose namely explicit congestion notification, why not think bolder? If the plan is to change everything the feasibility can not hinge upon the ability to re-using one old legacy bit, can it...
> Conceptually it seems much cleaner to use ECT(1) for congestion notification directly (there are already proposals out there and SCE just added another fully back-ward compatible one) and find another way to mark l4s traffic, sure that is going to be hard and inconvenient, but if you set out to change the internet that is par for the course.
> IMHO they would do more good if they actually fought for a better use of the 6 DSCP bits instead. (say by splitting into two groups of 3 bits, one group for reduced diffserv and one group for new features, that would even
The existing diffserv deployment is a failure. I have another ID
cooking that suggests a better way, going forward, to use them, but I
do not feel at this time it would be useful to present. One big battle
at a time. That ID is very simple, it basically proposes that all
diffserv codepoints be passed through ISPs and transit providers
unchanged, and if any given codepoint must be changed, the only
permissible change is to 0 (BE), and MUST be not be remarked to
anything else, especially not CS1.
>allow for concurrent use of the inevitable L5S and L6S ;) ). Especially since as far as I can understand l4s actually would like to have a more gradual congestion information stream than classic ECN, and since they need to modify DCTCP anyway to make it save for the wider internet, replacing its ECN response should be well inside the scope of work they already have on their list.
Next up for sce was going to be to find if dctcp could be modified to
use it. Also, bittorrent.
> If I sound a bit miffed, it is because after reading https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03 I do not have the feeling they are trying to build abetter internet, but rather that they are building an internet where I can be a better "product" and customer of newfangled services, and I do not look forward to such a facebooky future with much enthusiasm.
I have rigorously tried to keep the network neutrality debate out of
this. dualpi is just another aqm that needs the same thorough
technical and public evaluation done to it that pie, codel, fq_codel
were subjected to.The use of ect_1 in dualpi for it is nuts IMHO, and
I'd vastly prefer that another L4S codepoint be added to make it work,
but any attempt to do so would also require industry consolidation on
that ID and that would be distracting at this time.
It appears, also, ironically, (I have confirmation from several
sources now) that cake, fq_codel and dualpi are now illegal for an ISP
to use in their provided equipment under california law. The idea of
one day having to appear in court to defend our key algorithms reminds
me of the famous john fogerty case where he demonstrated how blues
music was made.
I wish I knew a lawyer willing to take on "bufferbloat.net vs the
state of california", though, as it may come down to that.
I blew up on this part of the issue here:
http://blog.cerowrt.org/post/net_neutrality_customers/
>
> I hope that the discussion in Prague go well and a compromise/consense can be hashed out as I see different implementations duking it out here, but the overall goal of the competitors seems quite compatible, improving the internet by focussing on latency.
>
> Best Regards
> Sebastian
>
> > On Mar 15, 2019, at 11:46, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > Bufferbloat.net's ecn-sane working group members have a co-ordinated response to your efforts brewing but it's not ready yet. We have a worldwide team of linux and freebsd developers co-ordinating on landing code for our competing proposal "Some Congestion Experienced", which we submitted to tsvwg last sunday.
> >
> > That draft is under continuous revision, here:
> >
> > https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
> >
> > Our Linux and FreeBSD team is also flying into prague for SCE presentations at netdevconf and ietf.
> >
> > Some background to this: after the L4S/TCP Prague/and dualpi experiments appeared stalled out indefinitely in the IETF, and with our own frustration with IETF processes, bufferbloat.net project members publicly formed our own working group to look into the problems with ecn, back in august of last year.
> >
> > Its charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/
> >
> > We were unaware, until last month, that the cable industry had 16 months back gone and formed its own private working group also, and was intending to turn the tcp prague/l4s/dualpi IETF "experiments" into an actual DOCSIS standard.
> >
> > Our SCE proposal appears to be backward compatible with the existing 10s-100s of millions of ecn-enabled fq_codel[1] and sch_cake[2] deployments, and doesn't require any changes to any RFC3168 tcps (or any tcp-friendly congestion control) at all in order to basically work. tcp-prague is subtly incompatible with that, and dualpi, more so. Our proposal is different also, it proposes some receiver side changes in order to get the full benefit of SCE while remaining backward compatible with the existing meaning of the CE codepoint.
> >
> > In either case, either approach essentially permanently redefines the ECT_1 codepoint incompatibly, once and for all, and for all time. This is a final battle over the meaning of a single bit in IP, and I expect the debates to be as difficult as the ones described in https://www.ietf.org/rfc/ien/ien137.txt - I would really, really, really prefer that they stay technical and not veer into politics, but I have little hope for that.
> >
> > The members of the ecn-sane working group are delighted to finally hear that running code for tcp-prague might land this ietf, and look forward to finally testing the whole l4s/tcpprague/dualpi architecture in conjunction with the flent.org 's and irtt's exhaustive suite of tests and servers in the cloud in the coming months, both against our existing, deployed, fq_codel, fq_pie, cake and pie derived solutions and our new SCE proposal. We hope to finally be able to write new tests for flent in particular, that can show tcpprague off in the ways that are important to those developing it. Flent has some basic dctcp tests, but nothing that can get down below a 20ms sample rate on modern hardware. (currently. It's easy to add tests, flent is written in python)
> >
> > We also hope that more test tools and implementations in ns2 and ns3 show up for tcpprauge and dualpi show up soon also, from members of those projects.
> >
> > Note: sunday's dual-pi linux submission was kicked back from the linux networking developers due to some technical and legal problems with linux net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/ ) , and I do hope that a corrected version lands soon, so we can safely test it with current versions of OpenWrt, etc.
> >
> > Finally, running code. Will we find consensus?
> >
> > Thx!
> >
> >
> > [1] https://tools.ietf.org/html/rfc8290
> > [2] sch_cake was available for 3 years out of tree and was mainlined last august, in linux 4.19. It is partially described by "Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways" "https://arxiv.org/pdf/1804.07617.pdf
> >
> > A second paper describing its fq_codel-derived "cobalt" AQM algorithm is awaiting publication in a peer reviewed journal. It has been part of openwrt and the related sqm-scripts for many years and is widely deployed on multiple commercial products, such as those from eero and evenroute.
> >
> > Cake has a docsis specific mode which we longed for cablelabs to evaluate.
> >
> >
> > On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
> > Forwarding to tcpm & iccrg - apologies if you were already on one of the lists that received this.
> >
> > Olivier has been working hard on integrating the pieces of a Linux implementation of TCP Prague, and is close to having a version ported against the tip of the Linux mainline tree. This is his request for more people to get involved.
> >
> >
> > Bob
> >
> >
> > -------- Forwarded Message --------
> > Subject: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
> > Date: Wed, 6 Mar 2019 10:26:05 +0000
> > From: Tilmans, Olivier (Nokia - BE/Antwerp) <olivier.tilmans@nokia-bell-labs.com>
> > To: hackathon@ietf.org <hackathon@ietf.org>, tcpprague@ietf.org <tcpprague@ietf.org>
> > CC: dlebrun@google.com <dlebrun@google.com>, Joakim Misund <joakim.misund@gmail.com>, Bob Briscoe <research@bobbriscoe.net>, Quentin De Coninck <quentin.deconinck@uclouvain.be>, François Michel <francois.michel@uclouvain.be>, Mirja Kuehlewind <mirja.kuehlewind@tik.ee.ethz.ch>, Maxime Piraux <maxime.piraux@uclouvain.be>, Olga Albisser <olga@albisser.org>, Fabien Duchêne <fabien.duchene@uclouvain.be>, De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
> >
> > Hi all,
> >
> > We'll be working on the "TCP Prague" congestion control/L4S architecture during the IETF-104 hackaton.
> > This topics aims at accelerating the work that started during the IETF93 (coincidentally also in Prague), in order to get TCP Prague to an 'usable' state—i.e., meet the safety requirements and have supporting materials (e.g., VMs, labs) to let people experiment with it. Depending on people's interest, prototyping something similar for QUIC is another possible output.
> >
> > Details and links to resources/supporting drafts are available at https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague and copied below.
> > Additionally, few topics will presented during netdev 0x13 the week before.
> >
> > See you in Prague.
> >
> > Best,
> > Olivier
> >
> >
> > Implementation and experimentation of TCP Prague/L4S
> >
> > * Champion
> > * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
> > * Projects
> > * Prototype the "TCP Prague" congestion control on Linux
> > * Finalize the implementation of accurate ECN (draft conformance), and port it on Linux v5.x * Build tooling around L4S to let people experiment with the technology (e.g., virtual machine, or mininet labs)
> > * Work towards "QUIC Prague"
> > * Resources
> > * TCP Prague
> > * Repository — https://github.com/L4STeam/tcp-prague
> > * Requirements — https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
> > * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
> > * Accurate ECN
> > * Specs — https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
> > * Implementation for Linux v4.17 — https://github.com/mirjak/linux-accecn
> > * Past netdev talk — https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
> > * Paced Chirping * Repository — https://github.com/JoakimMisund/PacedChirping
> > * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-chirp
> > * L4S architecture
> > * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
> > * DualPI2 AQM
> > * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
> > * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
> > * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
> > * RITE Project — https://riteproject.eu/dctth/#code
> > _______________________________________________
> > tcpPrague mailing list
> > tcpPrague@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpprague
> > _______________________________________________
> > iccrg mailing list
> > iccrg@irtf.org
> > https://www.irtf.org/mailman/listinfo/iccrg
> >
> >
> > --
> >
> > Dave Täht
> > CTO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-831-205-9740
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 13:01 ` Sebastian Moeller
2019-03-15 14:06 ` Dave Taht
@ 2019-03-15 14:27 ` Jonathan Morton
2019-03-15 14:44 ` Sebastian Moeller
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-15 14:27 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Dave Täht, bloat, ecn-sane
> On 15 Mar, 2019, at 3:01 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
>
> The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1)…
I think I would be less annoyed by L4S if it proposed to assign a DSCP or three to identify its traffic. In fact, I'd be interested to hear a defence of why DSCP identification of L4S flows was *not* adopted. I suspect it would revolve around the widespread practice of re-marking DSCPs by ISPs, which renders DSCP too unreliable for L4S' purposes.
But using the ECN field for this purpose is *also* unreliable, because it is *intended* to be altered on route (in prescribed ways). In particular, both ECT codepoints can be replaced by CE, erasing the distinction between them when it comes to choosing which half of a "dual queue" AQM to pass it through. I'm not convinced they've spent very much of the past several years thinking about that.
Fortunately, L4S flows should work with flow-isolating AQMs (including Cake) without modifying the latter, and without needing to specifically identify which flows are L4S or otherwise. That really says more about the robustness of proper flow isolation than anything else. But Codel-type AQMs don't provide the ideal ECN signal for L4S (and nor do RED-type AQMs without specific tuning for L4S), and any single-queue AQM is going to end up with L4S flows outcompeting standard ones quite seriously.
Since very little "big iron" hardware supports flow-isolating AQM so far, that puts a damper on any "incremental deployment" argument to be made for L4S, even if they claim their dual-queue AQM is easier to implement at high link rates than full flow isolation. The fact is that either dual-queue or flow-isolation is required in middleboxes *before* endpoint deployment of L4S can begin.
SCE explicitly avoids these problems, as both SCE-aware endpoints and SCE-aware middleboxes can safely be deployed without waiting for each other. Work remains on figuring out precisely how SCE information should be produced and consumed, and verifying that the resulting system behaves as intended when fully deployed - but its interaction with existing deployed hardware and software is explicitly defined to be benign.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 14:27 ` [Bloat] " Jonathan Morton
@ 2019-03-15 14:44 ` Sebastian Moeller
2019-03-15 15:49 ` Jonathan Morton
0 siblings, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-15 14:44 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Dave Täht, bloat, ecn-sane
Hi Jonathan,
> On Mar 15, 2019, at 15:27, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 15 Mar, 2019, at 3:01 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
>>
>> The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1)…
>
> I think I would be less annoyed by L4S if it proposed to assign a DSCP or three to identify its traffic. In fact, I'd be interested to hear a defence of why DSCP identification of L4S flows was *not* adopted. I suspect it would revolve around the widespread practice of re-marking DSCPs by ISPs, which renders DSCP too unreliable for L4S' purposes.
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05 contains a discussion of the alternatives, specifically to your point it argues:
"Appendix B. Alternative Identifiers
[...]
B.2. ECN Plus a Diffserv Codepoint (DSCP)
Definition:
For packets with a defined DSCP, all codepoints of the ECN field
(except Not-ECT) would signify alternative L4S semantics to those
for classic ECN [RFC3168], specifically:
* The L4S DSCP would signifiy that the packet came from an L4S-
capable sender;
* ECT(0) and ECT(1) would both signify that the packet was
travelling between transport endpoints that were both ECN-
capable;
* CE would signify that the packet had been marked by an AQM
implementing the L4S service.
Use of a DSCP is the only approach for alternative ECN semantics
given as an example in [RFC4774]. However, it was perhaps considered
more for controlled environments than new end-to-end services;
Cons:
Consumes DSCP pairs: A DSCP is obviously not orthogonal to Diffserv.
Therefore, wherever the L4S service is applied to multiple
Diffserv scheduling behaviours, it would be necessary to replace
each DSCP with a pair of DSCPs.
Uses critical lower-layer header space: The resulting increased
number of DSCPs might be hard to support for some lower layer
technologies, e.g. 802.1p and MPLS both offer only 3-bits for a
maximum of 8 traffic class identifiers. Although L4S should
reduce and possibly remove the need for some DSCPs intended for
differentiated queuing delay, it will not remove the need for
Diffserv entirely, because Diffserv is also used to allocate
bandwidth, e.g. by prioritising some classes of traffic over
others when traffic exceeds available capacity.
Not end-to-end (host-network): Very few networks honour a DSCP set
by a host. Typically a network will zero (bleach) the Diffserv
field from all hosts. Sometimes networks will attempt to identify
applications by some form of packet inspection and, based on
network policy, they will set the DSCP considered appropriate for
the identified application. Network-based application
identification might use some combination of protocol ID, port
numbers(s), application layer protocol headers, IP address(es),
VLAN ID(s) and even packet timing.
Not end-to-end (network-network): Very few networks honour a DSCP
received from a neighbouring network. Typically a network will
zero (bleach) the Diffserv field from all neighbouring networks at
an interconnection point. Sometimes bilateral arrangements are
made between networks, such that the receiving network remarks
some DSCPs to those it uses for roughly equivalent services. The
likelihood that a DSCP will be bleached or ignored depends on the
type of DSCP:
Local-use DSCP: These tend to be used to implement application-
specific network policies, but a bilateral arrangement to
remark certain DSCPs is often applied to DSCPs in the local-use
range simply because it is easier not to change all of a
network's internal configurations when a new arrangement is
made with a neighbour;
Global-use DSCP: These do not tend to be honoured across network
interconnections more than local-use DSCPs. However, if two
networks decide to honour certain of each other's DSCPs, the
reconfiguration is a little easier if both of their globally
recognised services are already represented by the relevant
global-use DSCPs.
Note that today a global-use DSCP gives little more assurance
of end-to-end service than a local-use DSCP. In future the
global-use range might give more assurance of end-to-end
service than local-use, but it is unlikely that either
assurance will be high, particularly given the hosts are
included in the end-to-end path.
Not all tunnels: Diffserv codepoints are often not propagated to the
outer header when a packet is encapsulated by a tunnel header.
DSCPs are propagated to the outer of uniform mode tunnels, but not
pipe mode [RFC2983], and pipe mode is fairly common.
ECN hard in some lower layers:: Because this approach uses both the
Diffserv and ECN fields, an AQM wil only work at a lower layer if
both can be supported. If individual network operators wished to
deploy an AQM at a lower layer, they would usually propagate an IP
Diffserv codepoint to the lower layer, using for example IEEE
802.1p. However, the ECN capability is harder to propagate down
to lower layers because few lower layers support it.
Pros:
Could migrate to e2e: If all usage of classic ECN migrates to usage
of L4S, the DSCP would become redundant, and the ECN capability
alone could eventually identify L4S packets without the
interconnection problems of Diffserv detailed above, and without
having permanently consumed more than one codepoint in the IP
header. Although the DSCP does not generally function as an end-
to-end identifier (see above), it could be used initially by
individual ISPs to introduce the L4S service for their own locally
generated traffic;"
In essence, they do not want to deal with the diffserv mess under the end-to-end perspective and making it reliable.
>
> But using the ECN field for this purpose is *also* unreliable, because it is *intended* to be altered on route (in prescribed ways). In particular, both ECT codepoints can be replaced by CE, erasing the distinction between them when it comes to choosing which half of a "dual queue" AQM to pass it through. I'm not convinced they've spent very much of the past several years thinking about that.
There is also a discussion of this in Appendix B. Alternative Identifiers, which I will not copy here ;)
>
> Fortunately, L4S flows should work with flow-isolating AQMs (including Cake) without modifying the latter, and without needing to specifically identify which flows are L4S or otherwise. That really says more about the robustness of proper flow isolation than anything else. But Codel-type AQMs don't provide the ideal ECN signal for L4S (and nor do RED-type AQMs without specific tuning for L4S), and any single-queue AQM is going to end up with L4S flows outcompeting standard ones quite seriously.
Except L$S flows are required to "degrade" to classic congestion contro once they have proof of not being handled by a l4s aware AQM, like real packet drops or ECT(0) + CE...
>
> Since very little "big iron" hardware supports flow-isolating AQM so far, that puts a damper on any "incremental deployment" argument to be made for L4S, even if they claim their dual-queue AQM is easier to implement at high link rates than full flow isolation. The fact is that either dual-queue or flow-isolation is required in middleboxes *before* endpoint deployment of L4S can begin.
Sure, the argument there seems to be that the typical bottleneck seems to be the access link and there the ISP who could controll this might have an interest of making that happen. In that light the involvement of cablelabs actually seems like a good thing, except for the part were they presumably took development underground/out of sight...
>
> SCE explicitly avoids these problems, as both SCE-aware endpoints and SCE-aware middleboxes can safely be deployed without waiting for each other. Work remains on figuring out precisely how SCE information should be produced and consumed, and verifying that the resulting system behaves as intended when fully deployed - but its interaction with existing deployed hardware and software is explicitly defined to be benign.
Sure, I am confident that should SCE prevail here, L4S would find uses for SCE, but that still faces the same roll-out issue...
Best Regards
Sebastian
>
> - Jonathan Morton
>
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 14:44 ` Sebastian Moeller
@ 2019-03-15 15:49 ` Jonathan Morton
0 siblings, 0 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-15 15:49 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Dave Täht, bloat, ecn-sane
> On 15 Mar, 2019, at 4:44 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> In essence, they do not want to deal with the diffserv mess under the end-to-end perspective and making it reliable.
Yeah, that's pretty much what I thought. Diffserv really does need to be fixed sometime.
>> But Codel-type AQMs don't provide the ideal ECN signal for L4S (and nor do RED-type AQMs without specific tuning for L4S), and any single-queue AQM is going to end up with L4S flows outcompeting standard ones quite seriously.
>
> Except L$S flows are required to "degrade" to classic congestion contro once they have proof of not being handled by a l4s aware AQM, like real packet drops or ECT(0) + CE...
That isn't enough, because ECN AQMs as presently specified won't change ECT(1) to ECT(0) (nor will SCE), and it would require a lot of overload before dropping actually started. So those signals don't reliably identify a legacy AQM.
> L4S would find uses for SCE, but that still faces the same roll-out issue...
It's not the same roll-out issue, because in an SCE context L4S would have to be legacy-compatible by default, and only employ the "new" behaviour when SCE information appears - the reverse of its present behaviour - which then makes it backwards compatible and incrementally deployable.
The real snag is that DCTCP's setpoint is 2 marks per RTT, which is different from SCE's specified setpoint of 50% packets marked (unless the cwnd is down to 4 packets). That'd make it difficult for a straight port of DCTCP to SCE to achieve full throughput.
A sane compromise could be for SCE to be marked in the same way as L4S marks CE, and a valid interpretation of SCE to follow the 1/n response of DCTCP. That would leverage existing TCP-Prague R&D, but in a backwards-compatible, incrementally-deployable manner. Perhaps that's something worth discussing at IETF?
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 14:06 ` Dave Taht
@ 2019-03-15 15:52 ` Sebastian Moeller
2019-03-15 17:01 ` [Bloat] [Ecn-sane] " David P. Reed
2019-03-15 18:07 ` Mikael Abrahamsson
1 sibling, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-15 15:52 UTC (permalink / raw)
To: Dave Täht; +Cc: bloat, ecn-sane
Hi Dave,
> On Mar 15, 2019, at 15:06, Dave Taht <dave.taht@gmail.com> wrote:
>
> I would really prefer to move this discussion to the ecn-sane mailing
> list, as IMHO, ecn is generally such a tiny thing needed for good
> congestion control compared to better transports like pacing + bbr,
> and things like bql, fq, and aqm using drop.
>
> I'm going to keep cc-ing that list in the hope that we can keep better
> track of the discussion here.
Ah, sure, I basically wanted to avoid annoying the ietf lists as I feel out of place there, having only had a cursory read of the relevant RFCs.
>
> On Fri, Mar 15, 2019 at 6:01 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>> I pruned the CC list as I am out of my league here and want to restrict the radius of my embarrassment to those that already know my level of incompetence before hand.
>
> IMHO, your work on educating the OpenWrt community over the years on
> how to use sqm, makes you much more than "only a grasshopper". You
> have a firm grip on what can be achieved in the real world.
>
>>
>> That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
>
> I am so glad someone other than I has now read it.
>
>> The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1) for a purpose outside of its nominal purpose namely explicit congestion notification, why not think bolder? If the plan is to change everything the feasibility can not hinge upon the ability to re-using one old legacy bit, can it...
>> Conceptually it seems much cleaner to use ECT(1) for congestion notification directly (there are already proposals out there and SCE just added another fully back-ward compatible one) and find another way to mark l4s traffic, sure that is going to be hard and inconvenient, but if you set out to change the internet that is par for the course.
>> IMHO they would do more good if they actually fought for a better use of the 6 DSCP bits instead. (say by splitting into two groups of 3 bits, one group for reduced diffserv and one group for new features, that would even
>
> The existing diffserv deployment is a failure.
Which is a shame, but one RFC's failure is another one's opportunity.
> I have another ID
> cooking that suggests a better way, going forward, to use them, but I
> do not feel at this time it would be useful to present. One big battle
> at a time.
:)
> That ID is very simple, it basically proposes that all
> diffserv codepoints be passed through ISPs and transit providers
> unchanged, and if any given codepoint must be changed, the only
> permissible change is to 0 (BE), and MUST be not be remarked to
> anything else, especially not CS1.
I like the simplicity, but I also like the split into two groups of 3 bits, say "active bit pattern" (for transport purposes) and "intenden bit pattern" for the sender to transmit intent, which than can be conveyed lossless to the receiver; my gut feeling is that throwing the transport people a bone here might work better, as in the end they are the one that will carry the "burden" of the change. IMHO this has the additional advantage that it will make "tabula rasa" of the existing distict set of bit patterns used in the real world. Which in turn probably is this ideas downfall...
>
>> allow for concurrent use of the inevitable L5S and L6S ;) ). Especially since as far as I can understand l4s actually would like to have a more gradual congestion information stream than classic ECN, and since they need to modify DCTCP anyway to make it save for the wider internet, replacing its ECN response should be well inside the scope of work they already have on their list.
>
> Next up for sce was going to be to find if dctcp could be modified to
> use it. Also, bittorrent.
YES! I endorse this message.
>
>> If I sound a bit miffed, it is because after reading https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03 I do not have the feeling they are trying to build abetter internet, but rather that they are building an internet where I can be a better "product" and customer of newfangled services, and I do not look forward to such a facebooky future with much enthusiasm.
>
> I have rigorously tried to keep the network neutrality debate out of
> this.
And I really do not want to start this here, as I agree that this is about a technical question. That said, I had hoped more out of the l4s promises from an end-user perspective. Then again, it if this is going to happen it needs the ISPs buy-in so it might be politically wise to emphasize the advantages for ISPs.
> dualpi is just another aqm that needs the same thorough
> technical and public evaluation done to it that pie, codel, fq_codel
> were subjected to.
+1, I note that the main argument against fq is "we can do it with two" without a convincing argument why two is better than the few 100s you realistically will never need for fq even on an busy core router.
> The use of ect_1 in dualpi for it is nuts IMHO,
At least it is unclean...
> and
> I'd vastly prefer that another L4S codepoint be added to make it work,
> but any attempt to do so would also require industry consolidation on
> that ID and that would be distracting at this time.
But as I said, L4S aims to change everything so why stop there ;)
>
> It appears, also, ironically, (I have confirmation from several
> sources now) that cake, fq_codel and dualpi are now illegal for an ISP
> to use in their provided equipment under california law.
I will happily look at specific sections of code if pointed to, but I will not actively search for. At least the European net neutrality rules do not make any of these illegal, as the rules allow for traffic management (and special services at extra cost as long as these are not built be restricting the best-effort net, no idea how that should be poiced) (and in the end the rules only apply for ISPs in one's own network one can DPI to a far greater extent if one is such inclined).
> The idea of
> one day having to appear in court to defend our key algorithms reminds
> me of the famous john fogerty case where he demonstrated how blues
> music was made.
I see a nice art-house movie coming up.
>
> I wish I knew a lawyer willing to take on "bufferbloat.net vs the
> state of california", though, as it may come down to that.
>
> I blew up on this part of the issue here:
> http://blog.cerowrt.org/post/net_neutrality_customers/
I read that, but it does not reflect to well on the situation this side of the pond. We had situations where ISPs worked hard (but without success) to switch from their flat-rate access to introduce volume caps, that served a dual purpose, a) extracting more revenue from customers and b) allowing to make "zero-rating" deals with content-providers (which in the end are also payed for by the end-users, indirectly). Sure, this is all normal in business, but that does not necessarily mean that I do need to like being a pawn in a business transaction between global corporations. (I consider this to be at least somewhat related to the net neutrality debate).
Best Regards
Sebastian
>>
>> I hope that the discussion in Prague go well and a compromise/consense can be hashed out as I see different implementations duking it out here, but the overall goal of the competitors seems quite compatible, improving the internet by focussing on latency.
>>
>> Best Regards
>> Sebastian
>>
>>> On Mar 15, 2019, at 11:46, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> Bufferbloat.net's ecn-sane working group members have a co-ordinated response to your efforts brewing but it's not ready yet. We have a worldwide team of linux and freebsd developers co-ordinating on landing code for our competing proposal "Some Congestion Experienced", which we submitted to tsvwg last sunday.
>>>
>>> That draft is under continuous revision, here:
>>>
>>> https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
>>>
>>> Our Linux and FreeBSD team is also flying into prague for SCE presentations at netdevconf and ietf.
>>>
>>> Some background to this: after the L4S/TCP Prague/and dualpi experiments appeared stalled out indefinitely in the IETF, and with our own frustration with IETF processes, bufferbloat.net project members publicly formed our own working group to look into the problems with ecn, back in august of last year.
>>>
>>> Its charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/
>>>
>>> We were unaware, until last month, that the cable industry had 16 months back gone and formed its own private working group also, and was intending to turn the tcp prague/l4s/dualpi IETF "experiments" into an actual DOCSIS standard.
>>>
>>> Our SCE proposal appears to be backward compatible with the existing 10s-100s of millions of ecn-enabled fq_codel[1] and sch_cake[2] deployments, and doesn't require any changes to any RFC3168 tcps (or any tcp-friendly congestion control) at all in order to basically work. tcp-prague is subtly incompatible with that, and dualpi, more so. Our proposal is different also, it proposes some receiver side changes in order to get the full benefit of SCE while remaining backward compatible with the existing meaning of the CE codepoint.
>>>
>>> In either case, either approach essentially permanently redefines the ECT_1 codepoint incompatibly, once and for all, and for all time. This is a final battle over the meaning of a single bit in IP, and I expect the debates to be as difficult as the ones described in https://www.ietf.org/rfc/ien/ien137.txt - I would really, really, really prefer that they stay technical and not veer into politics, but I have little hope for that.
>>>
>>> The members of the ecn-sane working group are delighted to finally hear that running code for tcp-prague might land this ietf, and look forward to finally testing the whole l4s/tcpprague/dualpi architecture in conjunction with the flent.org 's and irtt's exhaustive suite of tests and servers in the cloud in the coming months, both against our existing, deployed, fq_codel, fq_pie, cake and pie derived solutions and our new SCE proposal. We hope to finally be able to write new tests for flent in particular, that can show tcpprague off in the ways that are important to those developing it. Flent has some basic dctcp tests, but nothing that can get down below a 20ms sample rate on modern hardware. (currently. It's easy to add tests, flent is written in python)
>>>
>>> We also hope that more test tools and implementations in ns2 and ns3 show up for tcpprauge and dualpi show up soon also, from members of those projects.
>>>
>>> Note: sunday's dual-pi linux submission was kicked back from the linux networking developers due to some technical and legal problems with linux net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/ ) , and I do hope that a corrected version lands soon, so we can safely test it with current versions of OpenWrt, etc.
>>>
>>> Finally, running code. Will we find consensus?
>>>
>>> Thx!
>>>
>>>
>>> [1] https://tools.ietf.org/html/rfc8290
>>> [2] sch_cake was available for 3 years out of tree and was mainlined last august, in linux 4.19. It is partially described by "Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways" "https://arxiv.org/pdf/1804.07617.pdf
>>>
>>> A second paper describing its fq_codel-derived "cobalt" AQM algorithm is awaiting publication in a peer reviewed journal. It has been part of openwrt and the related sqm-scripts for many years and is widely deployed on multiple commercial products, such as those from eero and evenroute.
>>>
>>> Cake has a docsis specific mode which we longed for cablelabs to evaluate.
>>>
>>>
>>> On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
>>> Forwarding to tcpm & iccrg - apologies if you were already on one of the lists that received this.
>>>
>>> Olivier has been working hard on integrating the pieces of a Linux implementation of TCP Prague, and is close to having a version ported against the tip of the Linux mainline tree. This is his request for more people to get involved.
>>>
>>>
>>> Bob
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>>> Date: Wed, 6 Mar 2019 10:26:05 +0000
>>> From: Tilmans, Olivier (Nokia - BE/Antwerp) <olivier.tilmans@nokia-bell-labs.com>
>>> To: hackathon@ietf.org <hackathon@ietf.org>, tcpprague@ietf.org <tcpprague@ietf.org>
>>> CC: dlebrun@google.com <dlebrun@google.com>, Joakim Misund <joakim.misund@gmail.com>, Bob Briscoe <research@bobbriscoe.net>, Quentin De Coninck <quentin.deconinck@uclouvain.be>, François Michel <francois.michel@uclouvain.be>, Mirja Kuehlewind <mirja.kuehlewind@tik.ee.ethz.ch>, Maxime Piraux <maxime.piraux@uclouvain.be>, Olga Albisser <olga@albisser.org>, Fabien Duchêne <fabien.duchene@uclouvain.be>, De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
>>>
>>> Hi all,
>>>
>>> We'll be working on the "TCP Prague" congestion control/L4S architecture during the IETF-104 hackaton.
>>> This topics aims at accelerating the work that started during the IETF93 (coincidentally also in Prague), in order to get TCP Prague to an 'usable' state—i.e., meet the safety requirements and have supporting materials (e.g., VMs, labs) to let people experiment with it. Depending on people's interest, prototyping something similar for QUIC is another possible output.
>>>
>>> Details and links to resources/supporting drafts are available at https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague and copied below.
>>> Additionally, few topics will presented during netdev 0x13 the week before.
>>>
>>> See you in Prague.
>>>
>>> Best,
>>> Olivier
>>>
>>>
>>> Implementation and experimentation of TCP Prague/L4S
>>>
>>> * Champion
>>> * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
>>> * Projects
>>> * Prototype the "TCP Prague" congestion control on Linux
>>> * Finalize the implementation of accurate ECN (draft conformance), and port it on Linux v5.x * Build tooling around L4S to let people experiment with the technology (e.g., virtual machine, or mininet labs)
>>> * Work towards "QUIC Prague"
>>> * Resources
>>> * TCP Prague
>>> * Repository — https://github.com/L4STeam/tcp-prague
>>> * Requirements — https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
>>> * Accurate ECN
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
>>> * Implementation for Linux v4.17 — https://github.com/mirjak/linux-accecn
>>> * Past netdev talk — https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
>>> * Paced Chirping * Repository — https://github.com/JoakimMisund/PacedChirping
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-chirp
>>> * L4S architecture
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
>>> * DualPI2 AQM
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
>>> * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
>>> * RITE Project — https://riteproject.eu/dctth/#code
>>> _______________________________________________
>>> tcpPrague mailing list
>>> tcpPrague@ietf.org
>>> https://www.ietf.org/mailman/listinfo/tcpprague
>>> _______________________________________________
>>> iccrg mailing list
>>> iccrg@irtf.org
>>> https://www.irtf.org/mailman/listinfo/iccrg
>>>
>>>
>>> --
>>>
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 15:52 ` Sebastian Moeller
@ 2019-03-15 17:01 ` David P. Reed
2019-03-15 17:45 ` Sebastian Moeller
` (2 more replies)
0 siblings, 3 replies; 105+ messages in thread
From: David P. Reed @ 2019-03-15 17:01 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Dave Täht, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 18339 bytes --]
The absolute fundamental issue with diffserv, IMO, is that the carriers cannot agree on an actual specification of what routers and gateways are supposed to do, while being compatible with the core principle of the IP layer: do not hold packets if they cause increasing queueing delay. (this is in the Best Practices RFC)
And it's not for lack of trying. Dave Clark led a working group at the MIT communications futures program (where I was a principle) that included most major backbone providers' key folks that was specifically focused on getting a widespread agreement on what any of the code points might mean, not as bullshit English descriptions of what kind of traffic each one represented, but as an operational description of what should be done by a router to manage its queues.
They couldn't even agree (after about 18 months of meetings) about what the bullshit English intent was, much less what operational semantics in the packet forwarding network had to be.
So if the responsible network engineers in the carriers cannot agree on anything, IETF is wasting its time.
-----Original Message-----
From: "Sebastian Moeller" <moeller0@gmx.de>
Sent: Friday, March 15, 2019 11:52am
To: "Dave Täht" <dave.taht@gmail.com>
Cc: ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Hi Dave,
> On Mar 15, 2019, at 15:06, Dave Taht <dave.taht@gmail.com> wrote:
>
> I would really prefer to move this discussion to the ecn-sane mailing
> list, as IMHO, ecn is generally such a tiny thing needed for good
> congestion control compared to better transports like pacing + bbr,
> and things like bql, fq, and aqm using drop.
>
> I'm going to keep cc-ing that list in the hope that we can keep better
> track of the discussion here.
Ah, sure, I basically wanted to avoid annoying the ietf lists as I feel out of place there, having only had a cursory read of the relevant RFCs.
>
> On Fri, Mar 15, 2019 at 6:01 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi Dave,
>>
>> I pruned the CC list as I am out of my league here and want to restrict the radius of my embarrassment to those that already know my level of incompetence before hand.
>
> IMHO, your work on educating the OpenWrt community over the years on
> how to use sqm, makes you much more than "only a grasshopper". You
> have a firm grip on what can be achieved in the real world.
>
>>
>> That said, having read through the L4S architecture description and the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the conclusion, that this is a mess.
>
> I am so glad someone other than I has now read it.
>
>> The L4S project proposes a really wide-ranging change of basically the internet (but allow a concurrent operation with legacy probably to cater to the fact that an internet-wide flag-day seems daunting to organize). But then they chicken out when figuring out how to differentiate between their new and the old by proposing to use ECT(1) for a purpose outside of its nominal purpose namely explicit congestion notification, why not think bolder? If the plan is to change everything the feasibility can not hinge upon the ability to re-using one old legacy bit, can it...
>> Conceptually it seems much cleaner to use ECT(1) for congestion notification directly (there are already proposals out there and SCE just added another fully back-ward compatible one) and find another way to mark l4s traffic, sure that is going to be hard and inconvenient, but if you set out to change the internet that is par for the course.
>> IMHO they would do more good if they actually fought for a better use of the 6 DSCP bits instead. (say by splitting into two groups of 3 bits, one group for reduced diffserv and one group for new features, that would even
>
> The existing diffserv deployment is a failure.
Which is a shame, but one RFC's failure is another one's opportunity.
> I have another ID
> cooking that suggests a better way, going forward, to use them, but I
> do not feel at this time it would be useful to present. One big battle
> at a time.
:)
> That ID is very simple, it basically proposes that all
> diffserv codepoints be passed through ISPs and transit providers
> unchanged, and if any given codepoint must be changed, the only
> permissible change is to 0 (BE), and MUST be not be remarked to
> anything else, especially not CS1.
I like the simplicity, but I also like the split into two groups of 3 bits, say "active bit pattern" (for transport purposes) and "intenden bit pattern" for the sender to transmit intent, which than can be conveyed lossless to the receiver; my gut feeling is that throwing the transport people a bone here might work better, as in the end they are the one that will carry the "burden" of the change. IMHO this has the additional advantage that it will make "tabula rasa" of the existing distict set of bit patterns used in the real world. Which in turn probably is this ideas downfall...
>
>> allow for concurrent use of the inevitable L5S and L6S ;) ). Especially since as far as I can understand l4s actually would like to have a more gradual congestion information stream than classic ECN, and since they need to modify DCTCP anyway to make it save for the wider internet, replacing its ECN response should be well inside the scope of work they already have on their list.
>
> Next up for sce was going to be to find if dctcp could be modified to
> use it. Also, bittorrent.
YES! I endorse this message.
>
>> If I sound a bit miffed, it is because after reading https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03 I do not have the feeling they are trying to build abetter internet, but rather that they are building an internet where I can be a better "product" and customer of newfangled services, and I do not look forward to such a facebooky future with much enthusiasm.
>
> I have rigorously tried to keep the network neutrality debate out of
> this.
And I really do not want to start this here, as I agree that this is about a technical question. That said, I had hoped more out of the l4s promises from an end-user perspective. Then again, it if this is going to happen it needs the ISPs buy-in so it might be politically wise to emphasize the advantages for ISPs.
> dualpi is just another aqm that needs the same thorough
> technical and public evaluation done to it that pie, codel, fq_codel
> were subjected to.
+1, I note that the main argument against fq is "we can do it with two" without a convincing argument why two is better than the few 100s you realistically will never need for fq even on an busy core router.
> The use of ect_1 in dualpi for it is nuts IMHO,
At least it is unclean...
> and
> I'd vastly prefer that another L4S codepoint be added to make it work,
> but any attempt to do so would also require industry consolidation on
> that ID and that would be distracting at this time.
But as I said, L4S aims to change everything so why stop there ;)
>
> It appears, also, ironically, (I have confirmation from several
> sources now) that cake, fq_codel and dualpi are now illegal for an ISP
> to use in their provided equipment under california law.
I will happily look at specific sections of code if pointed to, but I will not actively search for. At least the European net neutrality rules do not make any of these illegal, as the rules allow for traffic management (and special services at extra cost as long as these are not built be restricting the best-effort net, no idea how that should be poiced) (and in the end the rules only apply for ISPs in one's own network one can DPI to a far greater extent if one is such inclined).
> The idea of
> one day having to appear in court to defend our key algorithms reminds
> me of the famous john fogerty case where he demonstrated how blues
> music was made.
I see a nice art-house movie coming up.
>
> I wish I knew a lawyer willing to take on "bufferbloat.net vs the
> state of california", though, as it may come down to that.
>
> I blew up on this part of the issue here:
> http://blog.cerowrt.org/post/net_neutrality_customers/
I read that, but it does not reflect to well on the situation this side of the pond. We had situations where ISPs worked hard (but without success) to switch from their flat-rate access to introduce volume caps, that served a dual purpose, a) extracting more revenue from customers and b) allowing to make "zero-rating" deals with content-providers (which in the end are also payed for by the end-users, indirectly). Sure, this is all normal in business, but that does not necessarily mean that I do need to like being a pawn in a business transaction between global corporations. (I consider this to be at least somewhat related to the net neutrality debate).
Best Regards
Sebastian
>>
>> I hope that the discussion in Prague go well and a compromise/consense can be hashed out as I see different implementations duking it out here, but the overall goal of the competitors seems quite compatible, improving the internet by focussing on latency.
>>
>> Best Regards
>> Sebastian
>>
>>> On Mar 15, 2019, at 11:46, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> Bufferbloat.net's ecn-sane working group members have a co-ordinated response to your efforts brewing but it's not ready yet. We have a worldwide team of linux and freebsd developers co-ordinating on landing code for our competing proposal "Some Congestion Experienced", which we submitted to tsvwg last sunday.
>>>
>>> That draft is under continuous revision, here:
>>>
>>> https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
>>>
>>> Our Linux and FreeBSD team is also flying into prague for SCE presentations at netdevconf and ietf.
>>>
>>> Some background to this: after the L4S/TCP Prague/and dualpi experiments appeared stalled out indefinitely in the IETF, and with our own frustration with IETF processes, bufferbloat.net project members publicly formed our own working group to look into the problems with ecn, back in august of last year.
>>>
>>> Its charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/
>>>
>>> We were unaware, until last month, that the cable industry had 16 months back gone and formed its own private working group also, and was intending to turn the tcp prague/l4s/dualpi IETF "experiments" into an actual DOCSIS standard.
>>>
>>> Our SCE proposal appears to be backward compatible with the existing 10s-100s of millions of ecn-enabled fq_codel[1] and sch_cake[2] deployments, and doesn't require any changes to any RFC3168 tcps (or any tcp-friendly congestion control) at all in order to basically work. tcp-prague is subtly incompatible with that, and dualpi, more so. Our proposal is different also, it proposes some receiver side changes in order to get the full benefit of SCE while remaining backward compatible with the existing meaning of the CE codepoint.
>>>
>>> In either case, either approach essentially permanently redefines the ECT_1 codepoint incompatibly, once and for all, and for all time. This is a final battle over the meaning of a single bit in IP, and I expect the debates to be as difficult as the ones described in https://www.ietf.org/rfc/ien/ien137.txt - I would really, really, really prefer that they stay technical and not veer into politics, but I have little hope for that.
>>>
>>> The members of the ecn-sane working group are delighted to finally hear that running code for tcp-prague might land this ietf, and look forward to finally testing the whole l4s/tcpprague/dualpi architecture in conjunction with the flent.org 's and irtt's exhaustive suite of tests and servers in the cloud in the coming months, both against our existing, deployed, fq_codel, fq_pie, cake and pie derived solutions and our new SCE proposal. We hope to finally be able to write new tests for flent in particular, that can show tcpprague off in the ways that are important to those developing it. Flent has some basic dctcp tests, but nothing that can get down below a 20ms sample rate on modern hardware. (currently. It's easy to add tests, flent is written in python)
>>>
>>> We also hope that more test tools and implementations in ns2 and ns3 show up for tcpprauge and dualpi show up soon also, from members of those projects.
>>>
>>> Note: sunday's dual-pi linux submission was kicked back from the linux networking developers due to some technical and legal problems with linux net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/ ) , and I do hope that a corrected version lands soon, so we can safely test it with current versions of OpenWrt, etc.
>>>
>>> Finally, running code. Will we find consensus?
>>>
>>> Thx!
>>>
>>>
>>> [1] https://tools.ietf.org/html/rfc8290
>>> [2] sch_cake was available for 3 years out of tree and was mainlined last august, in linux 4.19. It is partially described by "Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways" "https://arxiv.org/pdf/1804.07617.pdf
>>>
>>> A second paper describing its fq_codel-derived "cobalt" AQM algorithm is awaiting publication in a peer reviewed journal. It has been part of openwrt and the related sqm-scripts for many years and is widely deployed on multiple commercial products, such as those from eero and evenroute.
>>>
>>> Cake has a docsis specific mode which we longed for cablelabs to evaluate.
>>>
>>>
>>> On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
>>> Forwarding to tcpm & iccrg - apologies if you were already on one of the lists that received this.
>>>
>>> Olivier has been working hard on integrating the pieces of a Linux implementation of TCP Prague, and is close to having a version ported against the tip of the Linux mainline tree. This is his request for more people to get involved.
>>>
>>>
>>> Bob
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>>> Date: Wed, 6 Mar 2019 10:26:05 +0000
>>> From: Tilmans, Olivier (Nokia - BE/Antwerp) <olivier.tilmans@nokia-bell-labs.com>
>>> To: hackathon@ietf.org <hackathon@ietf.org>, tcpprague@ietf.org <tcpprague@ietf.org>
>>> CC: dlebrun@google.com <dlebrun@google.com>, Joakim Misund <joakim.misund@gmail.com>, Bob Briscoe <research@bobbriscoe.net>, Quentin De Coninck <quentin.deconinck@uclouvain.be>, François Michel <francois.michel@uclouvain.be>, Mirja Kuehlewind <mirja.kuehlewind@tik.ee.ethz.ch>, Maxime Piraux <maxime.piraux@uclouvain.be>, Olga Albisser <olga@albisser.org>, Fabien Duchêne <fabien.duchene@uclouvain.be>, De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
>>>
>>> Hi all,
>>>
>>> We'll be working on the "TCP Prague" congestion control/L4S architecture during the IETF-104 hackaton.
>>> This topics aims at accelerating the work that started during the IETF93 (coincidentally also in Prague), in order to get TCP Prague to an 'usable' state—i.e., meet the safety requirements and have supporting materials (e.g., VMs, labs) to let people experiment with it. Depending on people's interest, prototyping something similar for QUIC is another possible output.
>>>
>>> Details and links to resources/supporting drafts are available at https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague and copied below.
>>> Additionally, few topics will presented during netdev 0x13 the week before.
>>>
>>> See you in Prague.
>>>
>>> Best,
>>> Olivier
>>>
>>>
>>> Implementation and experimentation of TCP Prague/L4S
>>>
>>> * Champion
>>> * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
>>> * Projects
>>> * Prototype the "TCP Prague" congestion control on Linux
>>> * Finalize the implementation of accurate ECN (draft conformance), and port it on Linux v5.x * Build tooling around L4S to let people experiment with the technology (e.g., virtual machine, or mininet labs)
>>> * Work towards "QUIC Prague"
>>> * Resources
>>> * TCP Prague
>>> * Repository — https://github.com/L4STeam/tcp-prague
>>> * Requirements — https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
>>> * Accurate ECN
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
>>> * Implementation for Linux v4.17 — https://github.com/mirjak/linux-accecn
>>> * Past netdev talk — https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
>>> * Paced Chirping * Repository — https://github.com/JoakimMisund/PacedChirping
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-chirp
>>> * L4S architecture
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
>>> * DualPI2 AQM
>>> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
>>> * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
>>> * Upcoming netdev talk — https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
>>> * RITE Project — https://riteproject.eu/dctth/#code
>>> _______________________________________________
>>> tcpPrague mailing list
>>> tcpPrague@ietf.org
>>> https://www.ietf.org/mailman/listinfo/tcpprague
>>> _______________________________________________
>>> iccrg mailing list
>>> iccrg@irtf.org
>>> https://www.irtf.org/mailman/listinfo/iccrg
>>>
>>>
>>> --
>>>
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/ecn-sane
[-- Attachment #2: Type: text/html, Size: 22241 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 17:01 ` [Bloat] [Ecn-sane] " David P. Reed
@ 2019-03-15 17:45 ` Sebastian Moeller
2019-03-15 18:36 ` Mikael Abrahamsson
2019-03-16 4:04 ` Jonathan Morton
2 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-15 17:45 UTC (permalink / raw)
To: David P. Reed; +Cc: Dave Täht, ecn-sane, bloat
On March 15, 2019 6:01:23 PM GMT+01:00, "David P. Reed" <dpreed@deepplum.com> wrote:
>
>The absolute fundamental issue with diffserv, IMO, is that the carriers
>cannot agree on an actual specification of what routers and gateways
>are supposed to do, while being compatible with the core principle of
>the IP layer: do not hold packets if they cause increasing queueing
>delay. (this is in the Best Practices RFC)
>
IMHO it is the Charme of the 2 times 3 bits approach, that carriers get 3 bits they can do with whatever they want. As VLAN tags and MPLS? only support 3 bit priorities this looks to me like a match made in heaven, and we get 3 bits with end to end guarantees. Not that rolling that out would ever work in reality....
Best Regards
Sebastian
And yes, this is not an idea I came up with, I just forgot who proposed that initially.
>And it's not for lack of trying. Dave Clark led a working group at the
>MIT communications futures program (where I was a principle) that
>included most major backbone providers' key folks that was specifically
>focused on getting a widespread agreement on what any of the code
>points might mean, not as bullshit English descriptions of what kind of
>traffic each one represented, but as an operational description of what
>should be done by a router to manage its queues.
>
>They couldn't even agree (after about 18 months of meetings) about what
>the bullshit English intent was, much less what operational semantics
>in the packet forwarding network had to be.
>
>So if the responsible network engineers in the carriers cannot agree on
>anything, IETF is wasting its time.
>
>
>
>
>-----Original Message-----
>From: "Sebastian Moeller" <moeller0@gmx.de>
>Sent: Friday, March 15, 2019 11:52am
>To: "Dave Täht" <dave.taht@gmail.com>
>Cc: ecn-sane@lists.bufferbloat.net, "bloat"
><bloat@lists.bufferbloat.net>
>Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation
>and experimentation of TCP Prague/L4S hackaton at IETF104
>
>
>
>Hi Dave,
>
>
>> On Mar 15, 2019, at 15:06, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> I would really prefer to move this discussion to the ecn-sane mailing
>> list, as IMHO, ecn is generally such a tiny thing needed for good
>> congestion control compared to better transports like pacing + bbr,
>> and things like bql, fq, and aqm using drop.
>>
>> I'm going to keep cc-ing that list in the hope that we can keep
>better
>> track of the discussion here.
>
>Ah, sure, I basically wanted to avoid annoying the ietf lists as I feel
>out of place there, having only had a cursory read of the relevant
>RFCs.
>
>
>>
>> On Fri, Mar 15, 2019 at 6:01 AM Sebastian Moeller <moeller0@gmx.de>
>wrote:
>>>
>>> Hi Dave,
>>>
>>> I pruned the CC list as I am out of my league here and want to
>restrict the radius of my embarrassment to those that already know my
>level of incompetence before hand.
>>
>> IMHO, your work on educating the OpenWrt community over the years on
>> how to use sqm, makes you much more than "only a grasshopper". You
>> have a firm grip on what can be achieved in the real world.
>>
>>>
>>> That said, having read through the L4S architecture description and
>the related appendices of draft-ietf-tsvwg-ecn-l4s-id-05 I came to the
>conclusion, that this is a mess.
>>
>> I am so glad someone other than I has now read it.
>>
>>> The L4S project proposes a really wide-ranging change of basically
>the internet (but allow a concurrent operation with legacy probably to
>cater to the fact that an internet-wide flag-day seems daunting to
>organize). But then they chicken out when figuring out how to
>differentiate between their new and the old by proposing to use ECT(1)
>for a purpose outside of its nominal purpose namely explicit congestion
>notification, why not think bolder? If the plan is to change everything
>the feasibility can not hinge upon the ability to re-using one old
>legacy bit, can it...
>>> Conceptually it seems much cleaner to use ECT(1) for congestion
>notification directly (there are already proposals out there and SCE
>just added another fully back-ward compatible one) and find another way
>to mark l4s traffic, sure that is going to be hard and inconvenient,
>but if you set out to change the internet that is par for the course.
>>> IMHO they would do more good if they actually fought for a better
>use of the 6 DSCP bits instead. (say by splitting into two groups of 3
>bits, one group for reduced diffserv and one group for new features,
>that would even
>>
>> The existing diffserv deployment is a failure.
>
> Which is a shame, but one RFC's failure is another one's opportunity.
>
>
>> I have another ID
>> cooking that suggests a better way, going forward, to use them, but I
>> do not feel at this time it would be useful to present. One big
>battle
>> at a time.
>
>:)
>
>> That ID is very simple, it basically proposes that all
>> diffserv codepoints be passed through ISPs and transit providers
>> unchanged, and if any given codepoint must be changed, the only
>> permissible change is to 0 (BE), and MUST be not be remarked to
>> anything else, especially not CS1.
>
>I like the simplicity, but I also like the split into two groups of 3
>bits, say "active bit pattern" (for transport purposes) and "intenden
>bit pattern" for the sender to transmit intent, which than can be
>conveyed lossless to the receiver; my gut feeling is that throwing the
>transport people a bone here might work better, as in the end they are
>the one that will carry the "burden" of the change. IMHO this has the
>additional advantage that it will make "tabula rasa" of the existing
>distict set of bit patterns used in the real world. Which in turn
>probably is this ideas downfall...
>
>
>>
>>> allow for concurrent use of the inevitable L5S and L6S ;) ).
>Especially since as far as I can understand l4s actually would like to
>have a more gradual congestion information stream than classic ECN, and
>since they need to modify DCTCP anyway to make it save for the wider
>internet, replacing its ECN response should be well inside the scope of
>work they already have on their list.
>>
>> Next up for sce was going to be to find if dctcp could be modified to
>> use it. Also, bittorrent.
>
> YES! I endorse this message.
>
>>
>>> If I sound a bit miffed, it is because after reading
>https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03 I do not have
>the feeling they are trying to build abetter internet, but rather that
>they are building an internet where I can be a better "product" and
>customer of newfangled services, and I do not look forward to such a
>facebooky future with much enthusiasm.
>>
>> I have rigorously tried to keep the network neutrality debate out of
>> this.
>
>And I really do not want to start this here, as I agree that this is
>about a technical question. That said, I had hoped more out of the l4s
>promises from an end-user perspective. Then again, it if this is going
>to happen it needs the ISPs buy-in so it might be politically wise to
>emphasize the advantages for ISPs.
>
>
>> dualpi is just another aqm that needs the same thorough
>> technical and public evaluation done to it that pie, codel, fq_codel
>> were subjected to.
>
>+1, I note that the main argument against fq is "we can do it with two"
>without a convincing argument why two is better than the few 100s you
>realistically will never need for fq even on an busy core router.
>
>> The use of ect_1 in dualpi for it is nuts IMHO,
>
> At least it is unclean...
>
>> and
>> I'd vastly prefer that another L4S codepoint be added to make it
>work,
>> but any attempt to do so would also require industry consolidation on
>> that ID and that would be distracting at this time.
>
> But as I said, L4S aims to change everything so why stop there ;)
>
>>
>> It appears, also, ironically, (I have confirmation from several
>> sources now) that cake, fq_codel and dualpi are now illegal for an
>ISP
>> to use in their provided equipment under california law.
>
>I will happily look at specific sections of code if pointed to, but I
>will not actively search for. At least the European net neutrality
>rules do not make any of these illegal, as the rules allow for traffic
>management (and special services at extra cost as long as these are not
>built be restricting the best-effort net, no idea how that should be
>poiced) (and in the end the rules only apply for ISPs in one's own
>network one can DPI to a far greater extent if one is such inclined).
>
>
>> The idea of
>> one day having to appear in court to defend our key algorithms
>reminds
>> me of the famous john fogerty case where he demonstrated how blues
>> music was made.
>
> I see a nice art-house movie coming up.
>
>
>>
>> I wish I knew a lawyer willing to take on "bufferbloat.net vs the
>> state of california", though, as it may come down to that.
>>
>> I blew up on this part of the issue here:
>> http://blog.cerowrt.org/post/net_neutrality_customers/
>
>I read that, but it does not reflect to well on the situation this side
>of the pond. We had situations where ISPs worked hard (but without
>success) to switch from their flat-rate access to introduce volume
>caps, that served a dual purpose, a) extracting more revenue from
>customers and b) allowing to make "zero-rating" deals with
>content-providers (which in the end are also payed for by the
>end-users, indirectly). Sure, this is all normal in business, but that
>does not necessarily mean that I do need to like being a pawn in a
>business transaction between global corporations. (I consider this to
>be at least somewhat related to the net neutrality debate).
>
>Best Regards
> Sebastian
>
>
>>>
>>> I hope that the discussion in Prague go well and a
>compromise/consense can be hashed out as I see different
>implementations duking it out here, but the overall goal of the
>competitors seems quite compatible, improving the internet by focussing
>on latency.
>>>
>>> Best Regards
>>> Sebastian
>>>
>>>> On Mar 15, 2019, at 11:46, Dave Taht <dave.taht@gmail.com> wrote:
>>>>
>>>> Bufferbloat.net's ecn-sane working group members have a
>co-ordinated response to your efforts brewing but it's not ready yet.
>We have a worldwide team of linux and freebsd developers co-ordinating
>on landing code for our competing proposal "Some Congestion
>Experienced", which we submitted to tsvwg last sunday.
>>>>
>>>> That draft is under continuous revision, here:
>>>>
>>>>
>https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
>>>>
>>>> Our Linux and FreeBSD team is also flying into prague for SCE
>presentations at netdevconf and ietf.
>>>>
>>>> Some background to this: after the L4S/TCP Prague/and dualpi
>experiments appeared stalled out indefinitely in the IETF, and with our
>own frustration with IETF processes, bufferbloat.net project members
>publicly formed our own working group to look into the problems with
>ecn, back in august of last year.
>>>>
>>>> Its charter is here:
>https://www.bufferbloat.net/projects/ecn-sane/wiki/
>>>>
>>>> We were unaware, until last month, that the cable industry had 16
>months back gone and formed its own private working group also, and was
>intending to turn the tcp prague/l4s/dualpi IETF "experiments" into an
>actual DOCSIS standard.
>>>>
>>>> Our SCE proposal appears to be backward compatible with the
>existing 10s-100s of millions of ecn-enabled fq_codel[1] and
>sch_cake[2] deployments, and doesn't require any changes to any RFC3168
>tcps (or any tcp-friendly congestion control) at all in order to
>basically work. tcp-prague is subtly incompatible with that, and
>dualpi, more so. Our proposal is different also, it proposes some
>receiver side changes in order to get the full benefit of SCE while
>remaining backward compatible with the existing meaning of the CE
>codepoint.
>>>>
>>>> In either case, either approach essentially permanently redefines
>the ECT_1 codepoint incompatibly, once and for all, and for all time.
>This is a final battle over the meaning of a single bit in IP, and I
>expect the debates to be as difficult as the ones described in
>https://www.ietf.org/rfc/ien/ien137.txt - I would really, really,
>really prefer that they stay technical and not veer into politics, but
>I have little hope for that.
>>>>
>>>> The members of the ecn-sane working group are delighted to finally
>hear that running code for tcp-prague might land this ietf, and look
>forward to finally testing the whole l4s/tcpprague/dualpi architecture
>in conjunction with the flent.org 's and irtt's exhaustive suite of
>tests and servers in the cloud in the coming months, both against our
>existing, deployed, fq_codel, fq_pie, cake and pie derived solutions
>and our new SCE proposal. We hope to finally be able to write new tests
>for flent in particular, that can show tcpprague off in the ways that
>are important to those developing it. Flent has some basic dctcp tests,
>but nothing that can get down below a 20ms sample rate on modern
>hardware. (currently. It's easy to add tests, flent is written in
>python)
>>>>
>>>> We also hope that more test tools and implementations in ns2 and
>ns3 show up for tcpprauge and dualpi show up soon also, from members of
>those projects.
>>>>
>>>> Note: sunday's dual-pi linux submission was kicked back from the
>linux networking developers due to some technical and legal problems
>with linux net-next HEAD ( https://patchwork.ozlabs.org/patch/1054521/
>) , and I do hope that a corrected version lands soon, so we can safely
>test it with current versions of OpenWrt, etc.
>>>>
>>>> Finally, running code. Will we find consensus?
>>>>
>>>> Thx!
>>>>
>>>>
>>>> [1] https://tools.ietf.org/html/rfc8290
>>>> [2] sch_cake was available for 3 years out of tree and was
>mainlined last august, in linux 4.19. It is partially described by
>"Piece of CAKE: A Comprehensive Queue Management Solution for Home
>Gateways" "https://arxiv.org/pdf/1804.07617.pdf
>>>>
>>>> A second paper describing its fq_codel-derived "cobalt" AQM
>algorithm is awaiting publication in a peer reviewed journal. It has
>been part of openwrt and the related sqm-scripts for many years and is
>widely deployed on multiple commercial products, such as those from
>eero and evenroute.
>>>>
>>>> Cake has a docsis specific mode which we longed for cablelabs to
>evaluate.
>>>>
>>>>
>>>> On Fri, Mar 15, 2019 at 2:33 AM Bob Briscoe <ietf@bobbriscoe.net>
>wrote:
>>>> Forwarding to tcpm & iccrg - apologies if you were already on one
>of the lists that received this.
>>>>
>>>> Olivier has been working hard on integrating the pieces of a Linux
>implementation of TCP Prague, and is close to having a version ported
>against the tip of the Linux mainline tree. This is his request for
>more people to get involved.
>>>>
>>>>
>>>> Bob
>>>>
>>>>
>>>> -------- Forwarded Message --------
>>>> Subject: [tcpPrague] Implementation and experimentation of TCP
>Prague/L4S hackaton at IETF104
>>>> Date: Wed, 6 Mar 2019 10:26:05 +0000
>>>> From: Tilmans, Olivier (Nokia - BE/Antwerp)
><olivier.tilmans@nokia-bell-labs.com>
>>>> To: hackathon@ietf.org <hackathon@ietf.org>, tcpprague@ietf.org
><tcpprague@ietf.org>
>>>> CC: dlebrun@google.com <dlebrun@google.com>, Joakim Misund
><joakim.misund@gmail.com>, Bob Briscoe <research@bobbriscoe.net>,
>Quentin De Coninck <quentin.deconinck@uclouvain.be>, François Michel
><francois.michel@uclouvain.be>, Mirja Kuehlewind
><mirja.kuehlewind@tik.ee.ethz.ch>, Maxime Piraux
><maxime.piraux@uclouvain.be>, Olga Albisser <olga@albisser.org>, Fabien
>Duchêne <fabien.duchene@uclouvain.be>, De Schepper, Koen (Nokia -
>BE/Antwerp) <koen.de_schepper@nokia-bell-labs.com>
>>>>
>>>> Hi all,
>>>>
>>>> We'll be working on the "TCP Prague" congestion control/L4S
>architecture during the IETF-104 hackaton.
>>>> This topics aims at accelerating the work that started during the
>IETF93 (coincidentally also in Prague), in order to get TCP Prague to
>an 'usable' state—i.e., meet the safety requirements and have
>supporting materials (e.g., VMs, labs) to let people experiment with
>it. Depending on people's interest, prototyping something similar for
>QUIC is another possible output.
>>>>
>>>> Details and links to resources/supporting drafts are available at
>https://trac.ietf.org/trac/ietf/meeting/wiki/104hackathon#tcp-prague
>and copied below.
>>>> Additionally, few topics will presented during netdev 0x13 the week
>before.
>>>>
>>>> See you in Prague.
>>>>
>>>> Best,
>>>> Olivier
>>>>
>>>>
>>>> Implementation and experimentation of TCP Prague/L4S
>>>>
>>>> * Champion
>>>> * Olivier Tilmans <olivier.tilmans at nokia-bell-labs.com>
>>>> * Projects
>>>> * Prototype the "TCP Prague" congestion control on Linux
>>>> * Finalize the implementation of accurate ECN (draft conformance),
>and port it on Linux v5.x * Build tooling around L4S to let people
>experiment with the technology (e.g., virtual machine, or mininet labs)
>>>> * Work towards "QUIC Prague"
>>>> * Resources
>>>> * TCP Prague
>>>> * Repository — https://github.com/L4STeam/tcp-prague
>>>> * Requirements —
>https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-21
>>>> * Upcoming netdev talk —
>https://netdevconf.org/0x13/session.html?talk-tcp-prague-l4s
>>>> * Accurate ECN
>>>> * Specs —
>https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07
>>>> * Implementation for Linux v4.17 —
>https://github.com/mirjak/linux-accecn
>>>> * Past netdev talk —
>https://www.netdevconf.org/2.2/session.html?kuhlewind-accecn-talk
>>>> * Paced Chirping * Repository —
>https://github.com/JoakimMisund/PacedChirping
>>>> * Upcoming netdev talk —
>https://netdevconf.org/0x13/session.html?talk-chirp
>>>> * L4S architecture
>>>> * Specs — https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03
>>>> * DualPI2 AQM
>>>> * Specs —
>https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
>>>> * Repository — https://github.com/L4STeam/sch_dualpi2_upstream
>>>> * Upcoming netdev talk —
>https://netdevconf.org/0x13/session.html?talk-DUALPI2-AQM
>>>> * RITE Project — https://riteproject.eu/dctth/#code
>>>> _______________________________________________
>>>> tcpPrague mailing list
>>>> tcpPrague@ietf.org
>>>> https://www.ietf.org/mailman/listinfo/tcpprague
>>>> _______________________________________________
>>>> iccrg mailing list
>>>> iccrg@irtf.org
>>>> https://www.irtf.org/mailman/listinfo/iccrg
>>>>
>>>>
>>>> --
>>>>
>>>> Dave Täht
>>>> CTO, TekLibre, LLC
>>>> http://www.teklibre.com
>>>> Tel: 1-831-205-9740
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>
>>
>> --
>>
>> Dave Täht
>> CTO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-831-205-9740
>
>_______________________________________________
>Ecn-sane mailing list
>Ecn-sane@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/ecn-sane
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 14:06 ` Dave Taht
2019-03-15 15:52 ` Sebastian Moeller
@ 2019-03-15 18:07 ` Mikael Abrahamsson
1 sibling, 0 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-15 18:07 UTC (permalink / raw)
To: Dave Taht; +Cc: Sebastian Moeller, ecn-sane, bloat
On Fri, 15 Mar 2019, Dave Taht wrote:
> It appears, also, ironically, (I have confirmation from several sources
> now) that cake, fq_codel and dualpi are now illegal for an ISP to use in
> their provided equipment under california law. The idea of one day
> having to appear in court to defend our key algorithms reminds me of the
> famous john fogerty case where he demonstrated how blues music was made.
I would love to know more about this. Running an AQM on the customer
access that doesn't prioritize some specific service should be fine, ie it
doesn't explicitly do something the *customer* doesn't benefit from.
Net neutrality should be about what the ISP does to benefit itself, not
what is done on the customer access line (the scheduler per customer) that
just looks at packet flow characteristics and isn't configured to
prioritize any specific content (src/destination/port etc).
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 17:01 ` [Bloat] [Ecn-sane] " David P. Reed
2019-03-15 17:45 ` Sebastian Moeller
@ 2019-03-15 18:36 ` Mikael Abrahamsson
2019-03-15 19:23 ` Sebastian Moeller
` (2 more replies)
2019-03-16 4:04 ` Jonathan Morton
2 siblings, 3 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-15 18:36 UTC (permalink / raw)
To: David P. Reed; +Cc: Sebastian Moeller, ecn-sane, bloat
On Fri, 15 Mar 2019, David P. Reed wrote:
> So if the responsible network engineers in the carriers cannot agree on
> anything, IETF is wasting its time.
The IETF has already said that anything diffserv is domain-internal only.
I have joined the effort of the LE PHB and see if we can get some kind of
agreement and transparancy for a PHB that is aimed at customer access only
and "drop most of me and my pals at any sign of customer access line
congestion", and see if that can be agreed on.
Having a "lower-than-best-effort" diffserve codepoint might work, because
it means worse treatment, not preferential treatment.
The problem with having DSCP CPs that indicate preferential treatment is
typically a ddos magnet. See my emails on this topic on (this? other?)
mailing lists where I try to create a three class buffering system saying
"LE gets 5%, BE and 'everything-else' gets to split the difference".
I even got pushback on this here, and then we're not even close to people
running large ISP networks who see ddos attacks happen hourly.
Saying L4S should "just use diffserv" is as constructive to say "go away
and pound a rock" or "we want that bit pattern so.. screw you".
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 18:36 ` Mikael Abrahamsson
@ 2019-03-15 19:23 ` Sebastian Moeller
2019-03-15 19:32 ` Jonathan Morton
2019-03-16 21:38 ` Holland, Jake
2 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-15 19:23 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: David P. Reed, ecn-sane, bloat
Hi Mikael,
> On Mar 15, 2019, at 19:36, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 15 Mar 2019, David P. Reed wrote:
>
>> So if the responsible network engineers in the carriers cannot agree on anything, IETF is wasting its time.
>
> The IETF has already said that anything diffserv is domain-internal only. I have joined the effort of the LE PHB and see if we can get some kind of agreement and transparancy for a PHB that is aimed at customer access only and "drop most of me and my pals at any sign of customer access line congestion", and see if that can be agreed on.
+1
>
> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
>
> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
Hence splitting it up, three for the current transport domain to do with as it sees fit and 3 for signaling intent; this very much does not give a guarantee that any intermediate hop will follow the intent, but only make it possible for the endpoints to transmit intent. This IMHO is completely compatible with a LE PHB and transports honoring that request.
> See my emails on this topic on (this? other?) mailing lists where I try to create a three class buffering system saying "LE gets 5%, BE and 'everything-else' gets to split the difference".
We can haggle over the numbers but that seems a) sane and b) underspecified...
>
> I even got pushback on this here, and then we're not even close to people running large ISP networks who see ddos attacks happen hourly.
>
> Saying L4S should "just use diffserv" is as constructive to say "go away and pound a rock" or "we want that bit pattern so.. screw you".
But just nodding expertly when they go and claim an unrelated bit in the IP header for their separation l4s vs legacy (as if l4s would be the end all of network design), and then having resorting to modifying so-far not-deployed-at the edge DCTCP (instead of modifying well-deployed TCP) because they already spent the one bit usable to extend ECN for less binary congestion signaling in a backward-compatible fashion... I might be wording things to strongly here, but that is the general gist.
>
> L4S has a much better possibility of actually getting deployment into the wider Internet packet-moving equipment than anything being talked about here.
That is not a high bar to clear though...
> Same with PIE as opposed to FQ_CODEL. I know it's might not be as good,
Debatable, and from my perspective this is the reason to talk about it at all.
> but it fits better into actual silicon
Does it?
> and it's being proposed by people who actually have better channels into the people setting hard requirements.
That would be great if the proposal would throw end-user like me a bone instead of treating me as the product. It would also help if the architectural RFC would not be so breathlessly over-hyping/over-promising... But they really need end-points to switch over to a neutered DCTCP before things start to make sense, so they actually need to convince end-users and so far they are doing a terrible job IMHO. But what do I know...
>
> I suggest you consider joining them instead of opposing them.
Join where? it pretty much looks like a "fait accompli" as they do seem way past the design stages and seem pretty much crystallized in what they see the path forward.
Best Regards
Sebastian
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 18:36 ` Mikael Abrahamsson
2019-03-15 19:23 ` Sebastian Moeller
@ 2019-03-15 19:32 ` Jonathan Morton
2019-03-15 19:44 ` David P. Reed
2019-03-15 20:28 ` Jonathan Foulkes
2019-03-16 21:38 ` Holland, Jake
2 siblings, 2 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-15 19:32 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: David P. Reed, ecn-sane, bloat
> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
>
> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
0: Maximum Throughput (aka Best Effort)
1: Minimum Cost (aka Least Effort)
2: Minimum Latency (aka Maximum Responsiveness)
3: Minimum Loss (aka Maximum Reliability)
It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
But that's a separate topic from ECN per se.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 19:32 ` Jonathan Morton
@ 2019-03-15 19:44 ` David P. Reed
2019-03-15 20:13 ` Jonathan Morton
2019-03-15 20:28 ` Jonathan Foulkes
1 sibling, 1 reply; 105+ messages in thread
From: David P. Reed @ 2019-03-15 19:44 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Mikael Abrahamsson, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 2436 bytes --]
Just to throw in one more thing not well understood by engineers.
Economists I have discussed this with (real ones, not fringe right-wing true believers that the market "just works"), have observed that pricing (even dynamic pricing) of different qualities of service is unstable and extremely unlikely to reflect the correct price for the particular utility of the achieved service quality.
The point of that observation is that even a simple 2 classes of service system (so-called Paris Metro Pricing) is unstable, such that users of such a system will not be encouraged to set the priorities/service types to make system optimal or stable.
I can explain more, but the end user doesn't benefit from multiple choices of class of service at the packet level.
-----Original Message-----
From: "Jonathan Morton" <chromatix99@gmail.com>
Sent: Friday, March 15, 2019 3:32pm
To: "Mikael Abrahamsson" <swmike@swm.pp.se>
Cc: "David P. Reed" <dpreed@deepplum.com>, ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
>
> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
0: Maximum Throughput (aka Best Effort)
1: Minimum Cost (aka Least Effort)
2: Minimum Latency (aka Maximum Responsiveness)
3: Minimum Loss (aka Maximum Reliability)
It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
But that's a separate topic from ECN per se.
- Jonathan Morton
[-- Attachment #2: Type: text/html, Size: 3562 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 19:44 ` David P. Reed
@ 2019-03-15 20:13 ` Jonathan Morton
2019-03-15 23:43 ` David P. Reed
0 siblings, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-15 20:13 UTC (permalink / raw)
To: David P. Reed; +Cc: Mikael Abrahamsson, ecn-sane, bloat
> On 15 Mar, 2019, at 9:44 pm, David P. Reed <dpreed@deepplum.com> wrote:
>
> pricing (even dynamic pricing) of different qualities of service is unstable
An interesting result, but I should note that the four-way optimisation system I described doesn't rely on pricing, only a sufficiently correct implementation of those optimisations at enough bottlenecks to make it worthwhile for applications to mark their traffic appropriately. The technology exists to do so, but is not standardised in a way that makes it usable.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 19:32 ` Jonathan Morton
2019-03-15 19:44 ` David P. Reed
@ 2019-03-15 20:28 ` Jonathan Foulkes
2019-03-15 20:31 ` Dave Taht
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Foulkes @ 2019-03-15 20:28 UTC (permalink / raw)
To: Jonathan Morton; +Cc: ecn-sane, bloat
All this discussion of DSCP marking brings to mind what happened on the Windows platform, where the OS had to suppress ALL DSCP marks, as app authors were trying to game the system.
And even if not trying to ‘game’ it, they have non-obvious reasons why they don’t mark traffic how one would expect. Example:
I know an engineer who works at a cloud-storage solution company, and I asked why a long-standing customer request for DSCP marking (as bulk) was not implemented. His answer was they’d never do that, as that would impact benchmarks against their competitors for which service syncs faster. <sigh>
Which brings me to a question: Is anyone aware of an easy to use Windows app that will allow the user to select an application and tell the OS to mark the traffic (all or by port) with a user selected DSCP level?
There are many guides on using regedit and other error-prone (and geek-only) means of doing this, but is there a simple Windows 10 home app?
Now that Cake is out there with simple DiffServ3 support, it would be nice to lower the priority of cloud-storage services and other bulk traffic by correctly marking it at the origin.
Cheers,
Jonathan Foulkes
> On Mar 15, 2019, at 3:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
>>
>> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
>
> This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
>
> 0: Maximum Throughput (aka Best Effort)
> 1: Minimum Cost (aka Least Effort)
> 2: Minimum Latency (aka Maximum Responsiveness)
> 3: Minimum Loss (aka Maximum Reliability)
>
> It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
>
> The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
>
> But that's a separate topic from ECN per se.
>
> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 20:28 ` Jonathan Foulkes
@ 2019-03-15 20:31 ` Dave Taht
2019-03-15 23:45 ` David P. Reed
0 siblings, 1 reply; 105+ messages in thread
From: Dave Taht @ 2019-03-15 20:31 UTC (permalink / raw)
To: Jonathan Foulkes; +Cc: Jonathan Morton, ecn-sane, bloat
On Fri, Mar 15, 2019 at 1:28 PM Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>
> All this discussion of DSCP marking brings to mind what happened on the Windows platform, where the OS had to suppress ALL DSCP marks, as app authors were trying to game the system.
> And even if not trying to ‘game’ it, they have non-obvious reasons why they don’t mark traffic how one would expect. Example:
>
> I know an engineer who works at a cloud-storage solution company, and I asked why a long-standing customer request for DSCP marking (as bulk) was not implemented. His answer was they’d never do that, as that would impact benchmarks against their competitors for which service syncs faster. <sigh>
>
> Which brings me to a question: Is anyone aware of an easy to use Windows app that will allow the user to select an application and tell the OS to mark the traffic (all or by port) with a user selected DSCP level?
> There are many guides on using regedit and other error-prone (and geek-only) means of doing this, but is there a simple Windows 10 home app?
When I last tried it (years ago), in order to set the tos bits, an
application merely had to have admin privs.
> Now that Cake is out there with simple DiffServ3 support, it would be nice to lower the priority of cloud-storage services and other bulk traffic by correctly marking it at the origin.
>
> Cheers,
>
> Jonathan Foulkes
>
>
> > On Mar 15, 2019, at 3:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >>
> >> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
> >>
> >> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
> >
> > This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
> >
> > 0: Maximum Throughput (aka Best Effort)
> > 1: Minimum Cost (aka Least Effort)
> > 2: Minimum Latency (aka Maximum Responsiveness)
> > 3: Minimum Loss (aka Maximum Reliability)
> >
> > It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
> >
> > The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
> >
> > But that's a separate topic from ECN per se.
> >
> > - Jonathan Morton
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 10:46 ` [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104 Dave Taht
2019-03-15 13:01 ` Sebastian Moeller
@ 2019-03-15 21:34 ` Wesley Eddy
1 sibling, 0 replies; 105+ messages in thread
From: Wesley Eddy @ 2019-03-15 21:34 UTC (permalink / raw)
To: Dave Taht, Bob Briscoe; +Cc: tcpm IETF list, ecn-sane, iccrg IRTF list, bloat
Hi, Dave, strangely it looks like nobody has been copying TSVWG on this
thread, even though that is where the L4S work is happening in the IETF! :)
I just wanted to respond to one part of your message since I am
currently acting as document shepherd for the L4S drafts in TSVWG, and
it seems like maybe you don't know where to find all of the IETF
materials in order to participate (based on the "stalled out
indefinitely" comment below). So in case it's helpful here are some
pointers to where things are kept.
The 3 base L4S documents were adopted in the TSVWG WG, and have been
regularly updated. You can find the charter and milestone dates
(currently June 2019) quite easily on the WG datatracker page:
https://datatracker.ietf.org/group/tsvwg/about/ and of course the
"Documents" tab there takes you to the copies of all the documents.
There have been updates on L4S and presentations/discussions at the IETF
meetings (with minutes and charts posted to the proceedings), as well as
L4S draft reviews and comment threads on the TSVWG list whose archives
are under "List archive". You can find meeting minutes and slide decks
also readily linked from that WG page:
https://datatracker.ietf.org/group/tsvwg/meetings/
There are source code repositories, papers, videos, etc. that the
proponents have also made very easy to find, e.g.
https://riteproject.eu/dctth/#code (and as linked in meeting materials).
Since we (TSVWG chairs) are looking for inputs towards WGLC readiness to
proceed on these, hopefully this information is helpful for you.
On 3/15/2019 6:46 AM, Dave Taht wrote:
> ...
>
> Some background to this: after the L4S/TCP Prague/and dualpi
experiments appeared stalled out indefinitely in the IETF, and with our
own frustration with IETF processes, bufferbloat.net project members
publicly formed our own working group to look into the problems with
ecn, back in august of last year.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 20:13 ` Jonathan Morton
@ 2019-03-15 23:43 ` David P. Reed
2019-03-16 1:26 ` Jonathan Morton
2019-03-16 7:38 ` Sebastian Moeller
0 siblings, 2 replies; 105+ messages in thread
From: David P. Reed @ 2019-03-15 23:43 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Mikael Abrahamsson, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 3521 bytes --]
My point is that the argument for doing such balancing is that somehow ISPs at the entry points (representing somehow the goals of source and destination of each flow) will classify their flows correctly based on some criterion, and not select the option that allows them to "beat" all the others, which then causes them be "game theoreitically" incented to screw up the labeling.
The business argument that the users at both ends will choose the rignt labels is that they are responsive to price signals in a very sensitive way that will get them to mark things correctly. (that includes, by the way, the Internet Access Providers, if they take over the labeling job and force their choice on their users, because they become the "endpoint")
So if pricing mechanisms don't work to incent labeling correctly, it does not matter that there exists an optimum that can be achieved by an Oracle who fully decides the settings on all packets of all protocols ever to be invented.
And that's why I brought up the issue of pricing and economics, which sadly really affect any kind of queue management.
That's why pricing becomes a practical issue, and issues of "utility" to the users become important.
Now the other thing that is crucial is that the optimal state almost all of the time of every link in the network is that a utilization far from max capacity. The reason for this is the fact that the Internet (like almost all networks) is bursty and fractal. The law of large numbers doesn't smooth traffic volume over any timescale (that's the sense of fractalness here). There is no statistical smoothing of load - there are rare high peaks on some links but most links are underutilized, *if you want all the protocols currently used in the Internet to make users happy with minimal time-to-task-completion* (response time at the scale that matters for the particular user's needs at that moment).
So if most links are uncongested most of the time (and they should be if the folks maintaining the subnets are all doing their job by growing links that are congested to handle more traffic), then congestion management is only a peak load problem, and only affects things a small percentage of the time.
This is very, very different from the typical "benchmark" case, which focuses only on dealing with peak loads, which are transient in the real world. Most "benchmarks" make the strange and unrealistic assumption that overload is steady state, and that users themselves don't give up and stop using an overloaded system.
-----Original Message-----
From: "Jonathan Morton" <chromatix99@gmail.com>
Sent: Friday, March 15, 2019 4:13pm
To: "David P. Reed" <dpreed@deepplum.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
> On 15 Mar, 2019, at 9:44 pm, David P. Reed <dpreed@deepplum.com> wrote:
>
> pricing (even dynamic pricing) of different qualities of service is unstable
An interesting result, but I should note that the four-way optimisation system I described doesn't rely on pricing, only a sufficiently correct implementation of those optimisations at enough bottlenecks to make it worthwhile for applications to mark their traffic appropriately. The technology exists to do so, but is not standardised in a way that makes it usable.
- Jonathan Morton
[-- Attachment #2: Type: text/html, Size: 5599 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 20:31 ` Dave Taht
@ 2019-03-15 23:45 ` David P. Reed
2019-03-16 9:42 ` Michael Welzl
0 siblings, 1 reply; 105+ messages in thread
From: David P. Reed @ 2019-03-15 23:45 UTC (permalink / raw)
To: Dave Taht; +Cc: Jonathan Foulkes, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 3854 bytes --]
How many applications used by normal users have "admin" privileges? The Browser? Email? FTP?
-----Original Message-----
From: "Dave Taht" <dave.taht@gmail.com>
Sent: Friday, March 15, 2019 4:31pm
To: "Jonathan Foulkes" <jf@jonathanfoulkes.com>
Cc: ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
On Fri, Mar 15, 2019 at 1:28 PM Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>
> All this discussion of DSCP marking brings to mind what happened on the Windows platform, where the OS had to suppress ALL DSCP marks, as app authors were trying to game the system.
> And even if not trying to ‘game’ it, they have non-obvious reasons why they don’t mark traffic how one would expect. Example:
>
> I know an engineer who works at a cloud-storage solution company, and I asked why a long-standing customer request for DSCP marking (as bulk) was not implemented. His answer was they’d never do that, as that would impact benchmarks against their competitors for which service syncs faster. <sigh>
>
> Which brings me to a question: Is anyone aware of an easy to use Windows app that will allow the user to select an application and tell the OS to mark the traffic (all or by port) with a user selected DSCP level?
> There are many guides on using regedit and other error-prone (and geek-only) means of doing this, but is there a simple Windows 10 home app?
When I last tried it (years ago), in order to set the tos bits, an
application merely had to have admin privs.
> Now that Cake is out there with simple DiffServ3 support, it would be nice to lower the priority of cloud-storage services and other bulk traffic by correctly marking it at the origin.
>
> Cheers,
>
> Jonathan Foulkes
>
>
> > On Mar 15, 2019, at 3:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >>
> >> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
> >>
> >> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
> >
> > This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
> >
> > 0: Maximum Throughput (aka Best Effort)
> > 1: Minimum Cost (aka Least Effort)
> > 2: Minimum Latency (aka Maximum Responsiveness)
> > 3: Minimum Loss (aka Maximum Reliability)
> >
> > It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
> >
> > The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
> >
> > But that's a separate topic from ECN per se.
> >
> > - Jonathan Morton
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/ecn-sane
[-- Attachment #2: Type: text/html, Size: 4989 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 23:43 ` David P. Reed
@ 2019-03-16 1:26 ` Jonathan Morton
2019-03-16 7:38 ` Sebastian Moeller
1 sibling, 0 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-16 1:26 UTC (permalink / raw)
To: David P. Reed; +Cc: Mikael Abrahamsson, ecn-sane, bloat
> On 16 Mar, 2019, at 1:43 am, David P. Reed <dpreed@deepplum.com> wrote:
>
> Now the other thing that is crucial is that the optimal state almost all of the time of every link in the network is that a utilization far from max capacity. The reason for this is the fact that the Internet (like almost all networks) is bursty and fractal.
That can be said about some types of links but not others.
Last-mile links in particular are often saturated by their users' individual traffic for minutes or even hours at a time, especially the slower link technologies such as ADSL and 4G. The user wants their hundred-gigabyte game update installed as soon as possible, even if they only have 10Mbps to suck it through, and they still want to use their connection for other things while they wait. And this is not unreasonable; I do it regularly.
At peak times, ISP backhaul capacity can often be enough of a bottleneck for users to notice the congestion and induced latency; it is far from the case that all ISPs worldwide massively over-provision their networks to avoid routine congestion, even in modern technologically advanced nations. There are remote islands where hundreds or thousands of users must share a single satellite or microwave uplink. Cell towers are also a shared medium with decidedly finite capacity.
Only core networks, and the backhaul networks of certain particularly conscientious ISPs, can typically be described as congestion-free. And that is why we discuss AQM and ECN in such detail in the first place; without congestion, they wouldn't be required.
The extent to which traffic classification is needed on particular types of link can be debated. It could fairly be argued that we've done remarkably well without the benefit of a functioning Diffserv ecosystem, so there is no particular urgency to create one. At the same time, it's worth discussing *why* Diffserv fails to do its intended job, and how a better system *could* be designed, because that may reveal lessons for the future.
I will say this: there are times, even with the benefit of everything Cake does for me, when I would prefer that BitTorrent traffic would automatically defer to other stuff (as it was supposedly designed to). Several game updaters, including Wargaming.net's, use BitTorrent under the skin - opening and using several hundred flows in parallel, and thereby swamping any other traffic going to the same host. It would be very straightforward for them to mark all that traffic as Minimum Cost, while their games themselves use Minimum Latency for battle traffic.
Minimum Cost is a natural choice for any transport using LEDBAT, or with similarly altruistic philosophy.
Minimum Latency is a natural choice for any application requiring realtime response - games, VoIP, remote shells.
Minimum Loss is a natural choice for applications involved in network control, where dropped packets could have a much greater than normal impact on overall network performance.
Maximum Throughput is a natural choice for general-purpose applications not fitting any of the above.
Pricing is not required. Making the wrong choice will simply make your application perform badly on a loaded network, as the bottleneck link optimises for the thing you told it to optimise for, rather than for what you actually want. That's all that's really needed.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 17:01 ` [Bloat] [Ecn-sane] " David P. Reed
2019-03-15 17:45 ` Sebastian Moeller
2019-03-15 18:36 ` Mikael Abrahamsson
@ 2019-03-16 4:04 ` Jonathan Morton
2019-03-16 4:51 ` Dave Taht
2 siblings, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-16 4:04 UTC (permalink / raw)
To: David P. Reed; +Cc: Sebastian Moeller, ecn-sane, bloat
> On 15 Mar, 2019, at 7:01 pm, David P. Reed <dpreed@deepplum.com> wrote:
>
> > Next up for sce was going to be to find if dctcp could be modified to
> > use it. Also, bittorrent.
>
> YES! I endorse this message.
Actually, today I was still figuring out how to fit it to CUBIC. I *think* I've succeeded…
https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/ELR%20Proposal%203%20(TCP).txt
I recall DCTCP as being singularly built around a distinct setpoint from what either Classic ECN or SCE expects. So I think I might look at LEDBAT first.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 4:04 ` Jonathan Morton
@ 2019-03-16 4:51 ` Dave Taht
0 siblings, 0 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-16 4:51 UTC (permalink / raw)
To: Jonathan Morton; +Cc: David P. Reed, ecn-sane, bloat
On Fri, Mar 15, 2019 at 9:04 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 15 Mar, 2019, at 7:01 pm, David P. Reed <dpreed@deepplum.com> wrote:
> >
> > > Next up for sce was going to be to find if dctcp could be modified to
> > > use it. Also, bittorrent.
> >
> > YES! I endorse this message.
>
> Actually, today I was still figuring out how to fit it to CUBIC. I *think* I've succeeded…
>
> https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/ELR%20Proposal%203%20(TCP).txt
>
> I recall DCTCP as being singularly built around a distinct setpoint from what either Classic ECN or SCE expects. So I think I might look at LEDBAT first.
LEDBAT as defined by the ietf is a waste of time, as is it did not deploy.
In terms of dealing with bittorrent, I recommend you look over the UDP
implementation in libutp, and in particular the embedded
implementation in "transmission", which is easier to play with first.
Along the way, getting setsockopt to set the diffserv bits right
finally for libutp would be nice.
> - Jonathan Morton
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 23:43 ` David P. Reed
2019-03-16 1:26 ` Jonathan Morton
@ 2019-03-16 7:38 ` Sebastian Moeller
2019-03-16 18:56 ` Michael Richardson
1 sibling, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-16 7:38 UTC (permalink / raw)
To: bloat, David P. Reed, Jonathan Morton; +Cc: bloat, ecn-sane
Hi Dave,
On March 16, 2019 12:43:58 AM GMT+01:00, "David P. Reed" <dpreed@deepplum.com> wrote:
>
>My point is that the argument for doing such balancing is that somehow
>ISPs at the entry points (representing somehow the goals of source and
>destination of each flow) will classify their flows correctly based on
>some criterion, and not select the option that allows them to "beat"
>all the others, which then causes them be "game theoreitically"
>incented to screw up the labeling.
One more argument for splitting the 6 dscp bits, one half for the ISP one half for end to end signalling...
I do neither expect ISPs to honor the intent bits (with the possible exception of the LE patter, there e2e and ISP goals seem aligned) and I also do not see any approach gain traction that completely takes the dscp bits away from the intermediate transports, whether one likes it or not, these effectively are under transport control and are actually used, so hoping the current owner giving all of them back seems overly optimistic. Hence the 50/50 split.
>
>The business argument that the users at both ends will choose the rignt
>labels is that they are responsive to price signals in a very sensitive
>way that will get them to mark things correctly.
Right or wrong, or how to interpret the different pattern is a question that only becomes relevant once the patterns are signalled robustly end to end...
(that includes, by the
>way, the Internet Access Providers, if they take over the labeling job
>and force their choice on their users, because they become the
>"endpoint")
Not with the split proposal, and yes end users depend on their ISP doing reasonable traffic management, but that seems orthogonal to dscp bit patterns to me.
>
>So if pricing mechanisms don't work to incent labeling correctly, it
>does not matter that there exists an optimum that can be achieved by an
>Oracle who fully decides the settings on all packets of all protocols
>ever to be invented.
>
>And that's why I brought up the issue of pricing and economics, which
>sadly really affect any kind of queue management.
Sure in the context of hoping the ISPs and the wider internet respecting endpoint set dscps that seems all applicable.
>
>That's why pricing becomes a practical issue, and issues of "utility"
>to the users become important.
>
>Now the other thing that is crucial is that the optimal state almost
>all of the time of every link in the network is that a utilization far
>from max capacity. The reason for this is the fact that the Internet
>(like almost all networks) is bursty and fractal. The law of large
>numbers doesn't smooth traffic volume over any timescale (that's the
>sense of fractalness here). There is no statistical smoothing of load -
>there are rare high peaks on some links but most links are
>underutilized, *if you want all the protocols currently used in the
>Internet to make users happy with minimal time-to-task-completion*
>(response time at the scale that matters for the particular user's
>needs at that moment).
>
>So if most links are uncongested most of the time (and they should be
>if the folks maintaining the subnets are all doing their job by growing
>links that are congested to handle more traffic), then congestion
>management is only a peak load problem, and only affects things a small
>percentage of the time.
I concur with Jonathan, access links often run much closer to their limit than core networks, and the whole bufferbloat project demonstrated that a capable AQM system with mild tiering can make a saturated link still acceptable to use even for low latency applications...
But for ingress shaping it would be really great to have some trustworthy way of deducing the sender's intent, and dscps seem like a natural fit.
Best Regards
Sebastian
>
>This is very, very different from the typical "benchmark" case, which
>focuses only on dealing with peak loads, which are transient in the
>real world. Most "benchmarks" make the strange and unrealistic
>assumption that overload is steady state, and that users themselves
>don't give up and stop using an overloaded system.
>
>
>-----Original Message-----
>From: "Jonathan Morton" <chromatix99@gmail.com>
>Sent: Friday, March 15, 2019 4:13pm
>To: "David P. Reed" <dpreed@deepplum.com>
>Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>,
>ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
>Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation
>and experimentation of TCP Prague/L4S hackaton at IETF104
>
>
>
>> On 15 Mar, 2019, at 9:44 pm, David P. Reed <dpreed@deepplum.com>
>wrote:
>>
>> pricing (even dynamic pricing) of different qualities of service is
>unstable
>
>An interesting result, but I should note that the four-way optimisation
>system I described doesn't rely on pricing, only a sufficiently correct
>implementation of those optimisations at enough bottlenecks to make it
>worthwhile for applications to mark their traffic appropriately. The
>technology exists to do so, but is not standardised in a way that makes
>it usable.
>
> - Jonathan Morton
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 23:45 ` David P. Reed
@ 2019-03-16 9:42 ` Michael Welzl
2019-03-16 10:08 ` Sebastian Moeller
0 siblings, 1 reply; 105+ messages in thread
From: Michael Welzl @ 2019-03-16 9:42 UTC (permalink / raw)
To: David P. Reed; +Cc: Dave Taht, bloat, ecn-sane
[-- Attachment #1: Type: text/plain, Size: 4495 bytes --]
Good question! …. on Windows in particular, I’d really like to know this too.
The WebRTC Javascript API allows one to influence the DSCP, i.e. browsers normally can do that. Whether that’s true for all OSes, I don’t know.
Cheers,
Michael
> On Mar 16, 2019, at 12:45 AM, David P. Reed <dpreed@deepplum.com> wrote:
>
> How many applications used by normal users have "admin" privileges? The Browser? Email? FTP?
>
>
> -----Original Message-----
> From: "Dave Taht" <dave.taht@gmail.com>
> Sent: Friday, March 15, 2019 4:31pm
> To: "Jonathan Foulkes" <jf@jonathanfoulkes.com>
> Cc: ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> On Fri, Mar 15, 2019 at 1:28 PM Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> >
> > All this discussion of DSCP marking brings to mind what happened on the Windows platform, where the OS had to suppress ALL DSCP marks, as app authors were trying to game the system.
> > And even if not trying to ‘game’ it, they have non-obvious reasons why they don’t mark traffic how one would expect. Example:
> >
> > I know an engineer who works at a cloud-storage solution company, and I asked why a long-standing customer request for DSCP marking (as bulk) was not implemented. His answer was they’d never do that, as that would impact benchmarks against their competitors for which service syncs faster. <sigh>
> >
> > Which brings me to a question: Is anyone aware of an easy to use Windows app that will allow the user to select an application and tell the OS to mark the traffic (all or by port) with a user selected DSCP level?
> > There are many guides on using regedit and other error-prone (and geek-only) means of doing this, but is there a simple Windows 10 home app?
>
> When I last tried it (years ago), in order to set the tos bits, an
> application merely had to have admin privs.
>
> > Now that Cake is out there with simple DiffServ3 support, it would be nice to lower the priority of cloud-storage services and other bulk traffic by correctly marking it at the origin.
> >
> > Cheers,
> >
> > Jonathan Foulkes
> >
> >
> > > On Mar 15, 2019, at 3:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> > >
> > >> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> > >>
> > >> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
> > >>
> > >> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
> > >
> > > This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
> > >
> > > 0: Maximum Throughput (aka Best Effort)
> > > 1: Minimum Cost (aka Least Effort)
> > > 2: Minimum Latency (aka Maximum Responsiveness)
> > > 3: Minimum Loss (aka Maximum Reliability)
> > >
> > > It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
> > >
> > > The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
> > >
> > > But that's a separate topic from ECN per se.
> > >
> > > - Jonathan Morton
> > >
> > > _______________________________________________
> > > Bloat mailing list
> > > Bloat@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/bloat
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
[-- Attachment #2: Type: text/html, Size: 7448 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 9:42 ` Michael Welzl
@ 2019-03-16 10:08 ` Sebastian Moeller
2019-03-16 10:23 ` Nils Andreas Svee
0 siblings, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-16 10:08 UTC (permalink / raw)
To: Michael Welzl; +Cc: David P. Reed, ecn-sane, bloat
> On Mar 16, 2019, at 10:42, Michael Welzl <michawe@ifi.uio.no> wrote:
>
> Good question! …. on Windows in particular, I’d really like to know this too.
Well, as far as I can tell it is the group policy editor that is the tool to assign DSCPs to applications, IMHO that is exactly the right place, somewhere where the administrator/enduser can set her desired policy (personally I am fine with applications also using sensible defaults, as long as the user can override them all is well). The catch seems to be that group policies require a domain controller and are hence not available on stand-alond windows home installations. Anybody with deep contacts to microsoft here, that could try to get an sub-official standpoint from MS on the issue of opening the group policy editor up for everybody (at least the dscp marking part)?
Best Regards
Sebastian
>
> The WebRTC Javascript API allows one to influence the DSCP, i.e. browsers normally can do that. Whether that’s true for all OSes, I don’t know.
>
> Cheers,
> Michael
>
>
>
>> On Mar 16, 2019, at 12:45 AM, David P. Reed <dpreed@deepplum.com> wrote:
>>
>> How many applications used by normal users have "admin" privileges? The Browser? Email? FTP?
>>
>>
>> -----Original Message-----
>> From: "Dave Taht" <dave.taht@gmail.com>
>> Sent: Friday, March 15, 2019 4:31pm
>> To: "Jonathan Foulkes" <jf@jonathanfoulkes.com>
>> Cc: ecn-sane@lists.bufferbloat.net, "bloat" <bloat@lists.bufferbloat.net>
>> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>>
>> On Fri, Mar 15, 2019 at 1:28 PM Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>> >
>> > All this discussion of DSCP marking brings to mind what happened on the Windows platform, where the OS had to suppress ALL DSCP marks, as app authors were trying to game the system.
>> > And even if not trying to ‘game’ it, they have non-obvious reasons why they don’t mark traffic how one would expect. Example:
>> >
>> > I know an engineer who works at a cloud-storage solution company, and I asked why a long-standing customer request for DSCP marking (as bulk) was not implemented. His answer was they’d never do that, as that would impact benchmarks against their competitors for which service syncs faster. <sigh>
>> >
>> > Which brings me to a question: Is anyone aware of an easy to use Windows app that will allow the user to select an application and tell the OS to mark the traffic (all or by port) with a user selected DSCP level?
>> > There are many guides on using regedit and other error-prone (and geek-only) means of doing this, but is there a simple Windows 10 home app?
>>
>> When I last tried it (years ago), in order to set the tos bits, an
>> application merely had to have admin privs.
>>
>> > Now that Cake is out there with simple DiffServ3 support, it would be nice to lower the priority of cloud-storage services and other bulk traffic by correctly marking it at the origin.
>> >
>> > Cheers,
>> >
>> > Jonathan Foulkes
>> >
>> >
>> > > On Mar 15, 2019, at 3:32 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> > >
>> > >> On 15 Mar, 2019, at 8:36 pm, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> > >>
>> > >> Having a "lower-than-best-effort" diffserve codepoint might work, because it means worse treatment, not preferential treatment.
>> > >>
>> > >> The problem with having DSCP CPs that indicate preferential treatment is typically a ddos magnet.
>> > >
>> > > This is true, and also why I feel that just 2 bits should be sufficient for Diffserv (rather than 6). They are sufficient to express four different optimisation targets:
>> > >
>> > > 0: Maximum Throughput (aka Best Effort)
>> > > 1: Minimum Cost (aka Least Effort)
>> > > 2: Minimum Latency (aka Maximum Responsiveness)
>> > > 3: Minimum Loss (aka Maximum Reliability)
>> > >
>> > > It is legitimate for traffic to request any of these four optimisations, with the explicit tradeoff of *not* necessarily getting optimisation in the other three dimensions.
>> > >
>> > > The old TOS spec erred in specifying 4 non-exclusive bits to express this, in addition to 3 bits for a telegram-office style "priority level" (which was very much ripe for abuse if not strictly admission-controlled). TOS was rightly considered a mess, but was replaced with Diffserv which was far too loose a spec to be useful in practice.
>> > >
>> > > But that's a separate topic from ECN per se.
>> > >
>> > > - Jonathan Morton
>> > >
>> > > _______________________________________________
>> > > Bloat mailing list
>> > > Bloat@lists.bufferbloat.net
>> > > https://lists.bufferbloat.net/listinfo/bloat
>> >
>> > _______________________________________________
>> > Bloat mailing list
>> > Bloat@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/bloat
>>
>>
>>
>> --
>>
>> Dave Täht
>> CTO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-831-205-9740
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 10:08 ` Sebastian Moeller
@ 2019-03-16 10:23 ` Nils Andreas Svee
2019-03-16 14:55 ` Jonathan Foulkes
0 siblings, 1 reply; 105+ messages in thread
From: Nils Andreas Svee @ 2019-03-16 10:23 UTC (permalink / raw)
To: Sebastian Moeller, Michael Welzl; +Cc: ecn-sane, David P. Reed, bloat
You can use group policies on standalone clients, however the Local Group Policy Editor is not available on the Windows Home editions (neither is domain membership AFAIK), and so many people resort to applying the changes they require directly to the registry.
There are also tools like Policy Plus that allows you to edit local GPOs regardless of which edition of Windows you are running , but I haven't tried it myself: https://github.com/Fleex255/PolicyPlus
Best Regards
Nils
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 10:23 ` Nils Andreas Svee
@ 2019-03-16 14:55 ` Jonathan Foulkes
0 siblings, 0 replies; 105+ messages in thread
From: Jonathan Foulkes @ 2019-03-16 14:55 UTC (permalink / raw)
To: Nils Andreas Svee; +Cc: Sebastian Moeller, ecn-sane, bloat
Thanks Nils, I’d seen that PolicyPlus tool before, and while a nice stand-alone option for any Windows version, it is still a tool for admins. I think we need something regular technical users can use to do things like ‘Make DropBox traffic land in the bulk tin’.
Also, there is a good thread about other mechanisms to route traffic into the bulk tin using rules in the router here: https://github.com/dtaht/sch_cake/issues/97 Including a documented approach to setting windows DSCP settings via PowerShell.
Cheers,
Jonathan Foulkes
> On Mar 16, 2019, at 6:23 AM, Nils Andreas Svee <me@lochnair.net> wrote:
>
> You can use group policies on standalone clients, however the Local Group Policy Editor is not available on the Windows Home editions (neither is domain membership AFAIK), and so many people resort to applying the changes they require directly to the registry.
>
> There are also tools like Policy Plus that allows you to edit local GPOs regardless of which edition of Windows you are running , but I haven't tried it myself: https://github.com/Fleex255/PolicyPlus
>
> Best Regards
> Nils
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 7:38 ` Sebastian Moeller
@ 2019-03-16 18:56 ` Michael Richardson
0 siblings, 0 replies; 105+ messages in thread
From: Michael Richardson @ 2019-03-16 18:56 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: bloat, David P. Reed, Jonathan Morton, ecn-sane
[-- Attachment #1: Type: text/plain, Size: 728 bytes --]
The DSCP code points were also supposed to be local to the AS.
They are *supposed* to be recoded at the edges.
What is missing the diffedge protocol... MS actually implemented this in Win2K.
We have no way to signal the ToS we desire with the ISP, and so they are
forced to guess, and they do it wrong.
I actually see this as the critical NetNeutrality issue. We *DO*
want multiple classes of services, but we want to be in charge of
what traffic goes into each class, not the ISP.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | IoT architect [
] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-15 18:36 ` Mikael Abrahamsson
2019-03-15 19:23 ` Sebastian Moeller
2019-03-15 19:32 ` Jonathan Morton
@ 2019-03-16 21:38 ` Holland, Jake
2019-03-16 21:57 ` Vint Cerf
` (3 more replies)
2 siblings, 4 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-16 21:38 UTC (permalink / raw)
To: Mikael Abrahamsson, David P. Reed; +Cc: ecn-sane, bloat
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:38 ` Holland, Jake
@ 2019-03-16 21:57 ` Vint Cerf
2019-03-16 22:03 ` Dave Taht
` (2 more replies)
2019-03-16 22:03 ` Jonathan Morton
` (2 subsequent siblings)
3 siblings, 3 replies; 105+ messages in thread
From: Vint Cerf @ 2019-03-16 21:57 UTC (permalink / raw)
To: Holland, Jake; +Cc: Mikael Abrahamsson, David P. Reed, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 5471 bytes --]
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
> L4S has a much better possibility of actually getting deployment into
> the
> wider Internet packet-moving equipment than anything being talked
> about
> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be
> as
> good, but it fits better into actual silicon and it's being proposed
> by
> people who actually have better channels into the people setting hard
> requirements.
>
> I suggest you consider joining them instead of opposing them.
>
>
> Hi Mikael,
>
> I agree it makes sense that fq_anything has issues when you're talking
> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
> makes better sense there.
>
> But fq_x makes great sense and provides real value for the uplink in a
> home, small office, coffee shop, etc. (if you run the final rate limit
> on the home side of the access link.) I'm thinking maybe there's a
> disconnect here driven by the different use cases for where AQMs can go.
>
> The thing is, each of these is the most likely congestion point at
> different times, and it's worthwhile for each of them to be able to
> AQM (and mark packets) under congestion.
>
> One of the several things that bothers me with L4S is that I've seen
> precious little concern over interfering with the ability for another
> different AQM in-path to mark packets, and because it changes the
> semantics of CE, you can't have both working at the same time unless
> they both do L4S.
>
> SCE needs a lot of details filled in, but it's so much cleaner that it
> seems to me there's reasonably obvious answers to all (or almost all) of
> those detail questions, and because the semantics are so much cleaner,
> it's much easier to tell it's non-harmful.
>
> <aside regarding="non-harmful">
> The point you raised in another thread about reordering is mostly
> well-taken, and a good counterpoint to the claim "non-harmful relative
> to L4S".
>
> To me it seems sad and dumb that switches ended up trying to make
> ordering guarantees at cost of switching performance, because if it's
> useful to put ordering in the switch, then it must be equally useful to
> put it in the receiver's NIC or OS.
>
> So why isn't it in all the receivers' NIC or OS (where it would render
> the switch's ordering efforts moot) instead of in all the switches?
>
> I'm guessing the answer is a competition trap for the switch vendors,
> plus "with ordering goes faster than without, when you benchmark the
> switch with typical load and current (non-RACK) receivers".
>
> If that's the case, it seems like the drive for a competitive advantage
> caused deployment of a packet ordering workaround in the wrong network
> location(s), out of a pure misalignment of incentives.
>
> RACK rates to fix that in the end, but a lot of damage is already done,
> and the L4S approach gives switches a flag that can double as proof that
> RACK is there on the receiver, so they can stop trying to order those
> packets.
>
> So point granted, I understand and agree there's a cost to abandoning
> that advantage.
> </aside>
>
> But as you also said so well in another thread, this is important. ("The
> last unicorn", IIRC.) How much does it matter if there's a feature that
> has value today, but only until RACK is widely deployed? If you were
> convinced RACK would roll out everywhere within 3 years and SCE would
> produce better results than L4S over the following 15 years, would that
> change your mind?
>
> It would for me, and that's why I'd like to see SCE explored before
> making a call. I think at its core, it provides the same thing L4S does
> (a high-fidelity explicit congestion signal for the sender), but with
> much cleaner semantics that can be incrementally added to congestion
> controls that people are already using.
>
> Granted, it still remains to be seen whether SCE in practice can match
> the results of L4S, and L4S was here first. But it seems to me L4S comes
> with some problems that have not yet been examined, and that are nicely
> dodged by a SCE-based approach.
>
> If L4S really is as good as they seem to think, I could imagine getting
> behind it, but I don't think that's proven yet. I'm not certain, but
> all the comparative analyses I remember seeing have been from more or
> less the same team, and I'm not convinced they don't have some
> misaligned incentives of their own.
>
> I understand a lot of work has gone into L4S, but this move to jump it
> from interesting experiment to de-facto standard without a more critical
> review that digs deeper into some of the potential deployment problems
> has me concerned.
>
> If it really does turn out to be good enough to be permanent, I'm not
> opposed to it, but I'm just not convinced that it's non-harmful, and my
> default position is that the cleaner solution is going to be better in
> the long run, if they can do the same job.
>
> It's not that I want it to be a fight, but I do want to end up with the
> best solution we can get. We only have the one internet.
>
> Just my 2c.
>
> -Jake
>
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
[-- Attachment #2: Type: text/html, Size: 6717 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:57 ` Vint Cerf
@ 2019-03-16 22:03 ` Dave Taht
2019-03-16 22:05 ` Holland, Jake
2019-03-17 18:07 ` David P. Reed
2 siblings, 0 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-16 22:03 UTC (permalink / raw)
To: Vint Cerf; +Cc: Holland, Jake, bloat, ecn-sane
Dear Vint:
BBR, along with all "non ect_1 sending L4S compatable" transports,
gets relegated to the dualpi "Classic" queue.
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
On Sat, Mar 16, 2019 at 2:57 PM Vint Cerf <vint@google.com> wrote:
>
> where does BBR fit into all this?
>
> v
>
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
>>
>> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
>> L4S has a much better possibility of actually getting deployment into the
>> wider Internet packet-moving equipment than anything being talked about
>> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
>> good, but it fits better into actual silicon and it's being proposed by
>> people who actually have better channels into the people setting hard
>> requirements.
>>
>> I suggest you consider joining them instead of opposing them.
>>
>>
>> Hi Mikael,
>>
>> I agree it makes sense that fq_anything has issues when you're talking
>> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
>> makes better sense there.
>>
>> But fq_x makes great sense and provides real value for the uplink in a
>> home, small office, coffee shop, etc. (if you run the final rate limit
>> on the home side of the access link.) I'm thinking maybe there's a
>> disconnect here driven by the different use cases for where AQMs can go.
>>
>> The thing is, each of these is the most likely congestion point at
>> different times, and it's worthwhile for each of them to be able to
>> AQM (and mark packets) under congestion.
>>
>> One of the several things that bothers me with L4S is that I've seen
>> precious little concern over interfering with the ability for another
>> different AQM in-path to mark packets, and because it changes the
>> semantics of CE, you can't have both working at the same time unless
>> they both do L4S.
>>
>> SCE needs a lot of details filled in, but it's so much cleaner that it
>> seems to me there's reasonably obvious answers to all (or almost all) of
>> those detail questions, and because the semantics are so much cleaner,
>> it's much easier to tell it's non-harmful.
>>
>> <aside regarding="non-harmful">
>> The point you raised in another thread about reordering is mostly
>> well-taken, and a good counterpoint to the claim "non-harmful relative
>> to L4S".
>>
>> To me it seems sad and dumb that switches ended up trying to make
>> ordering guarantees at cost of switching performance, because if it's
>> useful to put ordering in the switch, then it must be equally useful to
>> put it in the receiver's NIC or OS.
>>
>> So why isn't it in all the receivers' NIC or OS (where it would render
>> the switch's ordering efforts moot) instead of in all the switches?
>>
>> I'm guessing the answer is a competition trap for the switch vendors,
>> plus "with ordering goes faster than without, when you benchmark the
>> switch with typical load and current (non-RACK) receivers".
>>
>> If that's the case, it seems like the drive for a competitive advantage
>> caused deployment of a packet ordering workaround in the wrong network
>> location(s), out of a pure misalignment of incentives.
>>
>> RACK rates to fix that in the end, but a lot of damage is already done,
>> and the L4S approach gives switches a flag that can double as proof that
>> RACK is there on the receiver, so they can stop trying to order those
>> packets.
>>
>> So point granted, I understand and agree there's a cost to abandoning
>> that advantage.
>> </aside>
>>
>> But as you also said so well in another thread, this is important. ("The
>> last unicorn", IIRC.) How much does it matter if there's a feature that
>> has value today, but only until RACK is widely deployed? If you were
>> convinced RACK would roll out everywhere within 3 years and SCE would
>> produce better results than L4S over the following 15 years, would that
>> change your mind?
>>
>> It would for me, and that's why I'd like to see SCE explored before
>> making a call. I think at its core, it provides the same thing L4S does
>> (a high-fidelity explicit congestion signal for the sender), but with
>> much cleaner semantics that can be incrementally added to congestion
>> controls that people are already using.
>>
>> Granted, it still remains to be seen whether SCE in practice can match
>> the results of L4S, and L4S was here first. But it seems to me L4S comes
>> with some problems that have not yet been examined, and that are nicely
>> dodged by a SCE-based approach.
>>
>> If L4S really is as good as they seem to think, I could imagine getting
>> behind it, but I don't think that's proven yet. I'm not certain, but
>> all the comparative analyses I remember seeing have been from more or
>> less the same team, and I'm not convinced they don't have some
>> misaligned incentives of their own.
>>
>> I understand a lot of work has gone into L4S, but this move to jump it
>> from interesting experiment to de-facto standard without a more critical
>> review that digs deeper into some of the potential deployment problems
>> has me concerned.
>>
>> If it really does turn out to be good enough to be permanent, I'm not
>> opposed to it, but I'm just not convinced that it's non-harmful, and my
>> default position is that the cleaner solution is going to be better in
>> the long run, if they can do the same job.
>>
>> It's not that I want it to be a fight, but I do want to end up with the
>> best solution we can get. We only have the one internet.
>>
>> Just my 2c.
>>
>> -Jake
>>
>>
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
>
> --
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:38 ` Holland, Jake
2019-03-16 21:57 ` Vint Cerf
@ 2019-03-16 22:03 ` Jonathan Morton
2019-03-16 22:09 ` Sebastian Moeller
2019-03-17 14:06 ` Mikael Abrahamsson
3 siblings, 0 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-16 22:03 UTC (permalink / raw)
To: Holland, Jake; +Cc: Mikael Abrahamsson, David P. Reed, ecn-sane, bloat
> On 16 Mar, 2019, at 11:38 pm, Holland, Jake <jholland@akamai.com> wrote:
>
> SCE needs a lot of details filled in, but it's so much cleaner that it
> seems to me there's reasonably obvious answers to all (or almost all) of
> those detail questions, and because the semantics are so much cleaner,
> it's much easier to tell it's non-harmful.
And we're actively working on filling in those details. They haven't yet made it as far as an I-D, partly because we'd like to do at least some preliminary testing first.
> where does BBR fit into all this?
The present version of BBR, which I've actually seen, ignores ECN completely. I'm told that the new version which I haven't yet seen, does *something* with ECN, but I'm unclear on what. I think it'll be up to the BBR developers to decide how to incorporate SCE information, using conventional TCPs as a guide as to what is reasonable - or to ignore it, which is also a valid design choice.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:57 ` Vint Cerf
2019-03-16 22:03 ` Dave Taht
@ 2019-03-16 22:05 ` Holland, Jake
2019-03-17 18:07 ` David P. Reed
2 siblings, 0 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-16 22:05 UTC (permalink / raw)
To: Vint Cerf; +Cc: Mikael Abrahamsson, David P. Reed, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 7282 bytes --]
Yeah, great question.
I don't want to answer for the L4S guys, I don't have a good feel for
what they might think. But it does concern me that there seems to be at
least one tuning parameter that was picked for Reno, which I think I
mentioned on the tsvwg list:
https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08#section-2.1
For SCE, I would assume they'll want to make use of it at some point,
and so they'll have to write a draft for how BBR will handle it.
I think there's an open question of what exactly the rate of SCE
markings would look like for a SCE-capable AQM, and presumably this also
needs to be nailed down before it can be useful. My initial instinct is
a probabilistic SCE setting based on current queue length, either when
forwarded or when enqueued, but I think this will take some more
thought, and I'm not sure that's best.
But whatever the most informative schedule we can figure out is, if that
info can get back to sender, it can essentially do whatever it thinks is
right, according to the cc it’s running, is my understanding.
-Jake
From: Vint Cerf <vint@google.com>
Date: 2019-03-16 at 14:57
To: "Holland, Jake" <jholland@akamai.com>
Cc: Mikael Abrahamsson <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=Z6SbAUystZUjAZ76eHgzuX1g5MhoKi4Ich8EPHag2YY&s=wSu1I5Whay2ozL9k8eMyqhqN-SQVdMnbPzCRx6tyEZ8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
[-- Attachment #2: Type: text/html, Size: 12695 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:38 ` Holland, Jake
2019-03-16 21:57 ` Vint Cerf
2019-03-16 22:03 ` Jonathan Morton
@ 2019-03-16 22:09 ` Sebastian Moeller
2019-03-17 14:06 ` Mikael Abrahamsson
3 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-16 22:09 UTC (permalink / raw)
To: Holland, Jake; +Cc: Mikael Abrahamsson, David P. Reed, ecn-sane, bloat
> On Mar 16, 2019, at 22:38, Holland, Jake <jholland@akamai.com> wrote:
>
> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
> L4S has a much better possibility of actually getting deployment into the
> wider Internet packet-moving equipment than anything being talked about
> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
> good, but it fits better into actual silicon and it's being proposed by
> people who actually have better channels into the people setting hard
> requirements.
>
> I suggest you consider joining them instead of opposing them.
>
>
> Hi Mikael,
>
> I agree it makes sense that fq_anything has issues when you're talking
> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
> makes better sense there.
Except PIE is not mandatory there, DOCSIS3.1 made PIE mandatory on the CPE or customer modems, CMTS AQM was I believe recommended.
>
> But fq_x makes great sense and provides real value for the uplink in a
> home, small office, coffee shop, etc. (if you run the final rate limit
> on the home side of the access link.) I'm thinking maybe there's a
> disconnect here driven by the different use cases for where AQMs can go.
>
> The thing is, each of these is the most likely congestion point at
> different times, and it's worthwhile for each of them to be able to
> AQM (and mark packets) under congestion.
>
> One of the several things that bothers me with L4S is that I've seen
> precious little concern over interfering with the ability for another
> different AQM in-path to mark packets, and because it changes the
> semantics of CE, you can't have both working at the same time unless
> they both do L4S.
The relevant section from https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06:
"A.1.4. Fall back to Reno-friendly congestion control on classic ECN
bottlenecks
Description: A scalable congestion control needs to react to ECN
marking from a non-L4S but ECN-capable bottleneck in a way that will
coexist with a TCP Reno congestion control [RFC5681].
Motivation: Similarly to the requirement in Appendix A.1.3, this
requirement is a safety condition to ensure a scalable congestion
control behaves properly when it builds a queue at a network
bottleneck that has not been upgraded to support L4S. On detecting
classic ECN marking (see below), a scalable congestion control will
need to fall back to classic congestion control behaviour. If it
does not comply with this requirement it could starve classic
traffic.
It would take time for endpoints to distinguish classic and L4S ECN
marking. An increase in queuing delay or in delay variation would be
a tell-tale sign, but it is not yet clear where a line would be drawn
between the two behaviours. It might be possible to cache what was
learned about the path to help subsequent attempts to detect the type
of marking."
In short L4S has not seem to have solved this problem yet except for identifying it.
IMHO this is a clear reason not to to re-use ECT(1) outside of ECN signaling.
>
> SCE needs a lot of details filled in, but it's so much cleaner that it
> seems to me there's reasonably obvious answers to all (or almost all) of
> those detail questions, and because the semantics are so much cleaner,
> it's much easier to tell it's non-harmful.
IMHO the beauty of the simple SCE proposal is that it simply supplies information a rational flow could/should react on purely by self interest, but ignoring it should do no harm, assuming the assumption holds that ECT(1) safely traverses the internet.
>
> <aside regarding="non-harmful">
> The point you raised in another thread about reordering is mostly
> well-taken, and a good counterpoint to the claim "non-harmful relative
> to L4S".
Would this not be better handled by a dedicated signal instead of assuming all L4S traffic is re-ordering tolerant (which as seen from my vantage point runs counter L4S goal of ultra-low latency).
>
> To me it seems sad and dumb that switches ended up trying to make
> ordering guarantees at cost of switching performance, because if it's
> useful to put ordering in the switch, then it must be equally useful to
> put it in the receiver's NIC or OS.
The issue I see, is that re-ordering with fast ARQ cycles on a fast link will be faster than pushing the un-ordered packets over the bottleneck access link, as in the case of data stretching over multiple packets the user might need them all before the data can be actually used.
>
> So why isn't it in all the receivers' NIC or OS (where it would render
> the switch's ordering efforts moot) instead of in all the switches?
>
> I'm guessing the answer is a competition trap for the switch vendors,
> plus "with ordering goes faster than without, when you benchmark the
> switch with typical load and current (non-RACK) receivers".
>
> If that's the case, it seems like the drive for a competitive advantage
> caused deployment of a packet ordering workaround in the wrong network
> location(s), out of a pure misalignment of incentives.
>
> RACK rates to fix that in the end, but a lot of damage is already done,
> and the L4S approach gives switches a flag that can double as proof that
> RACK is there on the receiver, so they can stop trying to order those
> packets.
>
> So point granted, I understand and agree there's a cost to abandoning
> that advantage.
> </aside>
>
> But as you also said so well in another thread, this is important. ("The
> last unicorn", IIRC.) How much does it matter if there's a feature that
> has value today, but only until RACK is widely deployed? If you were
> convinced RACK would roll out everywhere within 3 years and SCE would
> produce better results than L4S over the following 15 years, would that
> change your mind?
>
> It would for me, and that's why I'd like to see SCE explored before
> making a call. I think at its core, it provides the same thing L4S does
> (a high-fidelity explicit congestion signal for the sender), but with
> much cleaner semantics that can be incrementally added to congestion
> controls that people are already using.
>
> Granted, it still remains to be seen whether SCE in practice can match
> the results of L4S, and L4S was here first. But it seems to me L4S comes
> with some problems that have not yet been examined, and that are nicely
> dodged by a SCE-based approach.
>
> If L4S really is as good as they seem to think, I could imagine getting
> behind it, but I don't think that's proven yet. I'm not certain, but
> all the comparative analyses I remember seeing have been from more or
> less the same team, and I'm not convinced they don't have some
> misaligned incentives of their own.
>
> I understand a lot of work has gone into L4S, but this move to jump it
> from interesting experiment to de-facto standard without a more critical
> review that digs deeper into some of the potential deployment problems
> has me concerned.
>
> If it really does turn out to be good enough to be permanent, I'm not
> opposed to it, but I'm just not convinced that it's non-harmful, and my
> default position is that the cleaner solution is going to be better in
> the long run, if they can do the same job.
>
> It's not that I want it to be a fight, but I do want to end up with the
> best solution we can get. We only have the one internet.
>
> Just my 2c.
>
> -Jake
>
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:38 ` Holland, Jake
` (2 preceding siblings ...)
2019-03-16 22:09 ` Sebastian Moeller
@ 2019-03-17 14:06 ` Mikael Abrahamsson
2019-03-17 17:37 ` Loganaden Velvindron
2019-03-17 20:50 ` Luca Muscariello
3 siblings, 2 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-17 14:06 UTC (permalink / raw)
To: Holland, Jake; +Cc: David P. Reed, ecn-sane, bloat
On Sat, 16 Mar 2019, Holland, Jake wrote:
> Granted, it still remains to be seen whether SCE in practice can match
> the results of L4S, and L4S was here first. But it seems to me L4S comes
> with some problems that have not yet been examined, and that are nicely
> dodged by a SCE-based approach.
I'm actually not that interested in an academic competition about what
solution gives the ultimate "best" outcome in simulation or in a lab.
I am interested in good enough solutions that are actually deployable and
will get deployed, and doesn't have any pathological behaviour when it
comes to legacy traffic.
Right now the Internet is full of deep FIFOs and they're not going away,
and they're not getting FQ_CODEL or CAKE.
CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
congestion points we have in real life. These devices would have a much
easier time getting PIE or even RED, if it was just implemented.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 14:06 ` Mikael Abrahamsson
@ 2019-03-17 17:37 ` Loganaden Velvindron
2019-03-17 17:40 ` Toke Høiland-Jørgensen
` (2 more replies)
2019-03-17 20:50 ` Luca Muscariello
1 sibling, 3 replies; 105+ messages in thread
From: Loganaden Velvindron @ 2019-03-17 17:37 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Holland, Jake, ecn-sane, bloat
On Sun, Mar 17, 2019 at 6:06 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Sat, 16 Mar 2019, Holland, Jake wrote:
>
> > Granted, it still remains to be seen whether SCE in practice can match
> > the results of L4S, and L4S was here first. But it seems to me L4S comes
> > with some problems that have not yet been examined, and that are nicely
> > dodged by a SCE-based approach.
>
> I'm actually not that interested in an academic competition about what
> solution gives the ultimate "best" outcome in simulation or in a lab.
>
> I am interested in good enough solutions that are actually deployable and
> will get deployed, and doesn't have any pathological behaviour when it
> comes to legacy traffic.
>
> Right now the Internet is full of deep FIFOs and they're not going away,
> and they're not getting FQ_CODEL or CAKE.
>
> CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
> congestion points we have in real life. These devices would have a much
> easier time getting PIE or even RED, if it was just implemented.
>
is there an open source implementation of PIE which is close to what
is used by the DOCSIS modems ?
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 17:37 ` Loganaden Velvindron
@ 2019-03-17 17:40 ` Toke Høiland-Jørgensen
2019-03-17 17:44 ` Mikael Abrahamsson
2019-03-17 19:38 ` Rodney W. Grimes
2 siblings, 0 replies; 105+ messages in thread
From: Toke Høiland-Jørgensen @ 2019-03-17 17:40 UTC (permalink / raw)
To: ecn-sane, Loganaden Velvindron, Mikael Abrahamsson; +Cc: ecn-sane, bloat
On 17 March 2019 18:37:27 CET, Loganaden Velvindron <loganaden@gmail.com> wrote:
>On Sun, Mar 17, 2019 at 6:06 PM Mikael Abrahamsson <swmike@swm.pp.se>
>wrote:
>>
>> On Sat, 16 Mar 2019, Holland, Jake wrote:
>>
>> > Granted, it still remains to be seen whether SCE in practice can
>match
>> > the results of L4S, and L4S was here first. But it seems to me L4S
>comes
>> > with some problems that have not yet been examined, and that are
>nicely
>> > dodged by a SCE-based approach.
>>
>> I'm actually not that interested in an academic competition about
>what
>> solution gives the ultimate "best" outcome in simulation or in a lab.
>>
>> I am interested in good enough solutions that are actually deployable
>and
>> will get deployed, and doesn't have any pathological behaviour when
>it
>> comes to legacy traffic.
>>
>> Right now the Internet is full of deep FIFOs and they're not going
>away,
>> and they're not getting FQ_CODEL or CAKE.
>>
>> CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
>> congestion points we have in real life. These devices would have a
>much
>> easier time getting PIE or even RED, if it was just implemented.
>>
>
>is there an open source implementation of PIE which is close to what
>is used by the DOCSIS modems ?
Yup. sch_pie in the Linux kernel. I believe Dave originally helped the Cisco people get it upstream...
There's even an out of tree fq_pie somewhere. Don't have the link handy.
-Toke
>
>> --
>> Mikael Abrahamsson email: swmike@swm.pp.se
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>_______________________________________________
>Ecn-sane mailing list
>Ecn-sane@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/ecn-sane
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 17:37 ` Loganaden Velvindron
2019-03-17 17:40 ` Toke Høiland-Jørgensen
@ 2019-03-17 17:44 ` Mikael Abrahamsson
2019-03-17 18:00 ` Dave Taht
2019-03-17 19:38 ` Rodney W. Grimes
2 siblings, 1 reply; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-17 17:44 UTC (permalink / raw)
To: Loganaden Velvindron; +Cc: Holland, Jake, ecn-sane, bloat
On Sun, 17 Mar 2019, Loganaden Velvindron wrote:
> is there an open source implementation of PIE which is close to what is
> used by the DOCSIS modems ?
PIE was added to the Linux kernel in 3.14 according to
https://www.linux.com/news/linux-314-release-no-pi-new-pie-fights-bufferbloat
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 17:44 ` Mikael Abrahamsson
@ 2019-03-17 18:00 ` Dave Taht
0 siblings, 0 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-17 18:00 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Loganaden Velvindron, ecn-sane, bloat
I helped land a more rfc - compliant version of pie into net-next late
last year.
ironically, we had one open bug re ecn -
https://github.com/gautamramk/FQ-PIE-for-Linux-Kernel/issues/2 -
docsis pie supported ecn not at all. I actually hold a good opinion of
pie - it is a good single queue aqm, more responsive to overload than
codel is. Which is why I worked on it to bring it up to spec finally.
Next up was polishing that version of fq-pie for linux inclusion. I
was unsure if it had the same codel-derived rate estimator as the bsd
one, because pie's original estimator fails with many queues. I always
wondered if that behavior would show up also in todays hw (64 on many
10GigE cards).
"Pie was added". Um, er, I (we) worked hard through 7 or so revisions
of the original quite crappy contractor written code, to make it
acceptable for mainline and for test. For free. In terms of billable
time, I was probably out $60k or more.
No "was" there. I know I shouldn't be resenting that phrasing, but I'd
like to obtain some credit for being fair and impartial. Also
thoroughly benchmarked it. and discarded it as I felt that a 5ms codel
target for good queue was better than a 20ms one for pie.
I'd offered multiple times to help fix up the dualpi code, but could
not look at it due to the frand patent, which I asked privately, be
removed.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 18:07 ` David P. Reed
@ 2019-03-17 18:05 ` Vint Cerf
2019-03-19 1:06 ` Bob Briscoe
2019-03-19 4:44 ` Greg White
2 siblings, 0 replies; 105+ messages in thread
From: Vint Cerf @ 2019-03-17 18:05 UTC (permalink / raw)
To: David P. Reed; +Cc: Holland, Jake, Mikael Abrahamsson, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 6844 bytes --]
thanks, David - that's the information I was looking for.
v
On Sun, Mar 17, 2019 at 2:07 PM David P. Reed <dpreed@deepplum.com> wrote:
> Vint -
>
>
>
> BBR is the end-to-end control logic that adjusts the source rate to match
> the share of the bolttleneck link it should use.
>
>
>
> It depends on getting reliable current congestion information via packet
> drops and/or ECN.
>
>
>
> So the proposal by these guys (not the cable guys) is an attempt to
> improve the quality of the congestion signal inserted by the router with
> the bottleneck outbound link.
>
>
>
> THe cable guys are trying to get a "private" field in the IP header for
> their own use.
>
>
>
>
>
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <
> dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <
> ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation
> and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
>
>> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
>> L4S has a much better possibility of actually getting deployment into
>> the
>> wider Internet packet-moving equipment than anything being talked
>> about
>> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be
>> as
>> good, but it fits better into actual silicon and it's being proposed
>> by
>> people who actually have better channels into the people setting hard
>> requirements.
>>
>> I suggest you consider joining them instead of opposing them.
>>
>>
>> Hi Mikael,
>>
>> I agree it makes sense that fq_anything has issues when you're talking
>> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
>> makes better sense there.
>>
>> But fq_x makes great sense and provides real value for the uplink in a
>> home, small office, coffee shop, etc. (if you run the final rate limit
>> on the home side of the access link.) I'm thinking maybe there's a
>> disconnect here driven by the different use cases for where AQMs can go.
>>
>> The thing is, each of these is the most likely congestion point at
>> different times, and it's worthwhile for each of them to be able to
>> AQM (and mark packets) under congestion.
>>
>> One of the several things that bothers me with L4S is that I've seen
>> precious little concern over interfering with the ability for another
>> different AQM in-path to mark packets, and because it changes the
>> semantics of CE, you can't have both working at the same time unless
>> they both do L4S.
>>
>> SCE needs a lot of details filled in, but it's so much cleaner that it
>> seems to me there's reasonably obvious answers to all (or almost all) of
>> those detail questions, and because the semantics are so much cleaner,
>> it's much easier to tell it's non-harmful.
>>
>> <aside regarding="non-harmful">
>> The point you raised in another thread about reordering is mostly
>> well-taken, and a good counterpoint to the claim "non-harmful relative
>> to L4S".
>>
>> To me it seems sad and dumb that switches ended up trying to make
>> ordering guarantees at cost of switching performance, because if it's
>> useful to put ordering in the switch, then it must be equally useful to
>> put it in the receiver's NIC or OS.
>>
>> So why isn't it in all the receivers' NIC or OS (where it would render
>> the switch's ordering efforts moot) instead of in all the switches?
>>
>> I'm guessing the answer is a competition trap for the switch vendors,
>> plus "with ordering goes faster than without, when you benchmark the
>> switch with typical load and current (non-RACK) receivers".
>>
>> If that's the case, it seems like the drive for a competitive advantage
>> caused deployment of a packet ordering workaround in the wrong network
>> location(s), out of a pure misalignment of incentives.
>>
>> RACK rates to fix that in the end, but a lot of damage is already done,
>> and the L4S approach gives switches a flag that can double as proof that
>> RACK is there on the receiver, so they can stop trying to order those
>> packets.
>>
>> So point granted, I understand and agree there's a cost to abandoning
>> that advantage.
>> </aside>
>>
>> But as you also said so well in another thread, this is important. ("The
>> last unicorn", IIRC.) How much does it matter if there's a feature that
>> has value today, but only until RACK is widely deployed? If you were
>> convinced RACK would roll out everywhere within 3 years and SCE would
>> produce better results than L4S over the following 15 years, would that
>> change your mind?
>>
>> It would for me, and that's why I'd like to see SCE explored before
>> making a call. I think at its core, it provides the same thing L4S does
>> (a high-fidelity explicit congestion signal for the sender), but with
>> much cleaner semantics that can be incrementally added to congestion
>> controls that people are already using.
>>
>> Granted, it still remains to be seen whether SCE in practice can match
>> the results of L4S, and L4S was here first. But it seems to me L4S comes
>> with some problems that have not yet been examined, and that are nicely
>> dodged by a SCE-based approach.
>>
>> If L4S really is as good as they seem to think, I could imagine getting
>> behind it, but I don't think that's proven yet. I'm not certain, but
>> all the comparative analyses I remember seeing have been from more or
>> less the same team, and I'm not convinced they don't have some
>> misaligned incentives of their own.
>>
>> I understand a lot of work has gone into L4S, but this move to jump it
>> from interesting experiment to de-facto standard without a more critical
>> review that digs deeper into some of the potential deployment problems
>> has me concerned.
>>
>> If it really does turn out to be good enough to be permanent, I'm not
>> opposed to it, but I'm just not convinced that it's non-harmful, and my
>> default position is that the cleaner solution is going to be better in
>> the long run, if they can do the same job.
>>
>> It's not that I want it to be a fight, but I do want to end up with the
>> best solution we can get. We only have the one internet.
>>
>> Just my 2c.
>>
>> -Jake
>>
>>
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
> --
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
[-- Attachment #2: Type: text/html, Size: 9736 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-16 21:57 ` Vint Cerf
2019-03-16 22:03 ` Dave Taht
2019-03-16 22:05 ` Holland, Jake
@ 2019-03-17 18:07 ` David P. Reed
2019-03-17 18:05 ` Vint Cerf
` (2 more replies)
2 siblings, 3 replies; 105+ messages in thread
From: David P. Reed @ 2019-03-17 18:07 UTC (permalink / raw)
To: Vint Cerf; +Cc: Holland, Jake, Mikael Abrahamsson, ecn-sane, bloat
[-- Attachment #1: Type: text/plain, Size: 6616 bytes --]
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
THe cable guys are trying to get a "private" field in the IP header for their own use.
-----Original Message-----
From: "Vint Cerf" <vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <[ jholland@akamai.com ]( mailto:jholland@akamai.com )> wrote:On 2019-03-15, 11:37, "Mikael Abrahamsson" <[ swmike@swm.pp.se ]( mailto:swmike@swm.pp.se )> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
[ Ecn-sane@lists.bufferbloat.net ]( mailto:Ecn-sane@lists.bufferbloat.net )
[ https://lists.bufferbloat.net/listinfo/ecn-sane ]( https://lists.bufferbloat.net/listinfo/ecn-sane )
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
[-- Attachment #2: Type: text/html, Size: 9003 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 17:37 ` Loganaden Velvindron
2019-03-17 17:40 ` Toke Høiland-Jørgensen
2019-03-17 17:44 ` Mikael Abrahamsson
@ 2019-03-17 19:38 ` Rodney W. Grimes
2 siblings, 0 replies; 105+ messages in thread
From: Rodney W. Grimes @ 2019-03-17 19:38 UTC (permalink / raw)
To: Loganaden Velvindron; +Cc: Mikael Abrahamsson, ecn-sane, bloat
> On Sun, Mar 17, 2019 at 6:06 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >
> > On Sat, 16 Mar 2019, Holland, Jake wrote:
> >
> > > Granted, it still remains to be seen whether SCE in practice can match
> > > the results of L4S, and L4S was here first. But it seems to me L4S comes
> > > with some problems that have not yet been examined, and that are nicely
> > > dodged by a SCE-based approach.
> >
> > I'm actually not that interested in an academic competition about what
> > solution gives the ultimate "best" outcome in simulation or in a lab.
> >
> > I am interested in good enough solutions that are actually deployable and
> > will get deployed, and doesn't have any pathological behaviour when it
> > comes to legacy traffic.
> >
> > Right now the Internet is full of deep FIFOs and they're not going away,
> > and they're not getting FQ_CODEL or CAKE.
> >
> > CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
> > congestion points we have in real life. These devices would have a much
> > easier time getting PIE or even RED, if it was just implemented.
> >
>
> is there an open source implementation of PIE which is close to what
> is used by the DOCSIS modems ?
I do not know if it is close to the DOCSIS modems,
but FreeBSD has PIE implemented
/usr/src/sys/netpfil/ipfw/dn_aqm_pie.c
/usr/src/sys/netpfil/ipfw/dn_aqm_pie.h
/usr/src/sys/netpfil/ipfw/dn_sched_fq_pie.c
> > Mikael Abrahamsson email: swmike@swm.pp.se
--
Rod Grimes rgrimes@freebsd.org
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 14:06 ` Mikael Abrahamsson
2019-03-17 17:37 ` Loganaden Velvindron
@ 2019-03-17 20:50 ` Luca Muscariello
2019-03-17 21:51 ` Toke Høiland-Jørgensen
2019-03-18 4:26 ` Mikael Abrahamsson
1 sibling, 2 replies; 105+ messages in thread
From: Luca Muscariello @ 2019-03-17 20:50 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: David P. Reed, Holland, Jake, bloat, ecn-sane
[-- Attachment #1: Type: text/plain, Size: 2933 bytes --]
To me there is substantial difference from something like fq_codel or
fq_pie where service differentiation is largely implicit
and approches largely based on explicit marking.
Approaches based on marking face technical and non technical challenges
that have been largely mentioned in these lists.
Fq_codel has a ton of litterature behind both theoretical and experimental
and it is something very close to optimality, in terms of completion time
and latency.
Fq_codel also incentivizes the development of better congestion control as
the reward is immediate. It also makes Internet performance
predictable.
Once we know that, the logical approach would be to try to approximate that
thing when the full mechanism is not possible because of a variety of
limitations.
This is the case in some DC switches that implement AFD+priority fair
queueing at 40Gbps.
Fq_codel has an outstanding footprint in terms of deployment.
Iliad deployed SFQ in 2005 nation wide and Fq_codel as soon as it was
available in France and is the second largest ISP.
Iliad/Free controls the development of both the home GW and the DSLAM.
They have recently started to commercialize 10Gbps to the home using
switched Ethernet.
I’m very tempted to test it.
Kudos to them for being able to prove it is possible as long as you control
the development of your equipment.
A logical next step to me seems to push chipcos to build fq_codel in
silicon.
It is totally feasible.
If on the other hand we say that we can achieve all fq_codel provides with
current chipsets we’ll never create the incentives to make progress.
My2c
Luca
On Sun 17 Mar 2019 at 15:06, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 16 Mar 2019, Holland, Jake wrote:
>
> > Granted, it still remains to be seen whether SCE in practice can match
> > the results of L4S, and L4S was here first. But it seems to me L4S comes
> > with some problems that have not yet been examined, and that are nicely
> > dodged by a SCE-based approach.
>
> I'm actually not that interested in an academic competition about what
> solution gives the ultimate "best" outcome in simulation or in a lab.
>
> I am interested in good enough solutions that are actually deployable and
> will get deployed, and doesn't have any pathological behaviour when it
> comes to legacy traffic.
>
> Right now the Internet is full of deep FIFOs and they're not going away,
> and they're not getting FQ_CODEL or CAKE.
>
> CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
> congestion points we have in real life. These devices would have a much
> easier time getting PIE or even RED, if it was just implemented.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
[-- Attachment #2: Type: text/html, Size: 4131 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 20:50 ` Luca Muscariello
@ 2019-03-17 21:51 ` Toke Høiland-Jørgensen
2019-03-18 4:26 ` Mikael Abrahamsson
1 sibling, 0 replies; 105+ messages in thread
From: Toke Høiland-Jørgensen @ 2019-03-17 21:51 UTC (permalink / raw)
To: Luca Muscariello, Mikael Abrahamsson; +Cc: ecn-sane, bloat
Luca Muscariello <luca.muscariello@gmail.com> writes:
> To me there is substantial difference from something like fq_codel or
> fq_pie where service differentiation is largely implicit
> and approches largely based on explicit marking.
>
> Approaches based on marking face technical and non technical challenges
> that have been largely mentioned in these lists.
>
> Fq_codel has a ton of litterature behind both theoretical and experimental
> and it is something very close to optimality, in terms of completion time
> and latency.
>
> Fq_codel also incentivizes the development of better congestion control as
> the reward is immediate. It also makes Internet performance
> predictable.
>
> Once we know that, the logical approach would be to try to approximate that
> thing when the full mechanism is not possible because of a variety of
> limitations.
>
> This is the case in some DC switches that implement AFD+priority fair
> queueing at 40Gbps.
>
> Fq_codel has an outstanding footprint in terms of deployment.
> Iliad deployed SFQ in 2005 nation wide and Fq_codel as soon as it was
> available in France and is the second largest ISP.
> Iliad/Free controls the development of both the home GW and the DSLAM.
> They have recently started to commercialize 10Gbps to the home using
> switched Ethernet.
> I’m very tempted to test it.
>
> Kudos to them for being able to prove it is possible as long as you control
> the development of your equipment.
>
> A logical next step to me seems to push chipcos to build fq_codel in
> silicon.
> It is totally feasible.
>
> If on the other hand we say that we can achieve all fq_codel provides with
> current chipsets we’ll never create the incentives to make progress.
+100!
-Toke
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 20:50 ` Luca Muscariello
2019-03-17 21:51 ` Toke Høiland-Jørgensen
@ 2019-03-18 4:26 ` Mikael Abrahamsson
1 sibling, 0 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-18 4:26 UTC (permalink / raw)
To: Luca Muscariello; +Cc: David P. Reed, Holland, Jake, bloat, ecn-sane
On Sun, 17 Mar 2019, Luca Muscariello wrote:
> Fq_codel has an outstanding footprint in terms of deployment.
No, it doesn't.
> A logical next step to me seems to push chipcos to build fq_codel in
> silicon.
> It is totally feasible.
... and how do you plan to make that happen?
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 18:07 ` David P. Reed
2019-03-17 18:05 ` Vint Cerf
@ 2019-03-19 1:06 ` Bob Briscoe
2019-03-19 3:18 ` Dave Taht
2019-03-20 19:04 ` Holland, Jake
2019-03-19 4:44 ` Greg White
2 siblings, 2 replies; 105+ messages in thread
From: Bob Briscoe @ 2019-03-19 1:06 UTC (permalink / raw)
To: David P. Reed, Vint Cerf; +Cc: bloat, tsvwg IETF list
[-- Attachment #1: Type: text/plain, Size: 14094 bytes --]
David,
On 17/03/2019 18:07, David P. Reed wrote:
>
> Vint -
>
> BBR is the end-to-end control logic that adjusts the source rate to
> match the share of the bolttleneck link it should use.
>
> It depends on getting reliable current congestion information via
> packet drops and/or ECN.
>
> So the proposal by these guys (not the cable guys) is an attempt to
> improve the quality of the congestion signal inserted by the router
> with the bottleneck outbound link.
>
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
> THe cable guys are trying to get a "private" field in the IP header
> for their own use.
>
There is nothing private about this codepoint, and there never has been.
Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was
proposed for use as an alternative ECN behaviour in July 2105 in the
IETF AQM WG and the IETF's transport area WG (which handles all ECN
matters).
* A year later there followed a packed IETF BoF on the subject (after 2
open Bar BoFs).
* Long discussion ensued on the merits of different IP header field
combinations, on both these IETF lists, involving people active on this
list (bloat), including Dave Taht, who is acknowledged for his
contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of
draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and
co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to
lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very
visible at every IETF ever since, with multiple drafts to fix other
aspects of the protocols working their way through the IETF process in
multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van
Jacobson's first presentation of CoDel at the IETF in July 2012. VJ
overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project
(Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel
Lucent and myself (I had just switched from BT to Simula), along with
Olga Bondarenko (now Albisser), a PhD student at Simula who now works
for Microsoft (on something else) and is still doing her PhD part-time
with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM
WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/
packet from all the following simultaneous applications achieving ~1ms
queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable industry
has been funding my work at the IETF to help encourage a wider L4S
ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access
technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the
low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was
published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as
well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused
on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows
TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM
congestion controller for real-time media:
https://github.com/EricssonResearch/scream/
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the
necessary feedback in TCP https://github.com/mirjak/linux-accecn
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired
me later in the year to help develop and specify it, along with support
for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now
being implemented.
o You can find the primary companies involved at the end of the White
Paper.
https://cablela.bs/low-latency-docsis-technology-overview-february-2019
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the
necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the
moment
(the developer wants to announce it themselves, so I have been
asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out
highlights.
All this is good healthy development of much lower latency for Internet
technology.
I find it extremely disappointing that some people on this list are
taking such a negative attitude to the major development in their own
field that they seem not to have noticed since it first hit the
limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it and
make it better...
If I was in this position, having overlooked something important for
multiple years, I would certainly not condemn it while I was trying to
understand what it was and how it worked. Can I suggest everyone takes a
step back, and suspends judgement until they have understood the
potential, the goals and the depth of what they have missed. People who
know me, know that I am very careful with Internet architecture, and
particularly with balancing public policy against commercial issues.
Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into
the L4S queue. But only if it can keep latency down below around 1ms.
Currently those ~15-25ms delay spikes would not pass muster. Indeed, its
delay is not consistently low enough between the spikes either.
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed"
> <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"
> <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague]
> Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com
> <mailto:jholland@akamai.com>> wrote:
>
> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se
> <mailto:swmike@swm.pp.se>> wrote:
> L4S has a much better possibility of actually getting
> deployment into the
> wider Internet packet-moving equipment than anything being
> talked about
> here. Same with PIE as opposed to FQ_CODEL. I know it's might
> not be as
> good, but it fits better into actual silicon and it's being
> proposed by
> people who actually have better channels into the people
> setting hard
> requirements.
>
> I suggest you consider joining them instead of opposing them.
>
>
> Hi Mikael,
>
> I agree it makes sense that fq_anything has issues when you're talking
> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
> makes better sense there.
>
> But fq_x makes great sense and provides real value for the uplink in a
> home, small office, coffee shop, etc. (if you run the final rate limit
> on the home side of the access link.) I'm thinking maybe there's a
> disconnect here driven by the different use cases for where AQMs
> can go.
>
> The thing is, each of these is the most likely congestion point at
> different times, and it's worthwhile for each of them to be able to
> AQM (and mark packets) under congestion.
>
> One of the several things that bothers me with L4S is that I've seen
> precious little concern over interfering with the ability for another
> different AQM in-path to mark packets, and because it changes the
> semantics of CE, you can't have both working at the same time unless
> they both do L4S.
>
> SCE needs a lot of details filled in, but it's so much cleaner that it
> seems to me there's reasonably obvious answers to all (or almost
> all) of
> those detail questions, and because the semantics are so much cleaner,
> it's much easier to tell it's non-harmful.
>
> <aside regarding="non-harmful">
> The point you raised in another thread about reordering is mostly
> well-taken, and a good counterpoint to the claim "non-harmful relative
> to L4S".
>
> To me it seems sad and dumb that switches ended up trying to make
> ordering guarantees at cost of switching performance, because if it's
> useful to put ordering in the switch, then it must be equally
> useful to
> put it in the receiver's NIC or OS.
>
> So why isn't it in all the receivers' NIC or OS (where it would render
> the switch's ordering efforts moot) instead of in all the switches?
>
> I'm guessing the answer is a competition trap for the switch vendors,
> plus "with ordering goes faster than without, when you benchmark the
> switch with typical load and current (non-RACK) receivers".
>
> If that's the case, it seems like the drive for a competitive
> advantage
> caused deployment of a packet ordering workaround in the wrong network
> location(s), out of a pure misalignment of incentives.
>
> RACK rates to fix that in the end, but a lot of damage is already
> done,
> and the L4S approach gives switches a flag that can double as
> proof that
> RACK is there on the receiver, so they can stop trying to order those
> packets.
>
> So point granted, I understand and agree there's a cost to abandoning
> that advantage.
> </aside>
>
> But as you also said so well in another thread, this is
> important. ("The
> last unicorn", IIRC.) How much does it matter if there's a
> feature that
> has value today, but only until RACK is widely deployed? If you were
> convinced RACK would roll out everywhere within 3 years and SCE would
> produce better results than L4S over the following 15 years, would
> that
> change your mind?
>
> It would for me, and that's why I'd like to see SCE explored before
> making a call. I think at its core, it provides the same thing
> L4S does
> (a high-fidelity explicit congestion signal for the sender), but with
> much cleaner semantics that can be incrementally added to congestion
> controls that people are already using.
>
> Granted, it still remains to be seen whether SCE in practice can match
> the results of L4S, and L4S was here first. But it seems to me
> L4S comes
> with some problems that have not yet been examined, and that are
> nicely
> dodged by a SCE-based approach.
>
> If L4S really is as good as they seem to think, I could imagine
> getting
> behind it, but I don't think that's proven yet. I'm not certain, but
> all the comparative analyses I remember seeing have been from more or
> less the same team, and I'm not convinced they don't have some
> misaligned incentives of their own.
>
> I understand a lot of work has gone into L4S, but this move to jump it
> from interesting experiment to de-facto standard without a more
> critical
> review that digs deeper into some of the potential deployment problems
> has me concerned.
>
> If it really does turn out to be good enough to be permanent, I'm not
> opposed to it, but I'm just not convinced that it's non-harmful,
> and my
> default position is that the cleaner solution is going to be better in
> the long run, if they can do the same job.
>
> It's not that I want it to be a fight, but I do want to end up
> with the
> best solution we can get. We only have the one internet.
>
> Just my 2c.
>
> -Jake
>
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net <mailto:Ecn-sane@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
> --
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
[-- Attachment #2: Type: text/html, Size: 22073 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 1:06 ` Bob Briscoe
@ 2019-03-19 3:18 ` Dave Taht
2019-03-20 19:04 ` Holland, Jake
1 sibling, 0 replies; 105+ messages in thread
From: Dave Taht @ 2019-03-19 3:18 UTC (permalink / raw)
To: Bob Briscoe; +Cc: David P. Reed, Vint Cerf, tsvwg IETF list, bloat
On Tue, Mar 19, 2019 at 2:07 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
>
> David,
>
> On 17/03/2019 18:07, David P. Reed wrote:
>
> Vint -
>
>
>
> BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
>
>
>
> It depends on getting reliable current congestion information via packet drops and/or ECN.
>
>
>
> So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
>
> What do you mean 'not the cable guys'?
> This thread was reasonably civil until this intervention.
I think the inventor of udp, and the e2e argument has good reason to
blow his top. I also think vint asked a reasonable question - has this
been tested vs bbr?
>
>
> THe cable guys are trying to get a "private" field in the IP header for their own use.
>
>
> There is nothing private about this codepoint, and there never has been. Here's some data points:
I hope https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
gets comprehensively evaluated in comparison to the more private use
of ect_1 you are talking about,
> * The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
I called for the closure of that wg because it had become an endless
bikeshed. Since the aqm chairs refused to apply long honored
traditions of the ietf - like *running code*. I ended up forming a new
wg
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/
I like my rules a lot better than what ended up going on here. I do
hope more folk join my wg.
> * A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
From which many, including myself, ran screaming. I actually quit the
ietf last prague, and went off to implement real code in real
products. Shipping. I lost track of the numbers after they cracked
10m.
> * Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
I have called repeatedly that my name be removed from the l4s related
drafts, in private, and in public. I in no way endorsed this effort,
except as a (somewhat dubious) experiment.
> * That was when it was decided that ECT(1) was most appropriate.
Love the passive voice there. Everybody else had left the room.
> * The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
Yea, read that. Lousy logic. My notes from that period said - "prob
won't work in a container/vm/network namespace. design an experiment
to test it when running code arrives." Running code never did.
None of this stuff has been subjected to rigor we put the aqms through
in the aqm group. There's no public ns2, or ns3 models, no public
implementations at all... and y'all are proposing to write tcp prague
from scratch at a hackathon this weekend?
come on.
> * David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
yea, that was good. used that for wireguard and now SCE.
> * This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
amazing levels of funding for that bit. Why on earth couldn't you hire
a decent hacker and get a version of tcpprague.c done 3 years ago?
You're Doing it at a hackathon next week? You know how much testing
even a simple change to tcp or and aqm goes through at google or
bufferbloat.net? Even with running code and widescale live testing?
I've been trying to analyze one tiny change to fq_codel on real
networks for close to a year now.... seems to work, so that's part of
the now SCE enabled repo here:
https://github.com/dtaht/fq_codel_fast
> * L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
>
> Some history:
> * I had been researching the idea since 2012.
> * In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
> * Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
> * Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
> * By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
> o By that time, Al-Lu and Simula had a cool prototype running.
> o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
with a contrived benchmark.
> https://riteproject.eu/dctth/#1511dispatchwg
> * In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
not measuring any normal traffic. on a contrived, proprietary,
"bigbuckbunny" benchmark I have blasted for its misrepresentation of
how basic internet protocols work multiple times and suggested instead
that the enormous suite of tests we developed for pie/codel/fq_codel
at least be run. Your group has refused to do so.
Our flent and irtt benchmarks are GPL3, developed in public, with a
bog-std output format and terabytes of data published.They are
licensed that way so that they cannot be gamed.
Now that the code is finally starting to land everyone can go and run
some real benchmarks and form their own opinions.
> o cloud-rendered remote presence in a racing car within a VR headset
> o the interactive cloud-rendered video already linked above
> o an online gaming benchmark
> o HAS video streaming
> o a number of bulk file downloads
> o a heavy synthetic load of web browsing
Not one of these being independently reproducible or repeatable. No
source code, no means to verify.
>
> L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:
This works on wifi? Got code?
> * Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
> * BT was obviously interested in DSL,
> * but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
Just in terms of deployment fq_codel, in free.fr *alone* is well above
a million units (as of 2015), and persistently in the top rankings of
http://www.dslreports.com/speedtest/results/bufferbloat?up=1
You guys are at ZERO. Still.
> * In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
No open source developer can get near that code legally.
https://fsfe.org/activities/os/why-frand-is-bad-for-free-software.en.html
It's not open source with a frand patent. OSI statement here:
https://opensource.org/node/616
Period. I'd asked repeatedly that patent be lifted. crickets. I asked
nokia's contact for the ipr. crickets.
> Now at: https://github.com/L4STeam/sch_dualpi2_upstream
I am delighted you finally got some code possibly worth reviewing by experts.
> * Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows
Beating that patent took forever. I'm glad microsoft joined OIN.
> * In Jul 2016, the main IETF BoF on L4S was held:
> o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
> o along with Kevin Smith from Vodafone and
> o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
> o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/
I have to note that google congestion control (gcc) won in the
marketplace and I have no idea why anyone tries with scream anymore. I
liked some bits of nada.
> o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn
it does help to submit code to netdev for mainline consideration? this
repo has not had a commit since 2017.
> * In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
> o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.
Thought it was just another limited experiment til last month.
*really*. Not once in all our communications in the past 2 years did I
have the slightest clue what was up. Formed my own wg, under rules
true to the ietf's original spirit (
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/ ) - made
progress, submitted a *goooood * draft, and then, wow, all kinds of
stuff started coming to light that nobody knew about.
> o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019
> o Operators:
> Liberty Global
> Charter
> Rogers
> Comcast
> Shaw
> Cox Communications
I think david's characterization of the "cable industry" being behind
this is accurate.
> o Equipment vendors
> ARRIS
> Huawei
> Broadcom
> Intel
> Casa
> Nokia
> Cisco
> Videotron
> * Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
> * Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
where is the code?
> * L4S hardware is also being implemented for hi-speed switches at the moment
> (the developer wants to announce it themselves, so I have been asked not to identify them).
groovy.
>
> There's a lot more stuff been going on, but I've tried to pick out highlights.
>
> All this is good healthy development of much lower latency for Internet technology.
I look forward to benching it against normal traffic finally, esp
against bbr, and in a realistic scenario with fq_codeled home routers
(and sch_cake), and on wifi.
IMHO there's no L4S deployment plan that makes sense vs the existing
10-100m+ deployment of rfc3168 compliant fq_codel, but, now that
l4s/dualpi/tcppraguecode exists to test - I started drafting some
experiments to finally take a look at it. Maybe I'll get a chance to
talk about those.
>
>
> I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
>
> L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
>
> If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
>
> Best Regards
>
>
>
> Bob
>
> PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
>
>
>
>
>
>
>
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
>>
>> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
>> L4S has a much better possibility of actually getting deployment into the
>> wider Internet packet-moving equipment than anything being talked about
>> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
>> good, but it fits better into actual silicon and it's being proposed by
>> people who actually have better channels into the people setting hard
>> requirements.
>>
>> I suggest you consider joining them instead of opposing them.
>>
>>
>> Hi Mikael,
>>
>> I agree it makes sense that fq_anything has issues when you're talking
>> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
>> makes better sense there.
>>
>> But fq_x makes great sense and provides real value for the uplink in a
>> home, small office, coffee shop, etc. (if you run the final rate limit
>> on the home side of the access link.) I'm thinking maybe there's a
>> disconnect here driven by the different use cases for where AQMs can go.
>>
>> The thing is, each of these is the most likely congestion point at
>> different times, and it's worthwhile for each of them to be able to
>> AQM (and mark packets) under congestion.
>>
>> One of the several things that bothers me with L4S is that I've seen
>> precious little concern over interfering with the ability for another
>> different AQM in-path to mark packets, and because it changes the
>> semantics of CE, you can't have both working at the same time unless
>> they both do L4S.
>>
>> SCE needs a lot of details filled in, but it's so much cleaner that it
>> seems to me there's reasonably obvious answers to all (or almost all) of
>> those detail questions, and because the semantics are so much cleaner,
>> it's much easier to tell it's non-harmful.
>>
>> <aside regarding="non-harmful">
>> The point you raised in another thread about reordering is mostly
>> well-taken, and a good counterpoint to the claim "non-harmful relative
>> to L4S".
>>
>> To me it seems sad and dumb that switches ended up trying to make
>> ordering guarantees at cost of switching performance, because if it's
>> useful to put ordering in the switch, then it must be equally useful to
>> put it in the receiver's NIC or OS.
>>
>> So why isn't it in all the receivers' NIC or OS (where it would render
>> the switch's ordering efforts moot) instead of in all the switches?
>>
>> I'm guessing the answer is a competition trap for the switch vendors,
>> plus "with ordering goes faster than without, when you benchmark the
>> switch with typical load and current (non-RACK) receivers".
>>
>> If that's the case, it seems like the drive for a competitive advantage
>> caused deployment of a packet ordering workaround in the wrong network
>> location(s), out of a pure misalignment of incentives.
>>
>> RACK rates to fix that in the end, but a lot of damage is already done,
>> and the L4S approach gives switches a flag that can double as proof that
>> RACK is there on the receiver, so they can stop trying to order those
>> packets.
>>
>> So point granted, I understand and agree there's a cost to abandoning
>> that advantage.
>> </aside>
>>
>> But as you also said so well in another thread, this is important. ("The
>> last unicorn", IIRC.) How much does it matter if there's a feature that
>> has value today, but only until RACK is widely deployed? If you were
>> convinced RACK would roll out everywhere within 3 years and SCE would
>> produce better results than L4S over the following 15 years, would that
>> change your mind?
>>
>> It would for me, and that's why I'd like to see SCE explored before
>> making a call. I think at its core, it provides the same thing L4S does
>> (a high-fidelity explicit congestion signal for the sender), but with
>> much cleaner semantics that can be incrementally added to congestion
>> controls that people are already using.
>>
>> Granted, it still remains to be seen whether SCE in practice can match
>> the results of L4S, and L4S was here first. But it seems to me L4S comes
>> with some problems that have not yet been examined, and that are nicely
>> dodged by a SCE-based approach.
>>
>> If L4S really is as good as they seem to think, I could imagine getting
>> behind it, but I don't think that's proven yet. I'm not certain, but
>> all the comparative analyses I remember seeing have been from more or
>> less the same team, and I'm not convinced they don't have some
>> misaligned incentives of their own.
>>
>> I understand a lot of work has gone into L4S, but this move to jump it
>> from interesting experiment to de-facto standard without a more critical
>> review that digs deeper into some of the potential deployment problems
>> has me concerned.
>>
>> If it really does turn out to be good enough to be permanent, I'm not
>> opposed to it, but I'm just not convinced that it's non-harmful, and my
>> default position is that the cleaner solution is going to be better in
>> the long run, if they can do the same job.
>>
>> It's not that I want it to be a fight, but I do want to end up with the
>> best solution we can get. We only have the one internet.
>>
>> Just my 2c.
>>
>> -Jake
>>
>>
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
> --
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> --
> ________________________________________________________________
> Bob Briscoe http://bobbriscoe.net/
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-17 18:07 ` David P. Reed
2019-03-17 18:05 ` Vint Cerf
2019-03-19 1:06 ` Bob Briscoe
@ 2019-03-19 4:44 ` Greg White
2019-03-19 5:35 ` Jonathan Morton
` (2 more replies)
2 siblings, 3 replies; 105+ messages in thread
From: Greg White @ 2019-03-19 4:44 UTC (permalink / raw)
To: David P. Reed, Vint Cerf; +Cc: bloat, ecn-sane
[-- Attachment #1: Type: text/plain, Size: 10232 bytes --]
That is ridiculous.
You clearly haven’t read the drafts, and so are speaking from a position of ignorance. Please get informed before making statements like this.
There is *absolutely* nothing cable-specific or “private” about L4S. It is being developed in an open forum, the IETF!! Yes, the cable industry is adopting L4S – because we recognized its potential. Others are too, and anyone can. It is a totally open spec, and has been since the initial drafts came out of the RITE project. The cable industry was not involved in RITE (in fact the technology was first demonstrated on DSL equipment), and we learned about L4S when the rest of the world did. We decided to become early adopters. Yes, we were quiet about the fact that we were planning on adopting it (until now).
If individuals drop out of participating in the IETF, they shouldn’t be upset if the IETF continues to make progress on developing Internet technology in their absence. It seems pretty disingenuous for DT to form his own private working group to come up with an incompatible, and limited, alternative to the ongoing work in IETF, then spring it on the IETF and start this FUD war.
The craziest thing about this whole thread is that there is a heck of a lot in common between L4S and SCE. More in common than different. My initial belief was that we all had similar goals (eliminating buffering latency in the internet) and we’d be able to achieve a meeting of the minds on the best way to use ECT(1) to achieve it. Now, I’m not so sure what DT’s motivation is.
If I can boil this down for the people who are jumping into this without reading the drafts:
* Both L4S and SCE are attempting to provide congestion-controlled senders with better congestion signals so that flows can achieve link capacity without buffering delay.
* Both are proposing to use ECT(1) as part of the mechanism, but to use it in different ways.
* SCE’s usage of ECT(1) potentially allows an automatic fallback to traditional Cubic behavior if the bottleneck link is a single-queue classic-ECN AQM (do any of these exist?), whereas L4S will need to detect such a condition via RTT measurement
* L4S’s usage of ECT(1) allows links to identify new senders and take advantage of new sender features like reordering tolerance that can further drive down latency in many common link technologies.
* SCE will only work if the bottleneck link implements fq. Some bottleneck network gear will not be able to implement fq or will not implement it due to its undesirable side effects (see section 6 of RFC 8290).
* L4S will work if the bottleneck link implements *either* fq or dual queue.
Beyond that, they are *very,very* similar.
But, L4S has been demonstrated in real equipment and in simulation, and leverages an existing congestion controller that is available in Linux and Windows (with some tweaks). SCE leverages a paragraph in a draft that describes a first guess about how a congestion controller might work.
L4S has defined a congestion feedback mechanism so that these congestion signals can get back to the sender. SCE offers that “we’ll propose something later”.
BBR currently does not listen to explicit congestion signals, but it could be updated to do so (for either SCE or L4S).
-Greg
From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "David P. Reed" <dpreed@deepplum.com>
Date: Sunday, March 17, 2019 at 12:07 PM
To: Vint Cerf <vint@google.com>
Cc: bloat <bloat@lists.bufferbloat.net>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
THe cable guys are trying to get a "private" field in the IP header for their own use.
-----Original Message-----
From: "Vint Cerf" <vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
[-- Attachment #2: Type: text/html, Size: 21730 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 4:44 ` Greg White
@ 2019-03-19 5:35 ` Jonathan Morton
2019-03-19 5:52 ` Greg White
2019-03-19 8:50 ` Sebastian Moeller
2019-03-19 23:59 ` Dave Taht
2 siblings, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-19 5:35 UTC (permalink / raw)
To: Greg White; +Cc: David P. Reed, Vint Cerf, ecn-sane, bloat
> • SCE will only work if the bottleneck link implements fq. Some bottleneck network gear will not be able to implement fq or will not implement it due to its undesirable side effects (see section 6 of RFC 8290).
> SCE leverages a paragraph in a draft that describes a first guess about how a congestion controller might work.
We have an update in the works that should enable RTT-fair convergence with single-queue AQMs, whether they support SCE or not. To do this using the simplest reasonable adjustments to existing congestion control algorithms, the setpoint is no longer fixed at 50% but varies with the cwnd of each flow. And yes, we have worked out what those adjustments should look like in practice, but we haven't yet had time to run tests, or describe them in a nicely formatted I-D.
This update should also allow a very DCTCP-like congestion control algorithm to work correctly with SCE, as long as it acts conventionally with CE marks and only has the reduced response to SCE marks. That's something that DCTCP itself currently does not do.
> • SCE’s usage of ECT(1) potentially allows an automatic fallback to traditional Cubic behavior if the bottleneck link is a single-queue classic-ECN AQM (do any of these exist?), whereas L4S will need to detect such a condition via RTT measurement
From my standpoint, the major objection to L4S is that it is not incrementally deployable, because DCTCP starves conventional TCPs unless run through an isolated queue. This is something we quickly realised when L4S was first announced. It is simply not practical to require all middleboxes on the path to support L4S before L4S endpoints can safely be deployed, except in the isolated and very controlled environments where DCTCP was "proved".
> I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
It is not at all true that we were unaware of L4S. Rather, we quite reasonably believed it would never get traction in the IETF or in the Internet at large unless that problem was robustly solved - and we thought the obvious solution *was* to use ECT(1) as SCE does, and to fix Diffserv (so that it becomes e2e usable to some degree) or implement FQ if isolating low-latency-assured traffic is so important.
Incidentally, during that time we were implementing a good FQ system that is now in Linux and in commercial products. Granted, it isn't designed for the sort of high-capacity links where FQ is traditionally considered impractical.
> L4S has defined a congestion feedback mechanism so that these congestion signals can get back to the sender. SCE offers that “we’ll propose something later”.
It should be straightforward to adjust AccECN so that it primarily carries SCE information instead of unnecessarily detailed CE information. That's one obvious solution, which we hoped those already familiar with L4S would recognise without explicit prompting.
We have outlines of other feedback methods, still awaiting conversion to the proper document format, including one that doesn't require a new TCP option (I understand there are objections to such things, centred around paranoid firewalls) and which existing middleboxes and endpoints will transparently ignore (like the rest of SCE). It uses the NS bit which was also freed up by the obsoleting of Nonce Sum.
The I-D presently available defines the SCE codepoint and very little else. This is due to separation of concerns. Other I-Ds will define feedback mechanisms, detailed modifications to congestion control algorithms, and recommendations as to what AQMs should do.
Perhaps if we get a chance to work, instead of responding to manufactured outrage that we dare to propose something different, these extra documents might surface in time for discussion.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 5:35 ` Jonathan Morton
@ 2019-03-19 5:52 ` Greg White
2019-03-19 7:10 ` Jonathan Morton
0 siblings, 1 reply; 105+ messages in thread
From: Greg White @ 2019-03-19 5:52 UTC (permalink / raw)
To: Jonathan Morton; +Cc: David P. Reed, Vint Cerf, ecn-sane, bloat
On 3/18/19, 11:35 PM, "Jonathan Morton" <chromatix99@gmail.com> wrote:
From my standpoint, the major objection to L4S is that it is not incrementally deployable, because DCTCP starves conventional TCPs unless run through an isolated queue. This is something we quickly realised when L4S was first announced. It is simply not practical to require all middleboxes on the path to support L4S before L4S endpoints can safely be deployed, except in the isolated and very controlled environments where DCTCP was "proved".
[GW] But this isn't true! L4S utilizes TCP Prague, which falls back to traditional congestion control if the bottleneck link doesn't provide isolation.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 5:52 ` Greg White
@ 2019-03-19 7:10 ` Jonathan Morton
2019-03-19 8:07 ` Sebastian Moeller
0 siblings, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-19 7:10 UTC (permalink / raw)
To: Greg White; +Cc: David P. Reed, Vint Cerf, ecn-sane, bloat
> On 19 Mar, 2019, at 7:52 am, Greg White <g.white@CableLabs.com> wrote:
>
> L4S utilizes TCP Prague, which falls back to traditional congestion control if the bottleneck link doesn't provide isolation.
You see, this is the part I find difficult to believe that it will operate reliably. For a start, I have seen no detailed public description of TCP Prague, even though it has supposedly been in "open" development for so long. My most recent information is that it's essentially a slightly modified DCTCP.
" It would take time for endpoints to distinguish classic and L4S ECN
marking. An increase in queuing delay or in delay variation would be
a tell-tale sign, but it is not yet clear where a line would be drawn
between the two behaviours. "
Internet history is littered with failed attempts at implementing delay-sensitive TCPs. I can immediately think of several reasons why delay can and will vary for reasons other than the bottleneck not implementing an isolated queue (just ask the BBR devs). The mere presence of a wifi link on the path, even if it is never the bottleneck, would be a trivial and common example.
So please explain (or point to good documentation) how TCP Prague robustly avoids misbehaving in a standard ECN environment, as is presently deployed.
SCE explicitly does not rely on specific changes in behaviour by endpoints. It just provides a conduit of information from the network to the receiver, in addition to standard ECN behaviour. The receiver is free to ignore that information, without harming the network, and will naturally behave normally and safely when that information is absent. We have a proof-of-concept implementation (a trivial mod of sch_codel and sch_fq_codel) which successfully passes this information across the Internet and works with (is transparently ignored by) existing endpoints and middleboxes.
In short, SCE is incrementally deployable by design.
The broader system of feedback and modified congestion control, which I call ELR (Explicit Load Regulation) as an umbrella term, offers benefits which, yes, have not yet been proved - but which are straightforward in concept and should be amenable to analysis. It seems likely that some work from L4S can be used in this context.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 7:10 ` Jonathan Morton
@ 2019-03-19 8:07 ` Sebastian Moeller
0 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-19 8:07 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Greg White, ecn-sane, Vint Cerf, David P. Reed, bloat
> On Mar 19, 2019, at 08:10, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 19 Mar, 2019, at 7:52 am, Greg White <g.white@CableLabs.com> wrote:
>>
>> L4S utilizes TCP Prague, which falls back to traditional congestion control if the bottleneck link doesn't provide isolation.
>
> You see, this is the part I find difficult to believe that it will operate reliably. For a start, I have seen no detailed public description of TCP Prague, even though it has supposedly been in "open" development for so long. My most recent information is that it's essentially a slightly modified DCTCP.
>
> " It would take time for endpoints to distinguish classic and L4S ECN
> marking. An increase in queuing delay or in delay variation would be
> a tell-tale sign, but it is not yet clear where a line would be drawn
> between the two behaviours. "
IMHO this is caused by the fact that ECT(1) is simply not suitable as a classifier, as it will only reliably classify unmarked packets, anything marked CE will looses the destinction whether the flow considers itself L4S ready or not. Could anyone point me to the section in the L4S RFCs that discusses this, please?
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/ has the following:
" Given shortage of codepoints, both L4S and classic ECN sides of
an AQM would have to use the same CE codepoint to indicate that
a packet had experienced congestion. If a packet that had
already been marked CE in an upstream buffer arrived at a
subsequent AQM, this AQM would then have to guess whether to
classify CE packets as L4S or classic ECN. Choosing the L4S
treatment would be a safer choice, because then a few classic
packets might arrive early, rather than a few L4S packets
arriving late;"
But how is the L4S queue actually maintaining its low latency if it just admitted an non-L4S flow? This _might_ work if CE marked packets are rare, but IMHO this confirms my assessment that ECT(1) is a terrible choice for a classifier bit.
Best Regards
Sebastian
> Internet history is littered with failed attempts at implementing delay-sensitive TCPs. I can immediately think of several reasons why delay can and will vary for reasons other than the bottleneck not implementing an isolated queue (just ask the BBR devs). The mere presence of a wifi link on the path, even if it is never the bottleneck, would be a trivial and common example.
>
> So please explain (or point to good documentation) how TCP Prague robustly avoids misbehaving in a standard ECN environment, as is presently deployed.
>
>
> SCE explicitly does not rely on specific changes in behaviour by endpoints. It just provides a conduit of information from the network to the receiver, in addition to standard ECN behaviour. The receiver is free to ignore that information, without harming the network, and will naturally behave normally and safely when that information is absent. We have a proof-of-concept implementation (a trivial mod of sch_codel and sch_fq_codel) which successfully passes this information across the Internet and works with (is transparently ignored by) existing endpoints and middleboxes.
>
> In short, SCE is incrementally deployable by design.
>
> The broader system of feedback and modified congestion control, which I call ELR (Explicit Load Regulation) as an umbrella term, offers benefits which, yes, have not yet been proved - but which are straightforward in concept and should be amenable to analysis. It seems likely that some work from L4S can be used in this context.
>
> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 4:44 ` Greg White
2019-03-19 5:35 ` Jonathan Morton
@ 2019-03-19 8:50 ` Sebastian Moeller
2019-03-19 23:59 ` Dave Taht
2 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-19 8:50 UTC (permalink / raw)
To: Greg White; +Cc: ecn-sane, bloat
> On Mar 19, 2019, at 05:44, Greg White <g.white@CableLabs.com> wrote:
> If I can boil this down for the people who are jumping into this without reading the drafts:
>
> • Both L4S and SCE are attempting to provide congestion-controlled senders with better congestion signals so that flows can achieve link capacity without buffering delay.
> • Both are proposing to use ECT(1) as part of the mechanism, but to use it in different ways.
SCE tries to encode information about the quantitative congestion state of the marking AQM into ECT(1), while L4S tries to use this as a general identifier of promised behavior as a receiver of CE marks, or rather as an indication that flows marked ECT(1) will not respond to CE marks as described in rfc3168. Which realistically means any non-L4S AQM needs to learn quickly to drop ECT(1) packets instead of marking them CE; that seems better controlled than waiting for a fall-back to rfc3168-compliant CE response due to a heuristic based on RTT variation.
> • SCE’s usage of ECT(1) potentially allows an automatic fallback to traditional Cubic behavior if the bottleneck link is a single-queue classic-ECN AQM (do any of these exist?), whereas L4S will need to detect such a condition via RTT measurement
> • L4S’s usage of ECT(1) allows links to identify new senders and take advantage of new sender features like reordering tolerance that can further drive down latency in many common link technologies.
But L4S is incapable of _reliably_ classifying L4S flows/packets as CE-marked packets default to L4S-treatment. This indicates to me, that ECT(1) is not really suited as a reliable L4S identifier, what am I missing?
This ambiguity leads to the question of the side-effects of this leaky classification: what about re-ordering of CE-marked packets? I hope that out of caution CE-marked packets will not be re-ordered as these are very much not guaranteed to employ RACK. (And tangentially, how is a link that desires more latitude for re-ordering going to deal with the RACK requirement to keep the re-ordering windows <= 1 RTT, given that RTTs over the internet differ from a few to dozens of ms. . Is there any study showing how RACK and re-ordering actually interact in real-life?) And how is it going to help a link in regards to re-ordering at all? It has been argued, that links do not differentiate flows at all, and assuming TCP traffic to coexist for a long time with (DC)TCP_Prague traffic, how can a link actually allow more re-ordering than currently tolerable without severely impacting the TCP flows? If it just transmits all ECT(1) packets in its queue things will be a bit better than now, but after the egress queue is emptied the link might still be stalled until the re-transmit of ECT(0) and CE marked packets is finished, no?
> • SCE will only work if the bottleneck link implements fq. Some bottleneck network gear will not be able to implement fq or will not implement it due to its undesirable side effects (see section 6 of RFC 8290).
> • L4S will work if the bottleneck link implements *either* fq or dual queue.
The proof ought to e in the pudding ;) is there data showing an working L4S fq-AQM?
>
> Beyond that, they are *very,very* similar.
>
> But, L4S has been demonstrated in real equipment and in simulation, and leverages an existing congestion controller that is available in Linux and Windows (with some tweaks).
As far as I can see the public git repository for TCP Prague is only a few days old so how could that be "available in Linux and Windows" right now, and one could similarly argue that it will only take a few tweaks to teach cubic how to deal with SCE.
So I have no pony in this race as I am outside of the field, but the L4S RFCs seem to promise more than they
> SCE leverages a paragraph in a draft that describes a first guess about how a congestion controller might work.
>
> L4S has defined a congestion feedback mechanism so that these congestion signals can get back to the sender. SCE offers that “we’ll propose something later”.
>
> BBR currently does not listen to explicit congestion signals, but it could be updated to do so (for either SCE or L4S).
>
> -Greg
>
>
> From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "David P. Reed" <dpreed@deepplum.com>
> Date: Sunday, March 17, 2019 at 12:07 PM
> To: Vint Cerf <vint@google.com>
> Cc: bloat <bloat@lists.bufferbloat.net>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>
> Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> Vint -
>
> BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
>
> It depends on getting reliable current congestion information via packet drops and/or ECN.
>
> So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
>
> THe cable guys are trying to get a "private" field in the IP header for their own use.
>
>
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
>> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
>> L4S has a much better possibility of actually getting deployment into the
>> wider Internet packet-moving equipment than anything being talked about
>> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
>> good, but it fits better into actual silicon and it's being proposed by
>> people who actually have better channels into the people setting hard
>> requirements.
>>
>> I suggest you consider joining them instead of opposing them.
>>
>>
>> Hi Mikael,
>>
>> I agree it makes sense that fq_anything has issues when you're talking
>> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
>> makes better sense there.
>>
>> But fq_x makes great sense and provides real value for the uplink in a
>> home, small office, coffee shop, etc. (if you run the final rate limit
>> on the home side of the access link.) I'm thinking maybe there's a
>> disconnect here driven by the different use cases for where AQMs can go.
>>
>> The thing is, each of these is the most likely congestion point at
>> different times, and it's worthwhile for each of them to be able to
>> AQM (and mark packets) under congestion.
>>
>> One of the several things that bothers me with L4S is that I've seen
>> precious little concern over interfering with the ability for another
>> different AQM in-path to mark packets, and because it changes the
>> semantics of CE, you can't have both working at the same time unless
>> they both do L4S.
>>
>> SCE needs a lot of details filled in, but it's so much cleaner that it
>> seems to me there's reasonably obvious answers to all (or almost all) of
>> those detail questions, and because the semantics are so much cleaner,
>> it's much easier to tell it's non-harmful.
>>
>> <aside regarding="non-harmful">
>> The point you raised in another thread about reordering is mostly
>> well-taken, and a good counterpoint to the claim "non-harmful relative
>> to L4S".
>>
>> To me it seems sad and dumb that switches ended up trying to make
>> ordering guarantees at cost of switching performance, because if it's
>> useful to put ordering in the switch, then it must be equally useful to
>> put it in the receiver's NIC or OS.
>>
>> So why isn't it in all the receivers' NIC or OS (where it would render
>> the switch's ordering efforts moot) instead of in all the switches?
>>
>> I'm guessing the answer is a competition trap for the switch vendors,
>> plus "with ordering goes faster than without, when you benchmark the
>> switch with typical load and current (non-RACK) receivers".
>>
>> If that's the case, it seems like the drive for a competitive advantage
>> caused deployment of a packet ordering workaround in the wrong network
>> location(s), out of a pure misalignment of incentives.
>>
>> RACK rates to fix that in the end, but a lot of damage is already done,
>> and the L4S approach gives switches a flag that can double as proof that
>> RACK is there on the receiver, so they can stop trying to order those
>> packets.
>>
>> So point granted, I understand and agree there's a cost to abandoning
>> that advantage.
>> </aside>
>>
>> But as you also said so well in another thread, this is important. ("The
>> last unicorn", IIRC.) How much does it matter if there's a feature that
>> has value today, but only until RACK is widely deployed? If you were
>> convinced RACK would roll out everywhere within 3 years and SCE would
>> produce better results than L4S over the following 15 years, would that
>> change your mind?
>>
>> It would for me, and that's why I'd like to see SCE explored before
>> making a call. I think at its core, it provides the same thing L4S does
>> (a high-fidelity explicit congestion signal for the sender), but with
>> much cleaner semantics that can be incrementally added to congestion
>> controls that people are already using.
>>
>> Granted, it still remains to be seen whether SCE in practice can match
>> the results of L4S, and L4S was here first. But it seems to me L4S comes
>> with some problems that have not yet been examined, and that are nicely
>> dodged by a SCE-based approach.
>>
>> If L4S really is as good as they seem to think, I could imagine getting
>> behind it, but I don't think that's proven yet. I'm not certain, but
>> all the comparative analyses I remember seeing have been from more or
>> less the same team, and I'm not convinced they don't have some
>> misaligned incentives of their own.
>>
>> I understand a lot of work has gone into L4S, but this move to jump it
>> from interesting experiment to de-facto standard without a more critical
>> review that digs deeper into some of the potential deployment problems
>> has me concerned.
>>
>> If it really does turn out to be good enough to be permanent, I'm not
>> opposed to it, but I'm just not convinced that it's non-harmful, and my
>> default position is that the cleaner solution is going to be better in
>> the long run, if they can do the same job.
>>
>> It's not that I want it to be a fight, but I do want to end up with the
>> best solution we can get. We only have the one internet.
>>
>> Just my 2c.
>>
>> -Jake
>>
>>
>> _______________________________________________
>> Ecn-sane mailing list
>> Ecn-sane@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/ecn-sane
>
> --
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 4:44 ` Greg White
2019-03-19 5:35 ` Jonathan Morton
2019-03-19 8:50 ` Sebastian Moeller
@ 2019-03-19 23:59 ` Dave Taht
2019-03-20 10:17 ` Sebastian Moeller
2 siblings, 1 reply; 105+ messages in thread
From: Dave Taht @ 2019-03-19 23:59 UTC (permalink / raw)
To: Greg White; +Cc: David P. Reed, Vint Cerf, ecn-sane, bloat, tsvwg IETF list
On Tue, Mar 19, 2019 at 5:44 AM Greg White <g.white@cablelabs.com> wrote:
>
> That is ridiculous.
>
>
>
> You clearly haven’t read the drafts, and so are speaking from a position of ignorance. Please get informed before making statements like this.
I've read all the drafts, 3 times, and made pithy comments when they
came out. I've been trying to get other knowlegable people in the
daily grind of modern DC netops to read them also for years. I hope
more, now do.
>
>
> There is *absolutely* nothing cable-specific or “private” about L4S. It is being developed in an open forum, the IETF!! Yes, the cable industry is adopting L4S – because we recognized its potential. Others are too, and anyone can. It is a totally open spec, and has been since the initial drafts came out of the RITE project. The cable industry was not involved in RITE (in fact the technology was first demonstrated on DSL equipment), and we learned about L4S when the rest of the world did. We decided to become early adopters. Yes, we were quiet about the fact that we were planning on adopting it (until now).
>
>
>
> If individuals drop out of participating in the IETF, they shouldn’t be upset if the IETF continues to make progress on developing Internet technology in their absence. It seems pretty disingenuous for DT to form his own private working group to come up with an incompatible, and limited, alternative to the ongoing work in IETF, then spring it on the IETF and start this FUD war.
We announced it publicly back in august with our charter and goals on
bloat and aqm mailing lists.
Nobody spoke up then saying l4s was still alive either in the ietf or privately.
We published a charter and outline of scope:
https://www.bufferbloat.net/projects/ecn-sane/wiki/
And we formed into teams:
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/
And published position papers.
6 months of work on dealing with ecn issues finally has as it's first
result, the SCE draft.
You calling me disingenuous is disingenuous.
>
>
> The craziest thing about this whole thread is that there is a heck of a lot in common between L4S and SCE.
Yep, they do!
Yes, I think we achieved consensus long ago that a single queued AQM
cannot work with an earlier indication of congestion than drop without
changing the endpoints. We also achieved consensus that an earlier
congestion notification was potentially useful in some respects.
Repurposing CE, and using up ECT_1, was all that was on the table before.
>More in common than different. My initial belief was that we all had similar goals (eliminating buffering latency in the internet) and we’d be able to achieve a meeting of the minds on the best way to use ECT(1) to achieve it.
If I could merely have got the cable industry to reduce the buffering
in their cmts ends from 600+ms to 100ms in the last 8 years, I'd have
been happy.
Two things. 1) I believe - along with just about everybody I know in
the DC business - few that have bothered to think of ecn at all -
(none have deployed it). In every conversation about l4s is the
presumption that the other person i the conversation already thinks
ecn is a good idea. I know a lot of people that don't think it is. Me,
when it comes to ECN I am firmly on the yellow team - chicken - see
above position paper - in general.
2) Bufferbloat.net deployed solutions starting in 2012 that work for
fixing both inbound and outbound buffering. We shipped sch_cake in
august after 3 years of deployment testing. I would rather like y'all
to add sch_cake to your test matrix.
> Now, I’m not so sure what DT’s motivation is.
My motivation is always to eliminate bufferbloat and build a better
internet. It's the motto on my patreon page.
>
>
>
> If I can boil this down for the people who are jumping into this without reading the drafts:
>
>
>
> Both L4S and SCE are attempting to provide congestion-controlled senders with better congestion signals so that flows can achieve link capacity without buffering delay.
> Both are proposing to use ECT(1) as part of the mechanism, but to use it in different ways.
ECT(1) from a unwritten, specialized tcp that is incompatible with
BBR, Cubic, and Reno.
ECT(1) from a well established, shipping AQM.
> SCE’s usage of ECT(1) potentially allows an automatic fallback to traditional Cubic behavior if the bottleneck link is a single-queue classic-ECN AQM (do any of these exist?), whereas L4S will need to detect such a condition via RTT measurement
Strike "potentially". Does. Instead of writing further emails, I'm
focusing on patching tcptrace and xplot to show this clearly from
packet captures. Perhaps that will be ready by the hackathon.
> L4S’s usage of ECT(1) allows links to identify new senders and take advantage of new sender features like reordering tolerance that can further drive down latency in many common link technologies.
There's nothing in SCE that stops the exact same idea, except that the
behavior is chosen reciever side, not sender side.
> SCE will only work if the bottleneck link implements fq. Some bottleneck network gear will not be able to implement fq or will not implement it due to its undesirable side effects (see section 6 of RFC 8290).
This is overbroad.
And I didn't write that section.
> L4S will work if the bottleneck link implements *either* fq or dual queue.
And this is not a demonstrated fact yet - *at all* that needs to
yield to experiment and independent analysis.
That's a good start actually to writing up a good comparison document.
I'll save that and work on it, if
I get a talk slot for it.
>
>
> Beyond that, they are *very,very* similar.
Yes!
>
>
>
> But, L4S has been demonstrated in real equipment and in simulation, and leverages an existing congestion
Not under circumstances I can control. That's Not Science.
>controller that is available in Linux and Windows (with some tweaks).
SCE improves on an existing AQM that the default in roughly 100% of
linux deployments now, all the way up to the new RHEL8 release, which
was the last distro to adopt it, so far as I know. It doesn't require
a specialized tcp to use, either.
> SCE leverages a paragraph in a draft that describes a first guess about how a congestion controller might work.
We were kind of in a hurry. 'round here, we usually write the code and
test it thoroughly for years, before submitting a draft.
We did part of this work originally with the original definition of
ce_threshold (back in 2013?) when we too had fiddled with DCTCP,
before pacing and BBR had arrived.
Running code is up on several servers now and we hope to have a live
test over the internet at ietf. Maybe tcp-fu will land. Hopefully
tcp-prague will land.
>
>
>
> L4S has defined a congestion feedback mechanism so that these congestion signals can get back to the sender. SCE offers that “we’ll propose something later”.
>
>
>
> BBR currently does not listen to explicit congestion signals, but it could be updated to do so (for either SCE or L4S).
>
>
>
> -Greg
>
>
>
>
>
> From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "David P. Reed" <dpreed@deepplum.com>
> Date: Sunday, March 17, 2019 at 12:07 PM
> To: Vint Cerf <vint@google.com>
> Cc: bloat <bloat@lists.bufferbloat.net>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>
> Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
>
>
> Vint -
>
>
>
> BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
>
>
>
> It depends on getting reliable current congestion information via packet drops and/or ECN.
>
>
>
> So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
>
>
>
> THe cable guys are trying to get a "private" field in the IP header for their own use.
>
>
>
>
>
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net" <ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
>
> v
>
>
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com> wrote:
>
> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
> L4S has a much better possibility of actually getting deployment into the
> wider Internet packet-moving equipment than anything being talked about
> here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
> good, but it fits better into actual silicon and it's being proposed by
> people who actually have better channels into the people setting hard
> requirements.
>
> I suggest you consider joining them instead of opposing them.
>
>
> Hi Mikael,
>
> I agree it makes sense that fq_anything has issues when you're talking
> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
> makes better sense there.
>
> But fq_x makes great sense and provides real value for the uplink in a
> home, small office, coffee shop, etc. (if you run the final rate limit
> on the home side of the access link.) I'm thinking maybe there's a
> disconnect here driven by the different use cases for where AQMs can go.
>
> The thing is, each of these is the most likely congestion point at
> different times, and it's worthwhile for each of them to be able to
> AQM (and mark packets) under congestion.
>
> One of the several things that bothers me with L4S is that I've seen
> precious little concern over interfering with the ability for another
> different AQM in-path to mark packets, and because it changes the
> semantics of CE, you can't have both working at the same time unless
> they both do L4S.
>
> SCE needs a lot of details filled in, but it's so much cleaner that it
> seems to me there's reasonably obvious answers to all (or almost all) of
> those detail questions, and because the semantics are so much cleaner,
> it's much easier to tell it's non-harmful.
>
> <aside regarding="non-harmful">
> The point you raised in another thread about reordering is mostly
> well-taken, and a good counterpoint to the claim "non-harmful relative
> to L4S".
>
> To me it seems sad and dumb that switches ended up trying to make
> ordering guarantees at cost of switching performance, because if it's
> useful to put ordering in the switch, then it must be equally useful to
> put it in the receiver's NIC or OS.
>
> So why isn't it in all the receivers' NIC or OS (where it would render
> the switch's ordering efforts moot) instead of in all the switches?
>
> I'm guessing the answer is a competition trap for the switch vendors,
> plus "with ordering goes faster than without, when you benchmark the
> switch with typical load and current (non-RACK) receivers".
>
> If that's the case, it seems like the drive for a competitive advantage
> caused deployment of a packet ordering workaround in the wrong network
> location(s), out of a pure misalignment of incentives.
>
> RACK rates to fix that in the end, but a lot of damage is already done,
> and the L4S approach gives switches a flag that can double as proof that
> RACK is there on the receiver, so they can stop trying to order those
> packets.
>
> So point granted, I understand and agree there's a cost to abandoning
> that advantage.
> </aside>
>
> But as you also said so well in another thread, this is important. ("The
> last unicorn", IIRC.) How much does it matter if there's a feature that
> has value today, but only until RACK is widely deployed? If you were
> convinced RACK would roll out everywhere within 3 years and SCE would
> produce better results than L4S over the following 15 years, would that
> change your mind?
>
> It would for me, and that's why I'd like to see SCE explored before
> making a call. I think at its core, it provides the same thing L4S does
> (a high-fidelity explicit congestion signal for the sender), but with
> much cleaner semantics that can be incrementally added to congestion
> controls that people are already using.
>
> Granted, it still remains to be seen whether SCE in practice can match
> the results of L4S, and L4S was here first. But it seems to me L4S comes
> with some problems that have not yet been examined, and that are nicely
> dodged by a SCE-based approach.
>
> If L4S really is as good as they seem to think, I could imagine getting
> behind it, but I don't think that's proven yet. I'm not certain, but
> all the comparative analyses I remember seeing have been from more or
> less the same team, and I'm not convinced they don't have some
> misaligned incentives of their own.
>
> I understand a lot of work has gone into L4S, but this move to jump it
> from interesting experiment to de-facto standard without a more critical
> review that digs deeper into some of the potential deployment problems
> has me concerned.
>
> If it really does turn out to be good enough to be permanent, I'm not
> opposed to it, but I'm just not convinced that it's non-harmful, and my
> default position is that the cleaner solution is going to be better in
> the long run, if they can do the same job.
>
> It's not that I want it to be a fight, but I do want to end up with the
> best solution we can get. We only have the one internet.
>
> Just my 2c.
>
> -Jake
>
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
> --
>
> New postal address:
>
> Google
>
> 1875 Explorer Street, 10th Floor
>
> Reston, VA 20190
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 23:59 ` Dave Taht
@ 2019-03-20 10:17 ` Sebastian Moeller
0 siblings, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-20 10:17 UTC (permalink / raw)
To: Dave Täht; +Cc: Greg White, ecn-sane, tsvwg IETF list, bloat
Dear All,
> On Mar 20, 2019, at 00:59, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Tue, Mar 19, 2019 at 5:44 AM Greg White <g.white@cablelabs.com> wrote:
>> [...]
>> But, L4S has been demonstrated in real equipment and in simulation, and leverages an existing congestion
>
> Not under circumstances I can control. That's Not Science.
[...]
It would be great if the L4S project maybe could help kick-start independent testing, by creating an sharing two VMs one with the appropriate client side patches and one with a L4S aware AQM (probably curvy RED to avoid the patent issue, assuming the patent does not cover curvyRED). So that it is easier to "kick" the tiers in a way that tests what the L4S project considers compliant clients/AQM. Personally I am interested to see how robust and reliable the detection of non-L4S CE sources is and how well the L4S side of the AQM will tolerate CE-marked packets from non-L4S senders, or in other words how well the "isolation" works. And also how L4S endpoints will deal with SCE emitting AQMs on their path.
I admit that I have doubts that ECT(1), basically a single "constellation" of a 2-bit bitfield can serve as a replacement for a single independent bit in a single-bit bit-field, that seems required for real isolation of flows of different ECN-response types.
Best Regards
Sebastian
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-19 1:06 ` Bob Briscoe
2019-03-19 3:18 ` Dave Taht
@ 2019-03-20 19:04 ` Holland, Jake
2019-03-20 19:58 ` Stephen Hemminger
` (3 more replies)
1 sibling, 4 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-20 19:04 UTC (permalink / raw)
To: Bob Briscoe, David P. Reed, Vint Cerf; +Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 19173 bytes --]
Hi Bob & Greg,
I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.
However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).
With that said, I wondered whether either of you have any responses that
speak to these points:
1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.
I understand that l4s-id suggests the possibility of an alternate
scheme. However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en
2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.
As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput
https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.
But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.
Regards,
Jake
PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this. But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.
PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.
From: Bob Briscoe <ietf@bobbriscoe.net>
Date: 2019-03-18 at 18:07
To: "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
David,
On 17/03/2019 18:07, David P. Reed wrote:
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
THe cable guys are trying to get a "private" field in the IP header for their own use.
There is nothing private about this codepoint, and there never has been. Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
* A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
* Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.
o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the moment
(the developer wants to announce it themselves, so I have been asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out highlights.
All this is good healthy development of much lower latency for Internet technology.
I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
-----Original Message-----
From: "Vint Cerf" <vint@google.com><mailto:vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com><mailto:jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se><mailto:swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com><mailto:dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"<mailto:ecn-sane@lists.bufferbloat.net> <ecn-sane@lists.bufferbloat.net><mailto:ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net><mailto:bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
[-- Attachment #2: Type: text/html, Size: 32431 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 19:04 ` Holland, Jake
@ 2019-03-20 19:58 ` Stephen Hemminger
2019-03-20 20:05 ` Holland, Jake
[not found] ` <5C9296E1.4010703@erg.abdn.ac.uk>
` (2 subsequent siblings)
3 siblings, 1 reply; 105+ messages in thread
From: Stephen Hemminger @ 2019-03-20 19:58 UTC (permalink / raw)
To: Holland, Jake
Cc: Bob Briscoe, David P. Reed, Vint Cerf, tsvwg IETF list, bloat
On Wed, 20 Mar 2019 19:04:17 +0000
"Holland, Jake" <jholland@akamai.com> wrote:
> Hi Bob & Greg,
>
> I agree there has been a reasonably open conversation about the L4S
> work, and thanks for all your efforts to make it so.
>
> However, I think there's 2 senses in which "private" might be fair that
> I didn't see covered in your rebuttals (merging forks and including
> Greg's rebuttal by reference from here:
> https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
>
> Please note:
> I don't consider these senses of "private" a disqualifying argument
> against the use of L4S, though I do consider them costs that should be
> taken into account (and of course opinions may differ here).
>
> With that said, I wondered whether either of you have any responses that
> speak to these points:
>
>
> 1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
> patent-protected AQM scheduler.
>
> I understand that l4s-id suggests the possibility of an alternate
> scheme. However, comparing the MUSTs of the section 5 requirements
> with the claims made by the patent seems to leave no room for an
> alternate that would not infringe the patent claims, unless I'm missing
> something?
>
> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
> https://patents.google.com/patent/US20170019343A1/en
>
Has anyone done a detailed investigation for prior art?
The patent office does not do a good job of looking for prior art,
and the parties in the patent process are not motivated to look.
Other vendors often are not interested either because their own house of
cards built on patents of previous research might come falling down.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
[not found] ` <5C9296E1.4010703@erg.abdn.ac.uk>
@ 2019-03-20 20:00 ` Holland, Jake
2019-03-20 20:05 ` Jonathan Morton
1 sibling, 0 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-20 20:00 UTC (permalink / raw)
To: gorry; +Cc: Bob Briscoe, David P. Reed, Vint Cerf, tsvwg IETF list, bloat
On 2019-03-20, 12:40, "Gorry Fairhurst" <gorry@erg.abdn.ac.uk> wrote:
Concerning "Maximize Throughput", if you don't need scalability to very
high rates, then is your requirement met by TCP-like semantics, as in
TCP with SACK/loss or even better TCP with ABE/ECT(0)?
[JH] A problem with TCP with BE/ECT(0) is that it gets at most 1 ECE signal
per round-trip, so the kind of high-fidelity congestion response I hope to
see out of some upcoming SCE-echo proposal would be very welcome, especially
in BBR (or similar), as well as anything that uses slow-start.
I wonder .... if the intent is to scale to really high rates, then the
control loop delay for the congestion-controller becomes a limiting
issue, and in that case low-latency is necessary to safely climb the
rate to the high speed - and conversely to allow the controller to react
quickly when (or if) that overshoots a capacity bottleneck. In other
words, is scalable high throughput inseperable from low latency?
[JH] I agree lower latency, particularly including anything that avoids
buffer-bloat, is an important factor in returning a timely congestion
signal to the sender.
However, there may be a big difference in throughput between a CC that
allows for an increase of, say, 10-20ms, or +.5 base-RTT at a bottleneck,
vs. one that pushes back on anything above 1ms, especially when considering
paths with longer transit times.
In that sense of course a good bandwidth-maximizing approach benefits from
keeping latency low also, but perhaps with different thresholds.
Gorry
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 19:58 ` Stephen Hemminger
@ 2019-03-20 20:05 ` Holland, Jake
0 siblings, 0 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-20 20:05 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Bob Briscoe, David P. Reed, Vint Cerf, tsvwg IETF list, bloat
On 2019-03-20, 12:58, "Stephen Hemminger" <stephen@networkplumber.org> wrote:
Has anyone done a detailed investigation for prior art?
I don't know, but for serendipitous reasons I recently came across this,
which may be interesting to those who would be interested in digging
further on that question:
http://www.statslab.cam.ac.uk/~frank/PAPERS/tac.pdf
"On packet marking at priority queues"
R. J. Gibbens and F. P. Kelly
IEEE Transactions on Automatic Control 47 (2002) 1016-1020.
I wasn't sure if it was technically prior art or not, and I'm offering
no opinion, but it sounded similar to me in some significant ways.
HTH.
-Jake
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
[not found] ` <5C9296E1.4010703@erg.abdn.ac.uk>
2019-03-20 20:00 ` [Bloat] [tsvwg] " Holland, Jake
@ 2019-03-20 20:05 ` Jonathan Morton
2019-03-20 20:55 ` Greg White
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 20:05 UTC (permalink / raw)
To: gorry; +Cc: Holland, Jake, Vint Cerf, tsvwg IETF list, David P. Reed, bloat
> On 20 Mar, 2019, at 9:39 pm, Gorry Fairhurst <gorry@erg.abdn.ac.uk> wrote:
>
> Concerning "Maximize Throughput", if you don't need scalability to very high rates, then is your requirement met by TCP-like semantics, as in TCP with SACK/loss or even better TCP with ABE/ECT(0)?
My intention with "Maximise Throughput" is to support the bulk-transfer applications that TCP is commonly used for today. In Diffserv terminology, you may consider it equivalent to "Best Effort".
As far as I can see, L4S offers "Maximise Throughput" and "Minimise Latency" services, but not the other two.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 20:05 ` Jonathan Morton
@ 2019-03-20 20:55 ` Greg White
2019-03-20 22:12 ` Sebastian Moeller
2019-03-21 0:41 ` Holland, Jake
0 siblings, 2 replies; 105+ messages in thread
From: Greg White @ 2019-03-20 20:55 UTC (permalink / raw)
To: Jonathan Morton, gorry; +Cc: Vint Cerf, tsvwg IETF list, David P. Reed, bloat
In normal conditions, L4S offers "Maximize Throughput" + "Minimize Loss" + "Minimize Latency" all at once. It doesn't require an application to have to make that false choice (hence the name "Low Latency Low Loss Scalable throughput").
If an application would prefer to "Minimize Cost", then I suppose it could adjust its congestion control to be less aggressive (assuming it is congestion controlled). Also, as you point out the LEPHB could be an option as well.
What section 4.1 in the dualq draft is referring to is a case where the system needs to protect against unresponsive, overloading flows in the low latency queue. In that case something has to give (you can't ensure low latency & low loss to e.g. a 100 Mbps unresponsive flow arriving at a 50 Mbps bottleneck).
-Greg
On 3/20/19, 2:05 PM, "Bloat on behalf of Jonathan Morton" <bloat-bounces@lists.bufferbloat.net on behalf of chromatix99@gmail.com> wrote:
> On 20 Mar, 2019, at 9:39 pm, Gorry Fairhurst <gorry@erg.abdn.ac.uk> wrote:
>
> Concerning "Maximize Throughput", if you don't need scalability to very high rates, then is your requirement met by TCP-like semantics, as in TCP with SACK/loss or even better TCP with ABE/ECT(0)?
My intention with "Maximise Throughput" is to support the bulk-transfer applications that TCP is commonly used for today. In Diffserv terminology, you may consider it equivalent to "Best Effort".
As far as I can see, L4S offers "Maximise Throughput" and "Minimise Latency" services, but not the other two.
- Jonathan Morton
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 19:04 ` Holland, Jake
2019-03-20 19:58 ` Stephen Hemminger
[not found] ` <5C9296E1.4010703@erg.abdn.ac.uk>
@ 2019-03-20 21:48 ` Greg White
2019-03-20 21:56 ` Jonathan Morton
2019-03-20 22:38 ` Holland, Jake
2019-03-20 23:29 ` Bob Briscoe
3 siblings, 2 replies; 105+ messages in thread
From: Greg White @ 2019-03-20 21:48 UTC (permalink / raw)
To: Holland, Jake, Bob Briscoe, David P. Reed, Vint Cerf
Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 20794 bytes --]
I responded to point 2 separately. In response to point 1, the dual queue mechanism is not the only way to support L4S and TCP Prague. As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague. I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:
At dequeue:
If packet indicates ECT(1):
If sojourn_time > L4S_threshold:
Set CE*, and forward packet
Else:
Forward packet
Else:
Do all the normal CoDel stuff
In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will. But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including the ECN field in the flow hash would keep those packets separate.
*a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function
From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "Holland, Jake" <jholland@akamai.com>
Date: Wednesday, March 20, 2019 at 1:06 PM
To: Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Hi Bob & Greg,
I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.
However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).
With that said, I wondered whether either of you have any responses that
speak to these points:
1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.
I understand that l4s-id suggests the possibility of an alternate
scheme. However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en
2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.
As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput
https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.
But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.
Regards,
Jake
PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this. But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.
PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.
From: Bob Briscoe <ietf@bobbriscoe.net>
Date: 2019-03-18 at 18:07
To: "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
David,
On 17/03/2019 18:07, David P. Reed wrote:
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
THe cable guys are trying to get a "private" field in the IP header for their own use.
There is nothing private about this codepoint, and there never has been. Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
* A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
* Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.
o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the moment
(the developer wants to announce it themselves, so I have been asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out highlights.
All this is good healthy development of much lower latency for Internet technology.
I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
-----Original Message-----
From: "Vint Cerf" <vint@google.com><mailto:vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com><mailto:jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se><mailto:swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com><mailto:dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"<mailto:ecn-sane@lists.bufferbloat.net> <ecn-sane@lists.bufferbloat.net><mailto:ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net><mailto:bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
[-- Attachment #2: Type: text/html, Size: 39454 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 21:48 ` [Bloat] " Greg White
@ 2019-03-20 21:56 ` Jonathan Morton
2019-03-20 22:38 ` Holland, Jake
1 sibling, 0 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 21:56 UTC (permalink / raw)
To: Greg White
Cc: Holland, Jake, Bob Briscoe, David P. Reed, Vint Cerf,
tsvwg IETF list, bloat
> On 20 Mar, 2019, at 11:48 pm, Greg White <g.white@CableLabs.com> wrote:
>
> the dual queue mechanism is not the only way to support L4S and TCP Prague. As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague. I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:
>
> At dequeue:
> If packet indicates ECT(1):
> If sojourn_time > L4S_threshold:
> Set CE*, and forward packet
> Else:
> Forward packet
> Else:
> Do all the normal CoDel stuff
>
> In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will. But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including the ECN field in the flow hash would keep those packets separate.
>
> *a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function
The above looks remarkably similar to our proof-of-concept for an SCE middlebox. Essentially:
At dequeue:
Do all the normal Codel stuff
If packet (still) carries ECT(0):
If sojourn_time > SCE_threshold:
Set SCE
Forward packet
And yes, a ramp function between 0 and codel_target would work better. The above gives us something to test against with minimum hassle.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 20:55 ` Greg White
@ 2019-03-20 22:12 ` Sebastian Moeller
2019-03-20 22:31 ` Jonathan Morton
2019-03-21 0:41 ` Holland, Jake
1 sibling, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-20 22:12 UTC (permalink / raw)
To: ecn-sane; +Cc: Jonathan Morton, bloat
Hi ECN-sane,
I reduced the CC list as I do not expect this to further the discussion much, but since I feel I invested too much time reading the L4S RFCs and papers I still want to air this here to get some feedback.
> On Mar 20, 2019, at 21:55, Greg White <g.white@CableLabs.com> wrote:
>
> In normal conditions, L4S offers "Maximize Throughput" + "Minimize Loss" + "Minimize Latency" all at once. It doesn't require an application to have to make that false choice (hence the name "Low Latency Low Loss Scalable throughput").
>
> If an application would prefer to "Minimize Cost", then I suppose it could adjust its congestion control to be less aggressive (assuming it is congestion controlled). Also, as you point out the LEPHB could be an option as well.
>
> What section 4.1 in the dualq draft is referring to is a case where the system needs to protect against unresponsive, overloading flows in the low latency queue. In that case something has to give (you can't ensure low latency & low loss to e.g. a 100 Mbps unresponsive flow arriving at a 50 Mbps bottleneck).
Which somewhat puts the claim "ultra-low queueing latency" (see https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03) into perspective. IMHO, ultra low queueing latency is only going to happen in steady state if effectively pacing senders spread their sending rates such that packets arrive nicely spread out already. I note that with that kind of traffic pattern other AQ would also offer ultra-low queueing latency... I note that "‘Data Centre to the Home’: Ultra-Low Latency for All" states that they see 20ms queue delay with a 7ms base link delay @ 40 Mbps, back of the envelope calculations tell us that at that rate ((40 * 1000^2) / (1538 * 8)) * 0.02 = 65 packets will be paced out of the dualpi2 AQM add 65 more on the L4S side and queuing delay will be at 40ms. Switching to a pacing and less aggressive TCP version helps to smooth out the steady-stae bursts, but will do zilch for transients due to an increase in the number of active flows.
I wonder how this is going to behave once we have new flows come in at a high rate (at 3.251 KHz capacity adding loads at 100 Hz does not seem like that a heavy load to me especially given) (actually I don't wonder, the dual-AQM draft indicates that the queue will grow to 250ms and tail dropping will start). If I would be responsible at IETFI really would want to see some analysis of resistance against adversarial traffic patterns before going that route, especially in the light of the fuzzy classification vie ECT(1).
Best Regards
Sebastian
>
> -Greg
>
>
>
>
> On 3/20/19, 2:05 PM, "Bloat on behalf of Jonathan Morton" <bloat-bounces@lists.bufferbloat.net on behalf of chromatix99@gmail.com> wrote:
>
>> On 20 Mar, 2019, at 9:39 pm, Gorry Fairhurst <gorry@erg.abdn.ac.uk> wrote:
>>
>> Concerning "Maximize Throughput", if you don't need scalability to very high rates, then is your requirement met by TCP-like semantics, as in TCP with SACK/loss or even better TCP with ABE/ECT(0)?
>
> My intention with "Maximise Throughput" is to support the bulk-transfer applications that TCP is commonly used for today. In Diffserv terminology, you may consider it equivalent to "Best Effort".
>
> As far as I can see, L4S offers "Maximise Throughput" and "Minimise Latency" services, but not the other two.
>
> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 22:12 ` Sebastian Moeller
@ 2019-03-20 22:31 ` Jonathan Morton
2019-03-20 22:56 ` Sebastian Moeller
0 siblings, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 22:31 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: ecn-sane, bloat
> On 21 Mar, 2019, at 12:12 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> they see 20ms queue delay with a 7ms base link delay @ 40 Mbps
At 40Mbps you might as well be running Cake, and thereby getting 1ms inter-flow induced delay; an order of magnitude better. And we achieved that o a shoestring budget while they were submarining for a patent application.
If we're supposed to be impressed…
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 21:48 ` [Bloat] " Greg White
2019-03-20 21:56 ` Jonathan Morton
@ 2019-03-20 22:38 ` Holland, Jake
2019-03-20 22:56 ` Greg White
1 sibling, 1 reply; 105+ messages in thread
From: Holland, Jake @ 2019-03-20 22:38 UTC (permalink / raw)
To: Greg White; +Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 21845 bytes --]
Thanks Greg,
But wouldn’t this potentially violate at least one MUST from section 5.2 of l4s-id?
The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
be roughly proportional to the square of the likelihood that it would
have marked it if it had been an L4S packet (p_L)
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5.2
Maybe it depends on how far you stretch “roughly”, I guess... I’m not sure, but I’d imagine some realistic conditions could provide counterexamples, unless there’s some reason I’m missing that prevents it?
Regards,
Jake
From: Greg White <g.white@CableLabs.com>
Date: 2019-03-20 at 14:49
To: "Holland, Jake" <jholland@akamai.com>, Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
I responded to point 2 separately. In response to point 1, the dual queue mechanism is not the only way to support L4S and TCP Prague. As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague. I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:
At dequeue:
If packet indicates ECT(1):
If sojourn_time > L4S_threshold:
Set CE*, and forward packet
Else:
Forward packet
Else:
Do all the normal CoDel stuff
In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will. But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including the ECN field in the flow hash would keep those packets separate.
*a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function
From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "Holland, Jake" <jholland@akamai.com>
Date: Wednesday, March 20, 2019 at 1:06 PM
To: Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Hi Bob & Greg,
I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.
However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).
With that said, I wondered whether either of you have any responses that
speak to these points:
1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.
I understand that l4s-id suggests the possibility of an alternate
scheme. However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en
2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.
As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput
https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.
But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.
Regards,
Jake
PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this. But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.
PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.
From: Bob Briscoe <ietf@bobbriscoe.net>
Date: 2019-03-18 at 18:07
To: "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
David,
On 17/03/2019 18:07, David P. Reed wrote:
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
THe cable guys are trying to get a "private" field in the IP header for their own use.
There is nothing private about this codepoint, and there never has been. Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
* A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
* Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.
o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the moment
(the developer wants to announce it themselves, so I have been asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out highlights.
All this is good healthy development of much lower latency for Internet technology.
I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
-----Original Message-----
From: "Vint Cerf" <vint@google.com><mailto:vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com><mailto:jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se><mailto:swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com><mailto:dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"<mailto:ecn-sane@lists.bufferbloat.net> <ecn-sane@lists.bufferbloat.net><mailto:ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net><mailto:bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
[-- Attachment #2: Type: text/html, Size: 41243 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 22:31 ` Jonathan Morton
@ 2019-03-20 22:56 ` Sebastian Moeller
2019-03-20 23:03 ` Jonathan Morton
2019-03-20 23:11 ` Holland, Jake
0 siblings, 2 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-20 22:56 UTC (permalink / raw)
To: Jonathan Morton; +Cc: ecn-sane, bloat
> On Mar 20, 2019, at 23:31, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 21 Mar, 2019, at 12:12 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> they see 20ms queue delay with a 7ms base link delay @ 40 Mbps
>
> At 40Mbps you might as well be running Cake, and thereby getting 1ms inter-flow induced delay; an order of magnitude better. And we achieved that o a shoestring budget while they were submarining for a patent application.
>
> If we're supposed to be impressed…
Nah, there is this GEM:
Comparing Experiments 5, 7 with 6, 8, we can again conclude that our DualQ AQM very much approximates the fq CoDel AQM without the need for flow identi- fication and more complex processing. The main ad- vantage is DualQ’s lower queuing delay for L4S traffic.
So for normal traffic is is worse than fq_codel and better for traffic that does behave TCP-friendly, for which it was bespoke made. So at least they shoud have pimped fq_codel/cake to emit their required CE marking regime and do a test against that, if the goal is to compare apples and apples. I note that they do come into this with a grudge against fq "Per-flow queuing: Similarly per-flow queuing is not incompatible with the L4S approach. However, one queue for every flow can be thought of as overkill compared to the minimum of two queues for
all traffic needed for the L4S approach. The overkill of per-flow queuing has side-effects:" followed by a list of 4 more or less straw-man arguments. Heck these might be actually reasonable arguments at their core, but the short description in the RFC is fishy.
I believe the coupling between the two queues to be clever and elegant, but the whole premise seems odd to me. What they should have done, IMHO is teach their AQM something like SCE so it can easily react to CE and drops in a standard compliant TCP-friendly fashion, and only do the clever window/rate adjustments if the AQM signals ECT(1), add fair queueing to separate the different TCP variants behavior from each other, and bang no classification bit needed. And no patent (assuming the patent covers the coupling between the two queues)... I am sure I am missing something here, it can not be that simple.
Best Regards
Sebastian
P.S.: How did the SCE-Talk go, interesting feed-back and discussions?
>
> - Jonathan Morton
>
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 22:38 ` Holland, Jake
@ 2019-03-20 22:56 ` Greg White
0 siblings, 0 replies; 105+ messages in thread
From: Greg White @ 2019-03-20 22:56 UTC (permalink / raw)
To: Holland, Jake; +Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 22464 bytes --]
That is an interesting point. The goal of that MUST statement is to ensure per-flow-fairness. Since fq provides per-flow-fairness as a guarantee via DRR, there is no need to actively apply that rule, it should just be an emergent property of the scheduling.
-Greg
From: "Holland, Jake" <jholland@akamai.com>
Date: Wednesday, March 20, 2019 at 4:38 PM
To: Greg White <g.white@CableLabs.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Thanks Greg,
But wouldn’t this potentially violate at least one MUST from section 5.2 of l4s-id?
The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
be roughly proportional to the square of the likelihood that it would
have marked it if it had been an L4S packet (p_L)
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5.2
Maybe it depends on how far you stretch “roughly”, I guess... I’m not sure, but I’d imagine some realistic conditions could provide counterexamples, unless there’s some reason I’m missing that prevents it?
Regards,
Jake
From: Greg White <g.white@CableLabs.com>
Date: 2019-03-20 at 14:49
To: "Holland, Jake" <jholland@akamai.com>, Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
I responded to point 2 separately. In response to point 1, the dual queue mechanism is not the only way to support L4S and TCP Prague. As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague. I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:
At dequeue:
If packet indicates ECT(1):
If sojourn_time > L4S_threshold:
Set CE*, and forward packet
Else:
Forward packet
Else:
Do all the normal CoDel stuff
In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will. But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including the ECN field in the flow hash would keep those packets separate.
*a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function
From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of "Holland, Jake" <jholland@akamai.com>
Date: Wednesday, March 20, 2019 at 1:06 PM
To: Bob Briscoe <ietf@bobbriscoe.net>, "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
Hi Bob & Greg,
I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.
However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).
With that said, I wondered whether either of you have any responses that
speak to these points:
1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.
I understand that l4s-id suggests the possibility of an alternate
scheme. However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en
2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.
As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput
https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.
But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.
Regards,
Jake
PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this. But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.
PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.
From: Bob Briscoe <ietf@bobbriscoe.net>
Date: 2019-03-18 at 18:07
To: "David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
Cc: tsvwg IETF list <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
David,
On 17/03/2019 18:07, David P. Reed wrote:
Vint -
BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
THe cable guys are trying to get a "private" field in the IP header for their own use.
There is nothing private about this codepoint, and there never has been. Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).
* A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).
* Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.
o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the moment
(the developer wants to announce it themselves, so I have been asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out highlights.
All this is good healthy development of much lower latency for Internet technology.
I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
-----Original Message-----
From: "Vint Cerf" <vint@google.com><mailto:vint@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholland@akamai.com><mailto:jholland@akamai.com>
Cc: "Mikael Abrahamsson" <swmike@swm.pp.se><mailto:swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com><mailto:dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"<mailto:ecn-sane@lists.bufferbloat.net> <ecn-sane@lists.bufferbloat.net><mailto:ecn-sane@lists.bufferbloat.net>, "bloat" <bloat@lists.bufferbloat.net><mailto:bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com<mailto:jholland@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se<mailto:swmike@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting deployment into the
wider Internet packet-moving equipment than anything being talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
good, but it fits better into actual silicon and it's being proposed by
people who actually have better channels into the people setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the uplink in a
home, small office, coffee shop, etc. (if you run the final rate limit
on the home side of the access link.) I'm thinking maybe there's a
disconnect here driven by the different use cases for where AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that I've seen
precious little concern over interfering with the ability for another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner that it
seems to me there's reasonably obvious answers to all (or almost all) of
those detail questions, and because the semantics are so much cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because if it's
useful to put ordering in the switch, then it must be equally useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it would render
the switch's ordering efforts moot) instead of in all the switches?
I'm guessing the answer is a competition trap for the switch vendors,
plus "with ordering goes faster than without, when you benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive advantage
caused deployment of a packet ordering workaround in the wrong network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is already done,
and the L4S approach gives switches a flag that can double as proof that
RACK is there on the receiver, so they can stop trying to order those
packets.
So point granted, I understand and agree there's a cost to abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is important. ("The
last unicorn", IIRC.) How much does it matter if there's a feature that
has value today, but only until RACK is widely deployed? If you were
convinced RACK would roll out everywhere within 3 years and SCE would
produce better results than L4S over the following 15 years, would that
change your mind?
It would for me, and that's why I'd like to see SCE explored before
making a call. I think at its core, it provides the same thing L4S does
(a high-fidelity explicit congestion signal for the sender), but with
much cleaner semantics that can be incrementally added to congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice can match
the results of L4S, and L4S was here first. But it seems to me L4S comes
with some problems that have not yet been examined, and that are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could imagine getting
behind it, but I don't think that's proven yet. I'm not certain, but
all the comparative analyses I remember seeing have been from more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to jump it
from interesting experiment to de-facto standard without a more critical
review that digs deeper into some of the potential deployment problems
has me concerned.
If it really does turn out to be good enough to be permanent, I'm not
opposed to it, but I'm just not convinced that it's non-harmful, and my
default position is that the cleaner solution is going to be better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
Ecn-sane@lists.bufferbloat.net<mailto:Ecn-sane@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
[-- Attachment #2: Type: text/html, Size: 42851 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 22:56 ` Sebastian Moeller
@ 2019-03-20 23:03 ` Jonathan Morton
2019-03-20 23:11 ` Holland, Jake
1 sibling, 0 replies; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 23:03 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: ecn-sane, bloat
> On 21 Mar, 2019, at 12:56 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> What they should have done, IMHO is teach their AQM something like SCE so it can easily react to CE and drops in a standard compliant TCP-friendly fashion, and only do the clever window/rate adjustments if the AQM signals ECT(1)…
It's not even hard to teach DCTCP to respect SCE, as far as I can tell. All you have to do is start with a clean copy of NewReno with ABC, then subtract half of all SCE-marked bytes from the cwnd. That really is it.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 22:56 ` Sebastian Moeller
2019-03-20 23:03 ` Jonathan Morton
@ 2019-03-20 23:11 ` Holland, Jake
2019-03-20 23:28 ` Jonathan Morton
2019-03-20 23:30 ` [Bloat] [tsvwg] [Ecn-sane] " Sebastian Moeller
1 sibling, 2 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-20 23:11 UTC (permalink / raw)
To: Sebastian Moeller, Jonathan Morton; +Cc: ecn-sane, bloat
I think it's a fair point that even as close as the non-home side
of the access network, fq would need a lot of queues, and if you
want something in hardware it's going to be tricky. I hear
they're up to an average of ~6k homes per OLT.
I don't think the default assumption here should be that they
missed something obvious, but rather that they're trying to
solve a hard problem, and something with a classifier has a
legitimate value.
The question to me is about how much it breaks other things to
extract that value, and how much you get out of it in the end. If
you need fq and therefore the only viable place for AQM with good
results is on the home side of the router, that's got some bad
deployment problems too.
Just my 2c.
-Jake
On 2019-03-20, 15:56, "Sebastian Moeller" <moeller0@gmx.de> wrote:
> On Mar 20, 2019, at 23:31, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 21 Mar, 2019, at 12:12 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> they see 20ms queue delay with a 7ms base link delay @ 40 Mbps
>
> At 40Mbps you might as well be running Cake, and thereby getting 1ms inter-flow induced delay; an order of magnitude better. And we achieved that o a shoestring budget while they were submarining for a patent application.
>
> If we're supposed to be impressed…
Nah, there is this GEM:
Comparing Experiments 5, 7 with 6, 8, we can again conclude that our DualQ AQM very much approximates the fq CoDel AQM without the need for flow identi- fication and more complex processing. The main ad- vantage is DualQ’s lower queuing delay for L4S traffic.
So for normal traffic is is worse than fq_codel and better for traffic that does behave TCP-friendly, for which it was bespoke made. So at least they shoud have pimped fq_codel/cake to emit their required CE marking regime and do a test against that, if the goal is to compare apples and apples. I note that they do come into this with a grudge against fq "Per-flow queuing: Similarly per-flow queuing is not incompatible with the L4S approach. However, one queue for every flow can be thought of as overkill compared to the minimum of two queues for
all traffic needed for the L4S approach. The overkill of per-flow queuing has side-effects:" followed by a list of 4 more or less straw-man arguments. Heck these might be actually reasonable arguments at their core, but the short description in the RFC is fishy.
I believe the coupling between the two queues to be clever and elegant, but the whole premise seems odd to me. What they should have done, IMHO is teach their AQM something like SCE so it can easily react to CE and drops in a standard compliant TCP-friendly fashion, and only do the clever window/rate adjustments if the AQM signals ECT(1), add fair queueing to separate the different TCP variants behavior from each other, and bang no classification bit needed. And no patent (assuming the patent covers the coupling between the two queues)... I am sure I am missing something here, it can not be that simple.
Best Regards
Sebastian
P.S.: How did the SCE-Talk go, interesting feed-back and discussions?
>
> - Jonathan Morton
>
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:11 ` Holland, Jake
@ 2019-03-20 23:28 ` Jonathan Morton
2019-03-21 8:15 ` [Bloat] [Ecn-sane] [tsvwg] " Mikael Abrahamsson
2019-03-20 23:30 ` [Bloat] [tsvwg] [Ecn-sane] " Sebastian Moeller
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 23:28 UTC (permalink / raw)
To: Holland, Jake; +Cc: Sebastian Moeller, ecn-sane, bloat
> On 21 Mar, 2019, at 1:11 am, Holland, Jake <jholland@akamai.com> wrote:
>
> I think it's a fair point that even as close as the non-home side
> of the access network, fq would need a lot of queues, and if you
> want something in hardware it's going to be tricky. I hear
> they're up to an average of ~6k homes per OLT.
I think most of us would be relieved if competent single-queue AQMs became ubiquitous. It doesn't have to be Cake everywhere, just *something* unambiguously better than a dumb FIFO - and that's a rather low bar to clear.
Now, when you have a 6000-channel router, I expect it starts with some very simple and fast device to direct packets in *approximately* the right direction, where some other piece of hardware deals with some subset of the total port count. Let's say you have a 1Gbps+ link to a 64-line cluster. Immediately the problem of providing FQ on each of those lines is more tractable; you can give each one 1024 queues and AQMs, the same as fq_codel, for a quite manageable total of 65536, and it's not so hard to build hardware that can keep up with 1Gbps traffic these days.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 19:04 ` Holland, Jake
` (2 preceding siblings ...)
2019-03-20 21:48 ` [Bloat] " Greg White
@ 2019-03-20 23:29 ` Bob Briscoe
2019-03-20 23:51 ` Jonathan Morton
2019-03-24 20:15 ` alex.burr
3 siblings, 2 replies; 105+ messages in thread
From: Bob Briscoe @ 2019-03-20 23:29 UTC (permalink / raw)
To: Holland, Jake, David P. Reed, Vint Cerf; +Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 26323 bytes --]
Jake,
On 20/03/2019 19:04, Holland, Jake wrote:
>
> Hi Bob & Greg,
>
> I agree there has been a reasonably open conversation about the L4S
>
> work, and thanks for all your efforts to make it so.
>
> However, I think there's 2 senses in which "private" might be fair that
>
> I didn't see covered in your rebuttals (merging forks and including
>
> Greg's rebuttal by reference from here:
>
> https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
>
> Please note:
>
> I don't consider these senses of "private" a disqualifying argument
>
> against the use of L4S, though I do consider them costs that should be
>
> taken into account (and of course opinions may differ here).
>
Thanks for engaging in this.
I do also intend to address one or two other recurring questions on the
bloat list, but I have a lot on at the mo.
> With that said, I wondered whether either of you have any responses that
>
> speak to these points:
>
> 1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
>
> patent-protected AQM scheduler.
>
> I understand that l4s-id suggests the possibility of an alternate
>
> scheme. However, comparing the MUSTs of the section 5 requirements
>
> with the claims made by the patent seems to leave no room for an
>
> alternate that would not infringe the patent claims, unless I'm missing
>
> something?
>
> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
>
> https://patents.google.com/patent/US20170019343A1/en
>
1/ In 2016, I arranged for the hire of a patent attorney to undertake
the unusual step of filing a third party observation with the European
Patent Office. This went through Al-Lu's patent application claim by
claim pointing to prior art and giving the patent examiner all the
arguments to reject each claim. However, the examiner chose to take very
little note of it, which was disappointing and costly for us. The main
prior art is:
Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority
Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
The guys named as inventors in AL-Lu's filing published a paper on PI2
with me, in which we included a citation of this Gibbens paper as
inspiration for the coupling. The Gibbens paper was already cited as
background by other patents, so the EPO has it in their citation index.
The coupling was also based on my prior research with Mirja before I
started working with the guys from Al-Lu in the RITE European
Collaborative project. we had to go through a few rejections, but Mirja
and I finally got this work published in 2014 - still before the
priority date of the Al-Lu patent application:
Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using
Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom
Workshop on Telecommunications Standards: From Research to Standards
pp.583-588 (December 2014)
2/ The only claim that I could not find prior art for (in the original
EU filing) was a very specific claim about using a square root for the
coupling. The Linux implementation runs this the other way round so that
it only has to do a squaring. So I figured we were safe from that.
However, until just now, I had not noticed that Al-Lu has
retrospectively re-written the claims in the US patent and in the EU
patent application to claim this the other way round - as a squaring.
And to claim the two random number trick. Both restructuring to use a
squaring and the two random number trick were definitely my ideas (while
working for BT in a collaboration with Al-Lu). I have emails to prove
this (from memory they were actually both in the same email). This is
important, because a patent has to be about mechanism, not algorithm.
3/ This is a positive development. It means this patent is on very shaky
legal ground. I have been trying to put pressure on Nokia to license
this royalty free. But now I see what they have done, I am going to have
to get a different type of legal advice.
> 2. the L4S use of the ECT(1) codepoint privileges the low latency use
>
> case.
>
> As Jonathan Morton pointed out, low latency is only one of several
>
> known use cases that would be worthwhile to identify to an AQM
>
> scheduler, which the L4S approach cannot be extended to handle:
>
> - Minimize Latency
>
> - Minimize Loss
>
> - Minimize Cost
>
> - Maximize Throughput
>
> https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
>
> I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
>
> operator tuning parameters for a dualq node can possibly allow for
>
> tuning between minimizing loss and latency, as mentioned in section
>
> 4.1 of aqm-dualq, but since the signaling is bundled together, it can
>
> only happen for one at a time, if I'm reading it right.
>
Not at all. There is a reason why it's called Low Latency, Low Loss,
Scalable throughput (L4S).
The L4S definition of ECN marking addresses all four of these at once.
Strictly it directly addresses all except minimize cost, but minimize
cost can be built around the system. This was the subject of my PhD. I
haven't described L4S in these terms, because most people are only
interested in the latency. But this is the underlying reason for my
obsession with ECN.
Frank Kelly predicted that queuing delay would be removed from the
optimization as it was minimized. With L4S we've got very close to that.
ECN removes all congestion loss.
And the use of a inverse linear congestion controller gives the scalable
throughput. I shall be touching on this in my talk for netdev tomorrow,
but it's not really a subject for an implementation conference.
Minimize cost is something you do by combining the congestion signals
across a network. So any AQM is part of that. And congestion controllers
are the other part - they implicitly optimize cost, using the congestion
signals as shadow prices. The square root in classic TCP distorts this,
but DCTCP's inverse linear controller gives proportional fairness
directly. Without a weight term in the congestion controller, there is
not really an economic optimization, but that can be built onto a
proportionally fair system and competition will gradually cause that to
happen (or regulation as a proxy for competition). These are very long
term processes though.
> But more importantly, the L4S usage couples the minimized latency use
>
> case to any possibility of getting a high fidelity explicit congestion
>
> signal, so the "maximize throughput" use case can't ever get it.
>
Eh? There's definitely a misunderstanding or a difference in terminology
between us here. The whole point of using a congestion controller like
DCTCP is so that flow rate can scale indefinitely with capacity. Van
Jacobson actually noted that the original TCP was unscalable in a
footnote to the tech report version of the SIGCOMM paper.
The high fidelity congestion signal of what we call scalable congestion
controllers (like DCTCP) is inversely proportional to the window. So as
window scales up, the congestion signal scales down, so that their
product remains constant. That means the number of ECN marks per RTT is
scale-invariant. So the control signal remains just as tight at any scale.
Cheers
Bob
> Regards,
>
> Jake
>
> PS:
>
> If you get a chance, I'm still interested in seeing answers to my
>
> questions about deployment mitigations on the tsvwg list:
>
> https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
>
> I'm not surprised if it slipped by unnoticed, there have been a lot of
>
> emails on this. But good answers to those questions would go a long way
>
> toward easing my concerns about the urgency on this discussion.
>
> PPS:
>
> These issues are a bit sideways to my technical reasons for preferring
>
> the SCE formulation of ECT(1), which have more to do with the confusing
>
> semantics and proliferation of corner cases it creates for CE and ECE.
>
> But I thought I’d ask about them since it seemed like maybe there was a
>
> disconnect here.
>
> *From: *Bob Briscoe <ietf@bobbriscoe.net>
> *Date: *2019-03-18 at 18:07
> *To: *"David P. Reed" <dpreed@deepplum.com>, Vint Cerf <vint@google.com>
> *Cc: *tsvwg IETF list <tsvwg@ietf.org>, bloat
> <bloat@lists.bufferbloat.net>
> *Subject: *Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague]
> Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> David,
>
> On 17/03/2019 18:07, David P. Reed wrote:
>
> Vint -
>
> BBR is the end-to-end control logic that adjusts the source rate
> to match the share of the bolttleneck link it should use.
>
> It depends on getting reliable current congestion information via
> packet drops and/or ECN.
>
> So the proposal by these guys (not the cable guys) is an attempt
> to improve the quality of the congestion signal inserted by the
> router with the bottleneck outbound link.
>
> What do you mean 'not the cable guys'?
> This thread was reasonably civil until this intervention.
>
>
> THe cable guys are trying to get a "private" field in the IP
> header for their own use.
>
>
> There is nothing private about this codepoint, and there never has
> been. Here's some data points:
>
> * The IP header codepoint in question (ECT(1) in the ECN field) was
> proposed for use as an alternative ECN behaviour in July 2105 in the
> IETF AQM WG and the IETF's transport area WG (which handles all ECN
> matters).
> * A year later there followed a packed IETF BoF on the subject (after
> 2 open Bar BoFs).
> * Long discussion ensued on the merits of different IP header field
> combinations, on both these IETF lists, involving people active on
> this list (bloat), including Dave Taht, who is acknowledged for his
> contributions in the IETF draft.
> * That was when it was decided that ECT(1) was most appropriate.
> * The logic of the decision is written up in an appendix of
> draft-ietf-ecn-l4s-id.
> * David Black, one of the co-chairs of the IETF's transport area WG
> and co-author of both the original ECN and Diffserv RFCs, wrote
> RFC8311 to lay out the process for reclaiming and reusing the
> necessary codepoints.
> * This work and the process of freeing up codepoints has been very
> visible at every IETF ever since, with multiple drafts to fix other
> aspects of the protocols working their way through the IETF process in
> multiple WGs (tsvwg, tcpm, trill, etc).
> * L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
>
> Some history:
> * I had been researching the idea since 2012.
> * In fact my first presentation on it was scheduled directly after Van
> Jacobson's first presentation of CoDel at the IETF in July 2012. VJ
> overran by nearly 20mins leaving just 3 mins for my presentation.
> * Mirja Kuehlewind and I did early groundwork in 2013 and published a
> paper
> * Then I (working for BT) brought the work into the EU RITE project
> (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
> * By 2015 the two main L4S proponents were Koen De Schepper from
> Alcatel Lucent and myself (I had just switched from BT to Simula),
> along with Olga Bondarenko (now Albisser), a PhD student at Simula who
> now works for Microsoft (on something else) and is still doing her PhD
> part-time with Simula
> o By that time, Al-Lu and Simula had a cool prototype running.
> o This was demonstrated publicly for the first time in the IETF AQM
> WG over DC+core+backhaul+DSL+home networks.
> https://riteproject.eu/dctth/#1511dispatchwg
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
> * In May 2016, L4S was demonstrated at MultiMediaSystems'16 with
> /every/ packet from all the following simultaneous applications
> achieving ~1ms queuing delay or less over a 40Mb/s Internet access
> link (7ms base RTT):
> o cloud-rendered remote presence in a racing car within a VR headset
> o the interactive cloud-rendered video already linked above
> o an online gaming benchmark
> o HAS video streaming
> o a number of bulk file downloads
> o a heavy synthetic load of web browsing
>
> L4S has never been access-technology-specific. Indeed the cable
> industry has been funding my work at the IETF to help encourage a
> wider L4S ecosystem. There is nothing private to the cable industry in
> this:
> * Al-Lu used DSL as a use-case, but L4S was relevant to all the access
> technologies they supplied.
> * BT was obviously interested in DSL,
> * but BT's initial motivating use-case was to incrementally deploy the
> low queuing delay of DCTCP over BT's data centre interconnect networks.
> * In Jul 2016 the open-source Linux repo for the Coupled AQM was
> published, with a fully working version to be used and abused.
> Now at: https://github.com/L4STeam/sch_dualpi2_upstream
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
> * Of course, DCTCP was already open-sourced in Linux and FreeBSD, as
> well as available in Windows
> * In Jul 2016, the main IETF BoF on L4S was held:
> o Ingemar Johansson from Ericsson was one of the proponents, focused
> on using L4S in LTE
> o along with Kevin Smith from Vodafone and
> o Praveen Balasubramanian from Microsoft (who maintains the Windows
> TCP stack, including DCTCP).
> o Ingemar has since written an open-source L4S variant of the SCReAM
> congestion controller for real-time media:
> https://github.com/EricssonResearch/scream/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
> o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the
> necessary feedback in TCP https://github.com/mirjak/linux-accecn
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
> * In summer 2017 CableLabs started work on Low Latency DOCSIS, and
> hired me later in the year to help develop and specify it, along with
> support for L4S
> o In Jan 2019 the Low Latency DOCSIS spec was published and is now
> being implemented.
> o You can find the primary companies involved at the end of the
> White Paper.
> https://cablela.bs/low-latency-docsis-technology-overview-february-2019
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
> o Operators:
> Liberty Global
> Charter
> Rogers
> Comcast
> Shaw
> Cox Communications
> o Equipment vendors
> ARRIS
> Huawei
> Broadcom
> Intel
> Casa
> Nokia
> Cisco
> Videotron
> * Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
> * Magnus Westerlund of Ericsson with a team of others has written the
> necessary ECN feedback into QUIC
> * L4S hardware is also being implemented for hi-speed switches at the
> moment
> (the developer wants to announce it themselves, so I have been
> asked not to identify them).
>
> There's a lot more stuff been going on, but I've tried to pick out
> highlights.
>
> All this is good healthy development of much lower latency for
> Internet technology.
>
>
> I find it extremely disappointing that some people on this list are
> taking such a negative attitude to the major development in their own
> field that they seem not to have noticed since it first hit the
> limelight in 2015.
>
> L4S has been open-sourced since 2016 so that everyone can develop it
> and make it better...
>
> If I was in this position, having overlooked something important for
> multiple years, I would certainly not condemn it while I was trying to
> understand what it was and how it worked. Can I suggest everyone takes
> a step back, and suspends judgement until they have understood the
> potential, the goals and the depth of what they have missed. People
> who know me, know that I am very careful with Internet architecture,
> and particularly with balancing public policy against commercial
> issues. Please presume respect unless proven otherwise.
>
> Best Regards
>
>
>
> Bob
>
> PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get
> into the L4S queue. But only if it can keep latency down below around
> 1ms. Currently those ~15-25ms delay spikes would not pass muster.
> Indeed, its delay is not consistently low enough between the spikes
> either.
>
>
>
>
> -----Original Message-----
> From: "Vint Cerf" <vint@google.com> <mailto:vint@google.com>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <jholland@akamai.com> <mailto:jholland@akamai.com>
> Cc: "Mikael Abrahamsson" <swmike@swm.pp.se>
> <mailto:swmike@swm.pp.se>, "David P. Reed" <dpreed@deepplum.com>
> <mailto:dpreed@deepplum.com>, "ecn-sane@lists.bufferbloat.net"
> <mailto:ecn-sane@lists.bufferbloat.net>
> <ecn-sane@lists.bufferbloat.net>
> <mailto:ecn-sane@lists.bufferbloat.net>, "bloat"
> <bloat@lists.bufferbloat.net> <mailto:bloat@lists.bufferbloat.net>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague]
> Implementation and experimentation of TCP Prague/L4S hackaton at
> IETF104
>
> where does BBR fit into all this?
>
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholland@akamai.com
> <mailto:jholland@akamai.com>> wrote:
>
> On 2019-03-15, 11:37, "Mikael Abrahamsson" <swmike@swm.pp.se
> <mailto:swmike@swm.pp.se>> wrote:
> L4S has a much better possibility of actually getting
> deployment into the
> wider Internet packet-moving equipment than anything being
> talked about
> here. Same with PIE as opposed to FQ_CODEL. I know it's
> might not be as
> good, but it fits better into actual silicon and it's
> being proposed by
> people who actually have better channels into the people
> setting hard
> requirements.
>
> I suggest you consider joining them instead of opposing them.
>
>
> Hi Mikael,
>
> I agree it makes sense that fq_anything has issues when you're
> talking
> about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
> makes better sense there.
>
> But fq_x makes great sense and provides real value for the
> uplink in a
> home, small office, coffee shop, etc. (if you run the final
> rate limit
> on the home side of the access link.) I'm thinking maybe
> there's a
> disconnect here driven by the different use cases for where
> AQMs can go.
>
> The thing is, each of these is the most likely congestion point at
> different times, and it's worthwhile for each of them to be
> able to
> AQM (and mark packets) under congestion.
>
> One of the several things that bothers me with L4S is that
> I've seen
> precious little concern over interfering with the ability for
> another
> different AQM in-path to mark packets, and because it changes the
> semantics of CE, you can't have both working at the same time
> unless
> they both do L4S.
>
> SCE needs a lot of details filled in, but it's so much cleaner
> that it
> seems to me there's reasonably obvious answers to all (or
> almost all) of
> those detail questions, and because the semantics are so much
> cleaner,
> it's much easier to tell it's non-harmful.
>
> <aside regarding="non-harmful">
> The point you raised in another thread about reordering is mostly
> well-taken, and a good counterpoint to the claim "non-harmful
> relative
> to L4S".
>
> To me it seems sad and dumb that switches ended up trying to make
> ordering guarantees at cost of switching performance, because
> if it's
> useful to put ordering in the switch, then it must be equally
> useful to
> put it in the receiver's NIC or OS.
>
> So why isn't it in all the receivers' NIC or OS (where it
> would render
> the switch's ordering efforts moot) instead of in all the
> switches?
>
> I'm guessing the answer is a competition trap for the switch
> vendors,
> plus "with ordering goes faster than without, when you
> benchmark the
> switch with typical load and current (non-RACK) receivers".
>
> If that's the case, it seems like the drive for a competitive
> advantage
> caused deployment of a packet ordering workaround in the wrong
> network
> location(s), out of a pure misalignment of incentives.
>
> RACK rates to fix that in the end, but a lot of damage is
> already done,
> and the L4S approach gives switches a flag that can double as
> proof that
> RACK is there on the receiver, so they can stop trying to
> order those
> packets.
>
> So point granted, I understand and agree there's a cost to
> abandoning
> that advantage.
> </aside>
>
> But as you also said so well in another thread, this is
> important. ("The
> last unicorn", IIRC.) How much does it matter if there's a
> feature that
> has value today, but only until RACK is widely deployed? If
> you were
> convinced RACK would roll out everywhere within 3 years and
> SCE would
> produce better results than L4S over the following 15 years,
> would that
> change your mind?
>
> It would for me, and that's why I'd like to see SCE explored
> before
> making a call. I think at its core, it provides the same
> thing L4S does
> (a high-fidelity explicit congestion signal for the sender),
> but with
> much cleaner semantics that can be incrementally added to
> congestion
> controls that people are already using.
>
> Granted, it still remains to be seen whether SCE in practice
> can match
> the results of L4S, and L4S was here first. But it seems to
> me L4S comes
> with some problems that have not yet been examined, and that
> are nicely
> dodged by a SCE-based approach.
>
> If L4S really is as good as they seem to think, I could
> imagine getting
> behind it, but I don't think that's proven yet. I'm not
> certain, but
> all the comparative analyses I remember seeing have been from
> more or
> less the same team, and I'm not convinced they don't have some
> misaligned incentives of their own.
>
> I understand a lot of work has gone into L4S, but this move to
> jump it
> from interesting experiment to de-facto standard without a
> more critical
> review that digs deeper into some of the potential deployment
> problems
> has me concerned.
>
> If it really does turn out to be good enough to be permanent,
> I'm not
> opposed to it, but I'm just not convinced that it's
> non-harmful, and my
> default position is that the cleaner solution is going to be
> better in
> the long run, if they can do the same job.
>
> It's not that I want it to be a fight, but I do want to end up
> with the
> best solution we can get. We only have the one internet.
>
> Just my 2c.
>
> -Jake
>
>
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> <mailto:Ecn-sane@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/ecn-sane
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
>
>
> --
>
> New postal address:
>
> Google
>
> 1875 Explorer Street, 10th Floor
>
> Reston, VA 20190
>
>
>
> _______________________________________________
>
> Bloat mailing list
>
> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>
> https://lists.bufferbloat.net/listinfo/bloat <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
>
>
>
> --
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
[-- Attachment #2: Type: text/html, Size: 47963 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:11 ` Holland, Jake
2019-03-20 23:28 ` Jonathan Morton
@ 2019-03-20 23:30 ` Sebastian Moeller
2019-03-21 0:15 ` Holland, Jake
1 sibling, 1 reply; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-20 23:30 UTC (permalink / raw)
To: Holland, Jake; +Cc: Jonathan Morton, ecn-sane, bloat
Hi Jake,
> On Mar 21, 2019, at 00:11, Holland, Jake <jholland@akamai.com> wrote:
>
> I think it's a fair point that even as close as the non-home side
> of the access network, fq would need a lot of queues, and if you
> want something in hardware it's going to be tricky. I hear
> they're up to an average of ~6k homes per OLT.
Except they state "In practice it would also be important to de- ploy AQM in the residential gateway, but to minimise side-effects we kept upstream traffic below capacity." meaning in addition to the OLT/BNG/whatever shaper they also envision a shaper on the CPE. And I believe there is ample evidence (in openwrt with sqm-scripts) that in that case the downstream shaper can also be put on the CPE with reasonable success.
>
> I don't think the default assumption here should be that they
> missed something obvious, but rather that they're trying to
> solve a hard problem, and something with a classifier has a
> legitimate value.
I agree, except ECT(1) clearly is a very approximate "classifier" as it can not distinguish the L4S-ness of CE marked packets, which affects both the AQM part which will treat non-L4S traffic as false positive as well as TCP Prague endpoints that will mistreat CE-marked packets as L4S signals even if the CE mark is from a TCP-friendly AQM. I note that neither "‘Data Centre to the Home’: Ultra-Low Latency for All" nor "PI2: A Linearized AQM for both Classic and Scalable TCP" seem to discuss these classification errors and their effects on real traffic in sufficient depth.
It is one thing to soak of one of the last few available "codepoints" in the IP headers, but it is another in my book to do so and not reliably being able to extract the encoded information. At least from my layman's perspective I wonder why this does not seem to bother anybody here?
>
> The question to me is about how much it breaks other things to
> extract that value, and how much you get out of it in the end.
That is basically the core of my question above, how much do you get out in the end?
> If you need fq and therefore the only viable place for AQM with good
> results is on the home side of the router, that's got some bad
> deployment problems too.
As I state above, even the L4S project position seems to be that AQM on the CPE/router is essential, so we are only haggling about how much AQM needs to be done on the router. But from that perspective, I would not be unhappy if my ISP would employ a lower latency AQM solution upstream of my router than they currently do, sort of as a belt and suspender approach to have my router's back in cases of severe packet inrush.
Best Regards
Sebastian
>
> Just my 2c.
>
> -Jake
>
> On 2019-03-20, 15:56, "Sebastian Moeller" <moeller0@gmx.de> wrote:
>
>
>
>> On Mar 20, 2019, at 23:31, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>> On 21 Mar, 2019, at 12:12 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>
>>> they see 20ms queue delay with a 7ms base link delay @ 40 Mbps
>>
>> At 40Mbps you might as well be running Cake, and thereby getting 1ms inter-flow induced delay; an order of magnitude better. And we achieved that o a shoestring budget while they were submarining for a patent application.
>>
>> If we're supposed to be impressed…
>
> Nah, there is this GEM:
>
>
> Comparing Experiments 5, 7 with 6, 8, we can again conclude that our DualQ AQM very much approximates the fq CoDel AQM without the need for flow identi- fication and more complex processing. The main ad- vantage is DualQ’s lower queuing delay for L4S traffic.
>
> So for normal traffic is is worse than fq_codel and better for traffic that does behave TCP-friendly, for which it was bespoke made. So at least they shoud have pimped fq_codel/cake to emit their required CE marking regime and do a test against that, if the goal is to compare apples and apples. I note that they do come into this with a grudge against fq "Per-flow queuing: Similarly per-flow queuing is not incompatible with the L4S approach. However, one queue for every flow can be thought of as overkill compared to the minimum of two queues for
> all traffic needed for the L4S approach. The overkill of per-flow queuing has side-effects:" followed by a list of 4 more or less straw-man arguments. Heck these might be actually reasonable arguments at their core, but the short description in the RFC is fishy.
> I believe the coupling between the two queues to be clever and elegant, but the whole premise seems odd to me. What they should have done, IMHO is teach their AQM something like SCE so it can easily react to CE and drops in a standard compliant TCP-friendly fashion, and only do the clever window/rate adjustments if the AQM signals ECT(1), add fair queueing to separate the different TCP variants behavior from each other, and bang no classification bit needed. And no patent (assuming the patent covers the coupling between the two queues)... I am sure I am missing something here, it can not be that simple.
>
>
> Best Regards
> Sebastian
>
> P.S.: How did the SCE-Talk go, interesting feed-back and discussions?
>
>
>
>>
>> - Jonathan Morton
>>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:29 ` Bob Briscoe
@ 2019-03-20 23:51 ` Jonathan Morton
2019-03-21 6:04 ` Bob Briscoe
2019-03-24 20:15 ` alex.burr
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-20 23:51 UTC (permalink / raw)
To: Bob Briscoe
Cc: Holland, Jake, David P. Reed, Vint Cerf, tsvwg IETF list, bloat
> On 21 Mar, 2019, at 1:29 am, Bob Briscoe <ietf@bobbriscoe.net> wrote:
>
>> But more importantly, the L4S usage couples the minimized latency use
>> case to any possibility of getting a high fidelity explicit congestion
>> signal, so the "maximize throughput" use case can't ever get it.
> Eh? There's definitely a misunderstanding or a difference in terminology between us here. The whole point of using a congestion controller like DCTCP is so that flow rate can scale indefinitely with capacity. Van Jacobson actually noted that the original TCP was unscalable in a footnote to the tech report version of the SIGCOMM paper.
>
> The high fidelity congestion signal of what we call scalable congestion controllers (like DCTCP) is inversely proportional to the window. So as window scales up, the congestion signal scales down, so that their product remains constant. That means the number of ECN marks per RTT is scale-invariant. So the control signal remains just as tight at any scale.
If you'll indulge me for a moment, I'd like to lay out a compromise scenario where a lot of L4S' stated goals are still met.
There is no dualQ. There is an AQM at the bottleneck link, of unspecified type, which implements SCE. Assume that it produces CE marks like a conventional AQM, and also produces SCE marks like an L4S AQM produces CE.
A sender implements DCTCP-SCE, which is essentially Paced NewReno modified to subtract half of all acked data that was SCE-marked from its cwnd. (This is equivalent to the DCTCP algorithm with g=1 and an arbitrarily small measurement window, but acting on SCE instead of CE.) Any SCE mark also kicks it out of slow-start.
The means by which SCE information gets back to the sender is left vague for now; it's an orthogonal problem with several viable solutions.
What is missing from this scenario, from L4S' point of view? And why have I been able to describe it so succinctly?
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:30 ` [Bloat] [tsvwg] [Ecn-sane] " Sebastian Moeller
@ 2019-03-21 0:15 ` Holland, Jake
0 siblings, 0 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-21 0:15 UTC (permalink / raw)
To: Sebastian Moeller; +Cc: Jonathan Morton, ecn-sane, bloat
Hi Sebastian,
+1 on "approximate classifier" and CE. This does have me worried,
particularly in the case of multiple AQMs inline.
I think we mostly agree here, and my default position is still
that because SCE is cleaner, I like it better unless L4S can
demonstrate significant value over SCE.
I just wanted to make the point that there's a real chance
L4S might be able to demonstrate significant value that's much
harder to get in some other way (like DSCP), particularly in a
core or closer-to-core device where fq is impractical.
I basically was responding just because I felt like the dismissal
was premature, and seemed to rest on fq being practical in all
AQM cases, which I think is also unproven. (Even if the OLT can
be solved, I'm willing to stipulate there might be relevant
devices where fq is impractical.)
But I don't mean to argue against the conclusion that SCE seems
more likely to cause fewer problems. Just that it might be useful
to avoid dismissing legitimate points in L4S's favor.
Best Regards,
Jake
On 2019-03-20, 16:31, "Sebastian Moeller" <moeller0@gmx.de> wrote:
Hi Jake,
> On Mar 21, 2019, at 00:11, Holland, Jake <jholland@akamai.com> wrote:
>
> I think it's a fair point that even as close as the non-home side
> of the access network, fq would need a lot of queues, and if you
> want something in hardware it's going to be tricky. I hear
> they're up to an average of ~6k homes per OLT.
Except they state "In practice it would also be important to de- ploy AQM in the residential gateway, but to minimise side-effects we kept upstream traffic below capacity." meaning in addition to the OLT/BNG/whatever shaper they also envision a shaper on the CPE. And I believe there is ample evidence (in openwrt with sqm-scripts) that in that case the downstream shaper can also be put on the CPE with reasonable success.
>
> I don't think the default assumption here should be that they
> missed something obvious, but rather that they're trying to
> solve a hard problem, and something with a classifier has a
> legitimate value.
I agree, except ECT(1) clearly is a very approximate "classifier" as it can not distinguish the L4S-ness of CE marked packets, which affects both the AQM part which will treat non-L4S traffic as false positive as well as TCP Prague endpoints that will mistreat CE-marked packets as L4S signals even if the CE mark is from a TCP-friendly AQM. I note that neither "‘Data Centre to the Home’: Ultra-Low Latency for All" nor "PI2: A Linearized AQM for both Classic and Scalable TCP" seem to discuss these classification errors and their effects on real traffic in sufficient depth.
It is one thing to soak of one of the last few available "codepoints" in the IP headers, but it is another in my book to do so and not reliably being able to extract the encoded information. At least from my layman's perspective I wonder why this does not seem to bother anybody here?
>
> The question to me is about how much it breaks other things to
> extract that value, and how much you get out of it in the end.
That is basically the core of my question above, how much do you get out in the end?
> If you need fq and therefore the only viable place for AQM with good
> results is on the home side of the router, that's got some bad
> deployment problems too.
As I state above, even the L4S project position seems to be that AQM on the CPE/router is essential, so we are only haggling about how much AQM needs to be done on the router. But from that perspective, I would not be unhappy if my ISP would employ a lower latency AQM solution upstream of my router than they currently do, sort of as a belt and suspender approach to have my router's back in cases of severe packet inrush.
Best Regards
Sebastian
>
> Just my 2c.
>
> -Jake
>
> On 2019-03-20, 15:56, "Sebastian Moeller" <moeller0@gmx.de> wrote:
>
>
>
>> On Mar 20, 2019, at 23:31, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>> On 21 Mar, 2019, at 12:12 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>
>>> they see 20ms queue delay with a 7ms base link delay @ 40 Mbps
>>
>> At 40Mbps you might as well be running Cake, and thereby getting 1ms inter-flow induced delay; an order of magnitude better. And we achieved that o a shoestring budget while they were submarining for a patent application.
>>
>> If we're supposed to be impressed…
>
> Nah, there is this GEM:
>
>
> Comparing Experiments 5, 7 with 6, 8, we can again conclude that our DualQ AQM very much approximates the fq CoDel AQM without the need for flow identi- fication and more complex processing. The main ad- vantage is DualQ’s lower queuing delay for L4S traffic.
>
> So for normal traffic is is worse than fq_codel and better for traffic that does behave TCP-friendly, for which it was bespoke made. So at least they shoud have pimped fq_codel/cake to emit their required CE marking regime and do a test against that, if the goal is to compare apples and apples. I note that they do come into this with a grudge against fq "Per-flow queuing: Similarly per-flow queuing is not incompatible with the L4S approach. However, one queue for every flow can be thought of as overkill compared to the minimum of two queues for
> all traffic needed for the L4S approach. The overkill of per-flow queuing has side-effects:" followed by a list of 4 more or less straw-man arguments. Heck these might be actually reasonable arguments at their core, but the short description in the RFC is fishy.
> I believe the coupling between the two queues to be clever and elegant, but the whole premise seems odd to me. What they should have done, IMHO is teach their AQM something like SCE so it can easily react to CE and drops in a standard compliant TCP-friendly fashion, and only do the clever window/rate adjustments if the AQM signals ECT(1), add fair queueing to separate the different TCP variants behavior from each other, and bang no classification bit needed. And no patent (assuming the patent covers the coupling between the two queues)... I am sure I am missing something here, it can not be that simple.
>
>
> Best Regards
> Sebastian
>
> P.S.: How did the SCE-Talk go, interesting feed-back and discussions?
>
>
>
>>
>> - Jonathan Morton
>>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [tsvwg] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 20:55 ` Greg White
2019-03-20 22:12 ` Sebastian Moeller
@ 2019-03-21 0:41 ` Holland, Jake
1 sibling, 0 replies; 105+ messages in thread
From: Holland, Jake @ 2019-03-21 0:41 UTC (permalink / raw)
To: Greg White; +Cc: tsvwg IETF list, bloat
Hi Greg,
On 2019-03-20, 13:55, "Greg White" <g.white@CableLabs.com> wrote:
In normal conditions, L4S offers "Maximize Throughput" + "Minimize Loss" + "Minimize Latency" all at once. It doesn't require an application to have to make that false choice (hence the name "Low Latency Low Loss Scalable throughput").
[JH] This is an interesting claim, and I'm eager to see
how well it holds up under scrutiny and deployment.
I guess I'm not sure what exactly "normal" means, but I
would expect that there are a lot of cases that occur
frequently in practice where tradeoffs have to be made
between throughput, loss, and latency.
I'm finding I struggle to nail down exactly what I expect
from scenarios like a short-RTT L4S flow competing with a
long-RTT L4S flow (from transit delay) and with a BBR flow,
and likewise when a short and a long RTT L4S flow are
competing with a bunch of independent slow-start flows,
but if the L4S cases do indeed get a better throughput than
SCE-based approaches under the wide variety of situations
normal internet use can fall into, I think that would
convince me it's optimizing all of them at once, and it's
a mistake to call it focused on the latency use case.
But for now, I hope you'll forgive a little bit of
skepticism... I find this stuff complicated, and it's hard
for me to put high confidence on some of the predictions.
Best regards,
Jake
If an application would prefer to "Minimize Cost", then I suppose it could adjust its congestion control to be less aggressive (assuming it is congestion controlled). Also, as you point out the LEPHB could be an option as well.
What section 4.1 in the dualq draft is referring to is a case where the system needs to protect against unresponsive, overloading flows in the low latency queue. In that case something has to give (you can't ensure low latency & low loss to e.g. a 100 Mbps unresponsive flow arriving at a 50 Mbps bottleneck).
-Greg
On 3/20/19, 2:05 PM, "Bloat on behalf of Jonathan Morton" <bloat-bounces@lists.bufferbloat.net on behalf of chromatix99@gmail.com> wrote:
> On 20 Mar, 2019, at 9:39 pm, Gorry Fairhurst <gorry@erg.abdn.ac.uk> wrote:
>
> Concerning "Maximize Throughput", if you don't need scalability to very high rates, then is your requirement met by TCP-like semantics, as in TCP with SACK/loss or even better TCP with ABE/ECT(0)?
My intention with "Maximise Throughput" is to support the bulk-transfer applications that TCP is commonly used for today. In Diffserv terminology, you may consider it equivalent to "Best Effort".
As far as I can see, L4S offers "Maximise Throughput" and "Minimise Latency" services, but not the other two.
- Jonathan Morton
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:51 ` Jonathan Morton
@ 2019-03-21 6:04 ` Bob Briscoe
2019-03-21 7:46 ` Jonathan Morton
2019-03-21 8:45 ` Sebastian Moeller
0 siblings, 2 replies; 105+ messages in thread
From: Bob Briscoe @ 2019-03-21 6:04 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Holland, Jake, tsvwg IETF list, bloat
Jonathan,
On 20/03/2019 23:51, Jonathan Morton wrote:
>> On 21 Mar, 2019, at 1:29 am, Bob Briscoe <ietf@bobbriscoe.net> wrote:
>>
>>> But more importantly, the L4S usage couples the minimized latency use
>>> case to any possibility of getting a high fidelity explicit congestion
>>> signal, so the "maximize throughput" use case can't ever get it.
>> Eh? There's definitely a misunderstanding or a difference in terminology between us here. The whole point of using a congestion controller like DCTCP is so that flow rate can scale indefinitely with capacity. Van Jacobson actually noted that the original TCP was unscalable in a footnote to the tech report version of the SIGCOMM paper.
>>
>> The high fidelity congestion signal of what we call scalable congestion controllers (like DCTCP) is inversely proportional to the window. So as window scales up, the congestion signal scales down, so that their product remains constant. That means the number of ECN marks per RTT is scale-invariant. So the control signal remains just as tight at any scale.
> If you'll indulge me for a moment, I'd like to lay out a compromise scenario where a lot of L4S' stated goals are still met.
>
> There is no dualQ. There is an AQM at the bottleneck link, of unspecified type, which implements SCE. Assume that it produces CE marks like a conventional AQM, and also produces SCE marks like an L4S AQM produces CE.
>
> A sender implements DCTCP-SCE, which is essentially Paced NewReno modified to subtract half of all acked data that was SCE-marked from its cwnd. (This is equivalent to the DCTCP algorithm with g=1 and an arbitrarily small measurement window, but acting on SCE instead of CE.) Any SCE mark also kicks it out of slow-start.
>
> The means by which SCE information gets back to the sender is left vague for now; it's an orthogonal problem with several viable solutions.
>
> What is missing from this scenario, from L4S' point of view? And why have I been able to describe it so succinctly?
My goal is also to tighten the EWMA parameter, g, in DCTCP to 1 (or 2).
That is why we have recommended a queuing-time-based ramp AQM for the
Low Latency queue, which so far works equivalently to the step with g
set to its current default of 1/16. We have been doing experiments on
this for some time. But it is important to assess each change one at a
time.
Congestion controls are tricky to get stable in all situations. So it is
important to separate ideas and research from engineering of more mature
approaches that are ready for more widespread experimentation on the
public Internet. Our goal with L4S was to use proven algorithms, and put
in place mechanism to allow those algorithms to evolve.
As regards the desire to use SCE instead of the L4S approach of using a
classifier, please answer all the reasons I gave for why that won't
work, which I sent in response to your draft some days ago. The main one
is incremental deployment: the source does not identify its packets as
distinct from others, so the source needs the network to use some other
identifier if it wants the network to put it in a queue with latency
that is isolated from packets not using the scheme. The only way I can
see to so this would be to use per-flow-queuing. I think that is an
unstated assumption of SCE.
In contrast, L4S works with either per-flow queuing or dualQ, so it is
more appropriate for a wider spread of scenarios. Again, in that same
unanswered email, I described a way L4S can use per-flow queuing, and
Greg has since given pseudocode. There are other problems with doing the
codepoints the SCE way round - pls see that email.
There has been a general statement that the SCE way round is purer.
However, that concept is in the eye of the beholder. The SCE way round
does not allow the ECN field to be used as a classifier, so you don't
get the benefit above about support for per-flow-queueing and dual
queue. You also don't get the benefit of being able to relax
resequencing in the network, because the network has no classifier to
look at. For these, the SCE codepoint would need to be combined with a
DSCP, and I assume you don't want to do that.
The L4S way round signifies an alternative meaning of the ECN field,
which is exactly what it is. The problem of having to guess which type
of packet a CE used to be has been roundly discussed at the IETF in TCPM
and TSVWG WGs and it has been decided it is a non-problem if it is
assumed to have been ECT(1) even if it was not - as written up in
draft-ietf-ecn-l4s-id. And that discussion assumed TCP with 3DupACK
reordering tolerance, not the more liberal use of RACK (or a RACK-like
approach in other transports). With a RACK-like approach, it becomes
even less of a problem.
Bob
> - Jonathan Morton
>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 6:04 ` Bob Briscoe
@ 2019-03-21 7:46 ` Jonathan Morton
2019-03-21 8:02 ` Bob Briscoe
2019-03-21 8:45 ` Sebastian Moeller
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-21 7:46 UTC (permalink / raw)
To: Bob Briscoe; +Cc: Holland, Jake, tsvwg IETF list, bloat
> On 21 Mar, 2019, at 8:04 am, Bob Briscoe <ietf@bobbriscoe.net> wrote:
> Congestion controls are tricky to get stable in all situations. So it is important to separate ideas and research from engineering of more mature approaches that are ready for more widespread experimentation on the public Internet. Our goal with L4S was to use proven algorithms, and put in place mechanism to allow those algorithms to evolve.
I hope that from my example, you can see how to adapt a more flexible and "mature" version of DCTCP to use SCE. You should be able to use the same algorithms that you've worked so hard on; only the signalling method changes, and the trigger for falling back to Classic ECN behaviour is explicit (a plain old CE mark).
As for "proven algorithms", it was conclusively proven that DCTCP was *not* compatible with Classic ECN middleboxes, and had only been proven to work in tightly controlled environments. I am told that TCP Prague has a failsafe, but I do not yet understand how that failsafe works, and what I have been told sounds fragile. I am honestly perplexed that no explanation of this is forthcoming.
SCE works transparently with every deployed and proven congestion control algorithm out there, which simply ignores the information SCE provides. Adaptations of some of those algorithms to incorporate SCE information seem to be straightforward to implement, especially since ns-3 now supports AccECN, so initial full-system experiments should be forthcoming quite soon. We should even be able to rehabilitate DCTCP without resorting to failsafe workarounds - which *should* have you guys jumping for joy, in theory.
> As regards the desire to use SCE instead of the L4S approach of using a classifier, please answer all the reasons I gave for why that won't work, which I sent in response to your draft some days ago.
I'm afraid that must have got lost in the noise. There *was* a lot of noise; it gave me a headache.
Regardless, I haven't seen any real claims that SCE won't work, except for some quibbles about RTT-fair convergence with single queues, which I subsequently found an elegant way to address. We do have a bit of a publication bottleneck over here at the moment; limited manpower.
I have mainly seen claims that SCE isn't a one-for-one replacement for L4S using exactly the same mechanisms and infrastructure as L4S does. Which is true, but unhelpful, because that would make SCE literally identical to L4S with no advantages of its own. I'm willing to point out ways to implement L4S' goals using SCE; see below.
> The main one is incremental deployment: the source does not identify its packets as distinct from others, so the source needs the network to use some other identifier if it wants the network to put it in a queue with latency that is isolated from packets not using the scheme. The only way I can see to so this would be to use per-flow-queuing. I think that is an unstated assumption of SCE.
Strictly minimising latency for the individual flow, in the face of competing non-SCE traffic sharing a single queue, is not a goal of SCE per se; I consider it an orthogonal problem which is better addressed by existing solutions. Coexisting with existing endpoints, existing traffic and existing middleboxes is paramount, and forms our main argument for incremental deployability.
Solutions already available include FQ and Diffserv. I'll grant you that FQ is easier to implement at lowish speeds, where a cheap CPU can be loaded with flexible software to do the job. You appear to be more focused on relatively high link capacities, as that is your main argument against FQ. I'll just note in passing that good FQ can extract a lot of responsiveness from relatively low-capacity links.
Diffserv is widely deployed (in terms of hardware capabilities) and should be a natural fit for distinguishing classes of traffic from each other. It is rarely used by applications because the networks tend to corrupt it in transit, and rarely make good use of the information into the bargain. It strikes me that the cable industry may have more influence over that than I do.
> The SCE way round does not allow the ECN field to be used as a classifier…
The ECN field was never intended to be used as a classifier, except to distinguish Not-ECT flows from ECT flows (which a middlebox does need to know, to choose between mark and drop behaviours). It was intended to be used to convey congestion information from the network to the receiver. SCE adheres to that ideal.
There is a perfectly good and under-utilised 6-bit field for carrying classifier information, right there in the same byte as the ECN field. You might want to ask the LE PHB guys for advice on making good use of it.
> You also don't get the benefit of being able to relax resequencing in the network, because the network has no classifier to look at.
My position is that the network is already free to relax resequencing semantics, regardless of the traffic carried. IP does not guarantee anything about packet ordering, and protocols built on top of it have always had to cope with that, one way or another.
Wifi's head-of-line blocking while performing link-level retries can induce inter-flow coupled delays of many seconds in extreme cases, destroying reliability completely. Recent work already reduces the effort the Linux wifi stack puts into link-level retries, given that most transports and protocols can survive some level of random loss. This is done without relying on any classifier, because it benefits all traffic.
On high-capacity bonded links, the likelihood that two packets sent near-simultaneously on different component links, and consequently reordered, both belong to the same flow and will trigger a spurious retransmission, seems to be low enough to not care about, even with existing 3-dupack sensitive TCPs. Therefore, relaxing resequencing requirements on these links should already be safe. Perhaps you have hard data showing otherwise?
> …the SCE codepoint would need to be combined with a DSCP, and I assume you don't want to do that.
The SCE codepoint does not need to be combined with a DSCP. Whether or not a DSCP assignment fits a given application is completely orthogonal to SCE.
You could quite reasonably implement something that looks very like DualQ using a trivial DSCP classifier instead of an ECN-based classifier, and that would be absolutely fine, and it would work with SCE. It's just not necessary to make SCE work in the first place.
Meanwhile, I still have not seen a detailed answer as to how, precisely, TCP Prague reliably distinguishes a Classic ECN middlebox from an L4S one, in order to activate its failsafe mechanism. Without that, I'm afraid I must assume that TCP Prague is not incrementally deployable.
Indeed, I was under the impression that DualQ and the use of ECT(1) as a classifier stemmed from this incompatibility, rather than being considered features in their own right.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 7:46 ` Jonathan Morton
@ 2019-03-21 8:02 ` Bob Briscoe
2019-03-21 8:49 ` Bless, Roland (TM)
0 siblings, 1 reply; 105+ messages in thread
From: Bob Briscoe @ 2019-03-21 8:02 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Holland, Jake, tsvwg IETF list, bloat
Just to rapidly reply,
On 21/03/2019 07:46, Jonathan Morton wrote:
> The ECN field was never intended to be used as a classifier, except to distinguish Not-ECT flows from ECT flows (which a middlebox does need to know, to choose between mark and drop behaviours). It was intended to be used to convey congestion information from the network to the receiver. SCE adheres to that ideal.
Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an
ECN marking behaviour. The ECN field is the claissifer for the ECN
marking behaviour.
Bob
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [tsvwg] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:28 ` Jonathan Morton
@ 2019-03-21 8:15 ` Mikael Abrahamsson
2019-03-21 8:31 ` Mikael Abrahamsson
0 siblings, 1 reply; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-21 8:15 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Holland, Jake, ecn-sane, bloat
On Thu, 21 Mar 2019, Jonathan Morton wrote:
> Now, when you have a 6000-channel router, I expect it starts with some
> very simple and fast device to direct packets in *approximately* the
> right direction, where some other piece of hardware deals with some
> subset of the total port count. Let's say you have a 1Gbps+ link to a
> 64-line cluster. Immediately the problem of providing FQ on each of
> those lines is more tractable; you can give each one 1024 queues and
> AQMs, the same as fq_codel, for a quite manageable total of 65536, and
> it's not so hard to build hardware that can keep up with 1Gbps traffic
> these days.
I don't know the actual cost of having a high queue number router, I just
know that to currently buy these in the market there is something around a
magnitude difference in cost of buying something with 8 queues per port or
128k queues per port.
The PON OLT is sometimes a smallish device that might need to be
temperature hardened and required to be low power, because it's sitting in
a roadside cabinet or something. The problem here is that you're asking
for a major re-design of how this is done to a group of people that don't
have on the radar that what you're talking about is a serious problem to
solve.
As you said in an earlier email, we're in stupid large FIFO territory a
lot of the time, and even configuring WRED on existing equipment properly
would be a big improvement for the end users, and not even this is done.
I am attending my first BBF meeting ever. Just to give an example of how
networks are often built by the largest operators, you can look at
http://www.ieee1904.org/events/2014_06_workshop/s2_alter_access.pdf slide
"TR-156 and TR-200: GPON and EPON access"
In this case, queueing might happen at so many places along the packet
path and different encapsulations that it's really really hard (and
costly) to change things to do proper FQ. If we look at the BNG->eth
egg->OLT->ONT->RG packet path, typically queueing might happen at the BNG,
but in case of high usage you might have queueing in the OLT->ONT, and the
OLT just isn't looking into the PPPoE payload to be able to do FQ there.
I might not agree with how these people decide networks should be built,
but that's unfortunately the way things look in a lot of cases. Telling
them to just do "FQ, how hard can it be?". Typically, the answer is "hard,
for a multitude of reasons".
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [tsvwg] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 8:15 ` [Bloat] [Ecn-sane] [tsvwg] " Mikael Abrahamsson
@ 2019-03-21 8:31 ` Mikael Abrahamsson
0 siblings, 0 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-21 8:31 UTC (permalink / raw)
To: Jonathan Morton; +Cc: ecn-sane, bloat
On Thu, 21 Mar 2019, Mikael Abrahamsson wrote:
> I might not agree with how these people decide networks should be built,
> but that's unfortunately the way things look in a lot of cases. Telling
> them to just do "FQ, how hard can it be?". Typically, the answer is
> "hard, for a multitude of reasons".
Btw, in
http://1ukcym66nom10cmylunctf84-wpengine.netdna-ssl.com/wp-content/uploads/2018/11/TR-156_Issue-2-1.pdf
5.2.x you can see how scheduling is done. If you'd like something like
that changed then anything new needs to go into documents like these in
order to get further deplyment.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 6:04 ` Bob Briscoe
2019-03-21 7:46 ` Jonathan Morton
@ 2019-03-21 8:45 ` Sebastian Moeller
1 sibling, 0 replies; 105+ messages in thread
From: Sebastian Moeller @ 2019-03-21 8:45 UTC (permalink / raw)
To: Bob Briscoe; +Cc: Jonathan Morton, tsvwg IETF list, bloat
> On Mar 21, 2019, at 07:04, Bob Briscoe <ietf@bobbriscoe.net> wrote:
>
> [...]
> As regards the desire to use SCE instead of the L4S approach of using a classifier, please answer all the reasons I gave for why that won't work, which I sent in response to your draft some days ago.
Is there any work planned showing why the heuristic based on CE is not going to cause problems? Because as it stands ECT(1) might be a useful piece of information for a traffic classifier, but it certainly is not a reliable traffic category identifier (unless you are interested in the category: packets that promise to respond in the L4S-way if they should encounter congestion).
> The main one is incremental deployment: the source does not identify its packets as distinct from others, so the source needs the network to use some other identifier if it wants the network to put it in a queue with latency that is isolated from packets not using the scheme. The only way I can see to so this would be to use per-flow-queuing. I think that is an unstated assumption of SCE.
You write a whole section on alternative labeling schemes in https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-B that you rule out, but that is based on hand-waving over the fact that CE-marking destroys the neat L4S labeling by ECT(1) scheme in two ways (mis-classifiaction of by AQM and mis-interpretation by end-point).
You write: "Given shortage of codepoints, both L4S and classic ECN sides of an AQM would have to use the same CE codepoint to indicate that a packet had experienced congestion. If a packet that had already been marked CE in an upstream buffer arrived at a subsequent AQM, this AQM would then have to guess whether to classify CE packets as L4S or classic ECN. Choosing the L4S treatment would be a safer choice, because then a few classic packets might arrive early, rather than a few L4S packets arriving late;" but in all honestly this simply assumes the rate of CE-marked packets of non-L4S flows will be low, otherwise your four Ls will suffer.
Regarding the second ambiguity you write:
"A.1.4. Fall back to Reno-friendly congestion control on classic ECN bottlenecks [side note on https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-23 bottleneck appears in normal font instead of bold]
It would take time for endpoints to distinguish classic and L4S ECN marking. An increase in queuing delay or in delay variation would be a tell-tale sign, but it is not yet clear where a line would be drawn between the two behaviours. It might be possible to cache what was learned about the path to help subsequent attempts to detect the type of marking."
This has issues that IMHO need empirical data instead of hand-waving. Competent AQM solutions will control traffic rates without introducing massively increasing delays which will make it harder to do a heuristic based on RTT or RTT variation, this is the same mis-design LEDBAT suffers from, making inportant decisions based on un-reliable information... And trying to cache the L4S-ness of the active AQM assumes a steady-state behaviour of the network, which will not work for all cases.
>
> In contrast, L4S works with either per-flow queuing or dualQ, so it is more appropriate for a wider spread of scenarios. Again, in that same unanswered email, I described a way L4S can use per-flow queuing, and Greg has since given pseudocode. There are other problems with doing the codepoints the SCE way round - pls see that email.
>
> There has been a general statement that the SCE way round is purer. However, that concept is in the eye of the beholder. The SCE way round does not allow the ECN field to be used as a classifier,
Nor does the proposed L4S interpretation, as elaborated above.
> so you don't get the benefit above about support for per-flow-queueing and dual queue. You also don't get the benefit of being able to relax resequencing in the network,
I am still failing to grasp how this is supposed to work, and I had a look at the RACK RFC already. IMHO the link technology people should think about how much slack they require and ask the RACK developers to add that as a minimum to the specification, assuming it is reasonable. Say, lower level ARQ is expected to take 5ms worst-case, than ask RACK to specify a minimum reordering window of 10ms. The current RACK draft with re-ordering window bound to be <= RTT will not give reliable re-odering allowance to the lower levels unless they measure the RTT for each flow independently, I am sure that that is not going to fly...
> because the network has no classifier to look at.
Same here CE-marked packets have no reliable label, and I assume that with L4S at steady-state and link capacity one can expect quite a number of CE-marked packets.
I begin to wonder whether there is a nomenclature mismatch here, and I have a definition of classification that is alien to the field here.
> For these, the SCE codepoint would need to be combined with a DSCP, and I assume you don't want to do that.
>
> The L4S way round signifies an alternative meaning of the ECN field, which is exactly what it is. The problem of having to guess which type of packet a CE used to be has been roundly discussed at the IETF in TCPM and TSVWG WGs and it has been decided it is a non-problem
This is not something to "decide" but something to experimentally test, but I guess I am just getting disillusioned how internet sausage is made...
Best Regards
Sebastian
> if it is assumed to have been ECT(1) even if it was not - as written up in draft-ietf-ecn-l4s-id. And that discussion assumed TCP with 3DupACK reordering tolerance, not the more liberal use of RACK (or a RACK-like approach in other transports). With a RACK-like approach, it becomes even less of a problem.
>
>
> Bob
>
>> - Jonathan Morton
>>
>
> --
> ________________________________________________________________
> Bob Briscoe http://bobbriscoe.net/
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 8:02 ` Bob Briscoe
@ 2019-03-21 8:49 ` Bless, Roland (TM)
2019-03-21 13:24 ` Bob Briscoe
0 siblings, 1 reply; 105+ messages in thread
From: Bless, Roland (TM) @ 2019-03-21 8:49 UTC (permalink / raw)
To: Bob Briscoe, Jonathan Morton; +Cc: tsvwg IETF list, bloat
Hi,
Am 21.03.19 um 09:02 schrieb Bob Briscoe:
> Just to rapidly reply,
>
>
> On 21/03/2019 07:46, Jonathan Morton wrote:
>> The ECN field was never intended to be used as a classifier, except to
>> distinguish Not-ECT flows from ECT flows (which a middlebox does need
>> to know, to choose between mark and drop behaviours). It was intended
>> to be used to convey congestion information from the network to the
>> receiver. SCE adheres to that ideal.
>
> Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an
> ECN marking behaviour. The ECN field is the claissifer for the ECN
> marking behaviour.
That's exactly the reason, why using ECT(1) as classifier for L4S
behavior is not the right choice. L4S should use a DSCP for
classification, because it is actually defining a PHB.
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 8:49 ` Bless, Roland (TM)
@ 2019-03-21 13:24 ` Bob Briscoe
2019-03-22 12:53 ` Bless, Roland (TM)
0 siblings, 1 reply; 105+ messages in thread
From: Bob Briscoe @ 2019-03-21 13:24 UTC (permalink / raw)
To: Bless, Roland (TM), Jonathan Morton; +Cc: tsvwg IETF list, bloat
Roland,
On 21/03/2019 08:49, Bless, Roland (TM) wrote:
> Hi,
>
> Am 21.03.19 um 09:02 schrieb Bob Briscoe:
>> Just to rapidly reply,
>>
>>
>> On 21/03/2019 07:46, Jonathan Morton wrote:
>>> The ECN field was never intended to be used as a classifier, except to
>>> distinguish Not-ECT flows from ECT flows (which a middlebox does need
>>> to know, to choose between mark and drop behaviours). It was intended
>>> to be used to convey congestion information from the network to the
>>> receiver. SCE adheres to that ideal.
>> Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an
>> ECN marking behaviour. The ECN field is the claissifer for the ECN
>> marking behaviour.
> That's exactly the reason, why using ECT(1) as classifier for L4S
> behavior is not the right choice. L4S should use a DSCP for
> classification, because it is actually defining a PHB.
1/ First Terminology
The definition of 'PHB' includes the drop or ECN-marking behaviour. For
instance, you see this in WRED or in PCN (Pre-Congestion Notification).
If you want to solely talk about scheduling, pls say the scheduling PHB.
2/ The architectural intent of the ECN field
For many years (long before we thought of L4S) I have been making sure
that ECN propagation through the layers supports the duality of ECN
behaviours as both a classifier (on the way down from L7/L4 to L3/2) and
as a return value (on the way back up).
The architecture of ECN is determined by the valid codepoint
transitions. They are:
1. 00->11
2. 10->11
3. 01->11
4. 10->01
The first three were in RFC3168, but it did not preclude the fourth.
The fourth was first standardized in RFC6660 (which I co-authored). This
had to be isolated from the e2e use of ECN by inclusion of a DSCP as well.
The relatively late addition of the fourth approach means that an
attempt to mark using the SCE approach (10->01) is more likely to find
that it gets reversed when the outer header is decapsulated, if the
decapsulator hasn't been updated to the latest RFC that catered for this
fourth transition (RFC6040, also co-authored by me).
L4S follows the original RFC3168 approach
SCE uses the fourth
So, SCE proposes to use /a/ correct approach, but it might not work.
Whereas L4S uses the original correct approach.
3a/ DualQ L4S AQMs
With the DualQ, the difference between the two queues is both in their
ECN marking behaviour and in their forwarding/scheduling behaviour.
However, whenever there's traffic in the classic queue the coupling
between the AQMs overrides the network scheduler. The coupling is solely
ECN behaviour not scheduling behaviour. So the primary difference
between the queues is in their ECN-marking behaviour.
What do I mean by "the coupling overrides the network scheduler"? The
network scheduler certainly does give priority to L4S packets whenever
they arrive, but the coupling makes the L4S sources control how often
packets arrive. It's tough to reason about, because we haven't had a
mechanism like this before.
2b/ FQ L4S AQMs
If the AQM is implemented with per flow queues, the picture is clearer.
The only difference between the queues is in the ECN marking behaviour
of the different AQMs.
Bob
> Regards
> Roland
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-21 13:24 ` Bob Briscoe
@ 2019-03-22 12:53 ` Bless, Roland (TM)
2019-03-25 2:47 ` Bob Briscoe
0 siblings, 1 reply; 105+ messages in thread
From: Bless, Roland (TM) @ 2019-03-22 12:53 UTC (permalink / raw)
To: Bob Briscoe, Jonathan Morton; +Cc: tsvwg IETF list, bloat
Hi Bob,
see inline.
Am 21.03.19 um 14:24 schrieb Bob Briscoe:
> On 21/03/2019 08:49, Bless, Roland (TM) wrote:
>> Hi,
>>
>> Am 21.03.19 um 09:02 schrieb Bob Briscoe:
>>> Just to rapidly reply,
>>>
>>>
>>> On 21/03/2019 07:46, Jonathan Morton wrote:
>>>> The ECN field was never intended to be used as a classifier, except to
>>>> distinguish Not-ECT flows from ECT flows (which a middlebox does need
>>>> to know, to choose between mark and drop behaviours). It was intended
>>>> to be used to convey congestion information from the network to the
>>>> receiver. SCE adheres to that ideal.
>>> Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an
>>> ECN marking behaviour. The ECN field is the claissifer for the ECN
>>> marking behaviour.
>> That's exactly the reason, why using ECT(1) as classifier for L4S
>> behavior is not the right choice. L4S should use a DSCP for
>> classification, because it is actually defining a PHB.
>
> 1/ First Terminology
> The definition of 'PHB' includes the drop or ECN-marking behaviour. For
> instance, you see this in WRED or in PCN (Pre-Congestion Notification).
> If you want to solely talk about scheduling, pls say the scheduling PHB.
I thought that I'm well versed with Diffserv terminology, but I'm not
aware that a Diffserv PHB requires the definition of an ECN marking
behavior. In fact ECN is orthogonal to Diffserv as both RFCs 2474 and
2475 do not even mention ECN. RFC 2475:
"A per-hop behavior (PHB) is a description of the externally
observable forwarding behavior of a DS node applied to a particular
DS behavior aggregate." and "Useful behavioral distinctions
are mainly observed when multiple behavior aggregates compete for
buffer and bandwidth resources on a node."
Usually, there are different mechanisms how to implement a PHB,
e.g., for EF one could use a tail drop queue and Simple Priority
Queueing, Weighted Fair Queueing, or Deficit Round Robin and so
on. Consequently, queueing and scheduling behavior are used to
_implement_ a PHB, i.e., IMHO it makes sense to distinguish between
the PHB as externally observable behavior and a specific _PHB
implementation_ as also pointed out in RFC2475:
PHBs are implemented in nodes by means of some buffer management and
packet scheduling mechanisms. PHBs are defined in terms of behavior
characteristics relevant to service provisioning policies, and not in
terms of particular implementation mechanisms.
So some of the Diffserv PHBs do _not_ require using an AQM,
which is often the basis for ECN marking, e.g., for EF
tail drop should be sufficient. For other PHBs it may be
useful to say something about ECN usage (as I did for LE).
RFC 2475:
PHBs may be specified in terms of their resource (e.g., buffer,
bandwidth) priority relative to other PHBs, or in terms of their
relative observable traffic characteristics (e.g., delay, loss).
I think that L4S therefore specifies such a PHB as it is defined
in relation to the default PHB (as in the L4S arch draft
"Classic service").
> 2/ The architectural intent of the ECN field
>
> For many years (long before we thought of L4S) I have been making sure
> that ECN propagation through the layers supports the duality of ECN
> behaviours as both a classifier (on the way down from L7/L4 to L3/2) and
> as a return value (on the way back up).
>
> The architecture of ECN is determined by the valid codepoint
> transitions. They are:
I wouldn't say that it's determined solely by the transitions.
> 1. 00->11
> 2. 10->11
> 3. 01->11
> 4. 10->01
>
> The first three were in RFC3168, but it did not preclude the fourth.
> The fourth was first standardized in RFC6660 (which I co-authored). This
> had to be isolated from the e2e use of ECN by inclusion of a DSCP as well.
>
> The relatively late addition of the fourth approach means that an
> attempt to mark using the SCE approach (10->01) is more likely to find
> that it gets reversed when the outer header is decapsulated, if the
> decapsulator hasn't been updated to the latest RFC that catered for this
> fourth transition (RFC6040, also co-authored by me).
>
> L4S follows the original RFC3168 approach
> SCE uses the fourth
>
> So, SCE proposes to use /a/ correct approach, but it might not work.
In case of nodes that implement RFC6040? I think that it would
be useful to measure how many boxes out there actually do this
(or how widespread is ECN usage actually, e.g., how many boxes
actually set CE on congestion? MAMI results anyone?).
> Whereas L4S uses the original correct approach.
Which might also not work...in case RFC3168 boxes set CE, so
the L4S receivers/senders cannot know that the CE wasn't set
by an L4S node.
> 3a/ DualQ L4S AQMs
> With the DualQ, the difference between the two queues is both in their
> ECN marking behaviour and in their forwarding/scheduling behaviour.
> However, whenever there's traffic in the classic queue the coupling
> between the AQMs overrides the network scheduler. The coupling is solely
> ECN behaviour not scheduling behaviour. So the primary difference
> between the queues is in their ECN-marking behaviour.
>
> What do I mean by "the coupling overrides the network scheduler"? The
> network scheduler certainly does give priority to L4S packets whenever
> they arrive, but the coupling makes the L4S sources control how often
> packets arrive. It's tough to reason about, because we haven't had a
> mechanism like this before.
Yes, the DualQ mechanism is actually nice, but what I particularly don't
like is to fix the coupling into nodes in this way. If the congestion
control behavior is different from your expectation it will not work
(as already experienced with BBR) properly. This would ossify congestion
control evolution and I see this a very big disadvantage of this
approach.
> 2b/ FQ L4S AQMs
> If the AQM is implemented with per flow queues, the picture is clearer.
> The only difference between the queues is in the ECN marking behaviour
> of the different AQMs.
This would at least avoid the baked-in coupling law problem...
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-20 23:29 ` Bob Briscoe
2019-03-20 23:51 ` Jonathan Morton
@ 2019-03-24 20:15 ` alex.burr
2019-03-25 1:34 ` Bob Briscoe
1 sibling, 1 reply; 105+ messages in thread
From: alex.burr @ 2019-03-24 20:15 UTC (permalink / raw)
To: Bob Briscoe; +Cc: tsvwg IETF list, bloat
Hi Bob,
I note that all the non-dependent claims of US20170019343A1 (claims 1,14,22) seem to assume use of the proportional-integral controller (Note, I am not a lawyer, and especially not a patent lawyer). In Appendix B of draft-briscoe-tsvwg-aqm-dualq-coupled, an alternate algorithm 'Curvy RED' seems to replace PI, but it is noted that 'the Curvy RED algorithm has not been maintained to the same degree as the DualPI2 algorithm '.
Can you comment on whether the Curvy RED algorithm could form a non-patent-encumbered dualq? In particular:
- Why wasn't curvy red further developed? Was it found to contain some deficiency? Are you intending to present it as an alternative?
- Does Curvy RED actually completely replace PI?
- Can we have reasonable assurance that no patents will surface covering Curvy RED?
Thanks,
Alex
On Wednesday, March 20, 2019, 11:29:38 PM GMT, Bob Briscoe <ietf@bobbriscoe.net> wrote:
1/ In 2016, I arranged for the hire of a patent attorney to undertake the unusual step of filing a third party observation with the European Patent Office. This went through Al-Lu's patent application claim by claim pointing to prior art and giving the patent examiner all the arguments to reject each claim. However, the examiner chose to take very little note of it, which was disappointing and costly for us. The main prior art is:
Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
The guys named as inventors in AL-Lu's filing published a paper on PI2 with me, in which we included a citation of this Gibbens paper as inspiration for the coupling. The Gibbens paper was already cited as background by other patents, so the EPO has it in their citation index.
The coupling was also based on my prior research with Mirja before I started working with the guys from Al-Lu in the RITE European Collaborative project. we had to go through a few rejections, but Mirja and I finally got this work published in 2014 - still before the priority date of the Al-Lu patent application:
Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom Workshop on Telecommunications Standards: From Research to Standards pp.583-588 (December 2014)
2/ The only claim that I could not find prior art for (in the original EU filing) was a very specific claim about using a square root for the coupling. The Linux implementation runs this the other way round so that it only has to do a squaring. So I figured we were safe from that.
However, until just now, I had not noticed that Al-Lu has retrospectively re-written the claims in the US patent and in the EU patent application to claim this the other way round - as a squaring. And to claim the two random number trick. Both restructuring to use a squaring and the two random number trick were definitely my ideas (while working for BT in a collaboration with Al-Lu). I have emails to prove this (from memory they were actually both in the same email). This is important, because a patent has to be about mechanism, not algorithm.
3/ This is a positive development. It means this patent is on very shaky legal ground. I have been trying to put pressure on Nokia to license this royalty free. But now I see what they have done, I am going to have to get a different type of legal advice.
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-24 20:15 ` alex.burr
@ 2019-03-25 1:34 ` Bob Briscoe
2019-03-27 17:52 ` Alex Burr
0 siblings, 1 reply; 105+ messages in thread
From: Bob Briscoe @ 2019-03-25 1:34 UTC (permalink / raw)
To: alex.burr; +Cc: tsvwg IETF list, bloat
Alex, inline...
On 24/03/2019 21:15, alex.burr@ealdwulf.org.uk wrote:
>
> Hi Bob,
>
>
> I note that all the non-dependent claims of US20170019343A1 (claims 1,14,22) seem to assume use of the proportional-integral controller (Note, I am not a lawyer, and especially not a patent lawyer).
Yes, as I understand it, Nokia's intention with this filing was to cover
use of the PI controller in particular, in combination with various
other ideas.
> In Appendix B of draft-briscoe-tsvwg-aqm-dualq-coupled, an alternate algorithm 'Curvy RED' seems to replace PI, but it is noted that 'the Curvy RED algorithm has not been maintained to the same degree as the DualPI2 algorithm '.
>
> Can you comment on whether the Curvy RED algorithm could form a non-patent-encumbered dualq? In particular:
> - Why wasn't curvy red further developed? Was it found to contain some deficiency? Are you intending to present it as an alternative?
We just didn't develop it further, cos we were getting better results
with PI2. However, I am aware of a hardware implementation based on
Curvy RED going on at the moment, and you will see there have recently
been review comments on that Curvy RED appendix on the list.
So, even tho PI might be better, Curvy RED (or another AQM) might be
preferable for other reasons that performance (e.g. ease of
implementation, or similarity to an existing hardware implementation).
And indeed, there's nothing to stop anyone using other AQMs, either to
work round the IPR, or because they're preferable in their own right -
the DualQ Coupled AQM is intentionally a framework into which you drop 2
AQMs.
> - Does Curvy RED actually completely replace PI?
Yes.
> - Can we have reasonable assurance that no patents will surface covering Curvy RED?
Well, I developed the idea of Curvy RED and I / my employer (BT) did not
file any IPR at the time. I got approval to publish a tech report
jointly with Al-Lu. http://bobbriscoe.net/pubs.html#CRED-insights
That was May 2015, so given nothing has surfaced by now, there can't be
anything from that time from us (where us = Al-Lu & BT).
Of course, I cannot guarantee that there is not another patent in the
system from some other random company that my searches haven't found.
There are large numbers of AQM patents. Also, I cannot guarantee that an
implementer working now isn't filing patents around their
implementation. All we can do is publish as much as possible as early as
possible to try to keep some areas of the field open.
Bob
>
> Thanks,
> Alex
>
>
> On Wednesday, March 20, 2019, 11:29:38 PM GMT, Bob Briscoe <ietf@bobbriscoe.net> wrote:
>
>
>
>
>
>
> 1/ In 2016, I arranged for the hire of a patent attorney to undertake the unusual step of filing a third party observation with the European Patent Office. This went through Al-Lu's patent application claim by claim pointing to prior art and giving the patent examiner all the arguments to reject each claim. However, the examiner chose to take very little note of it, which was disappointing and costly for us. The main prior art is:
> Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
> The guys named as inventors in AL-Lu's filing published a paper on PI2 with me, in which we included a citation of this Gibbens paper as inspiration for the coupling. The Gibbens paper was already cited as background by other patents, so the EPO has it in their citation index.
>
> The coupling was also based on my prior research with Mirja before I started working with the guys from Al-Lu in the RITE European Collaborative project. we had to go through a few rejections, but Mirja and I finally got this work published in 2014 - still before the priority date of the Al-Lu patent application:
> Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom Workshop on Telecommunications Standards: From Research to Standards pp.583-588 (December 2014)
>
> 2/ The only claim that I could not find prior art for (in the original EU filing) was a very specific claim about using a square root for the coupling. The Linux implementation runs this the other way round so that it only has to do a squaring. So I figured we were safe from that.
>
> However, until just now, I had not noticed that Al-Lu has retrospectively re-written the claims in the US patent and in the EU patent application to claim this the other way round - as a squaring. And to claim the two random number trick. Both restructuring to use a squaring and the two random number trick were definitely my ideas (while working for BT in a collaboration with Al-Lu). I have emails to prove this (from memory they were actually both in the same email). This is important, because a patent has to be about mechanism, not algorithm.
>
> 3/ This is a positive development. It means this patent is on very shaky legal ground. I have been trying to put pressure on Nokia to license this royalty free. But now I see what they have done, I am going to have to get a different type of legal advice.
>
>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-22 12:53 ` Bless, Roland (TM)
@ 2019-03-25 2:47 ` Bob Briscoe
0 siblings, 0 replies; 105+ messages in thread
From: Bob Briscoe @ 2019-03-25 2:47 UTC (permalink / raw)
To: Bless, Roland (TM), Jonathan Morton; +Cc: tsvwg IETF list, bloat
Roland,
On 22/03/2019 13:53, Bless, Roland (TM) wrote:
> Hi Bob,
>
> see inline.
>
> Am 21.03.19 um 14:24 schrieb Bob Briscoe:
>> On 21/03/2019 08:49, Bless, Roland (TM) wrote:
>>> Hi,
>>>
>>> Am 21.03.19 um 09:02 schrieb Bob Briscoe:
>>>> Just to rapidly reply,
>>>>
>>>>
>>>> On 21/03/2019 07:46, Jonathan Morton wrote:
>>>>> The ECN field was never intended to be used as a classifier, except to
>>>>> distinguish Not-ECT flows from ECT flows (which a middlebox does need
>>>>> to know, to choose between mark and drop behaviours). It was intended
>>>>> to be used to convey congestion information from the network to the
>>>>> receiver. SCE adheres to that ideal.
>>>> Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an
>>>> ECN marking behaviour. The ECN field is the claissifer for the ECN
>>>> marking behaviour.
>>> That's exactly the reason, why using ECT(1) as classifier for L4S
>>> behavior is not the right choice. L4S should use a DSCP for
>>> classification, because it is actually defining a PHB.
>> 1/ First Terminology
>> The definition of 'PHB' includes the drop or ECN-marking behaviour. For
>> instance, you see this in WRED or in PCN (Pre-Congestion Notification).
>> If you want to solely talk about scheduling, pls say the scheduling PHB.
> I thought that I'm well versed with Diffserv terminology, but I'm not
> aware that a Diffserv PHB requires the definition of an ECN marking
> behavior.
Ah well, I do not think that any living human has visited all the dark
corners of the Diffserv world.
I didn't mean (or say) that a Diffserv PHB /requires/ an ECN marking
behaviour, just that if you are talking about a Diffserv PHB that
includes an ECN marking behaviour, it helps to say when you are solely
talking about the scheduling part of the PHB.
In many cases when there is no ECN marking behaviour, it makes no
difference if you omit the word scheduling, cos that is all there is to
the behaviour.
> In fact ECN is orthogonal to Diffserv as both RFCs 2474 and
> 2475 do not even mention ECN. RFC 2475:
> "A per-hop behavior (PHB) is a description of the externally
> observable forwarding behavior of a DS node applied to a particular
> DS behavior aggregate." and "Useful behavioral distinctions
> are mainly observed when multiple behavior aggregates compete for
> buffer and bandwidth resources on a node."
Even the original experimental ECN spec RFC2481 was published just after
2474 & 2475. So you wouldn't expect the original Diffserv specs to
mention something that didn't exist then.
>
> Usually, there are different mechanisms how to implement a PHB,
> e.g., for EF one could use a tail drop queue and Simple Priority
> Queueing, Weighted Fair Queueing, or Deficit Round Robin and so
> on. Consequently, queueing and scheduling behavior are used to
> _implement_ a PHB, i.e., IMHO it makes sense to distinguish between
> the PHB as externally observable behavior and a specific _PHB
> implementation_ as also pointed out in RFC2475:
> PHBs are implemented in nodes by means of some buffer management and
> packet scheduling mechanisms. PHBs are defined in terms of behavior
> characteristics relevant to service provisioning policies, and not in
> terms of particular implementation mechanisms.
>
>
> So some of the Diffserv PHBs do _not_ require using an AQM,
> which is often the basis for ECN marking, e.g., for EF
> tail drop should be sufficient. For other PHBs it may be
> useful to say something about ECN usage (as I did for LE).
>
> RFC 2475:
>
> PHBs may be specified in terms of their resource (e.g., buffer,
> bandwidth) priority relative to other PHBs, or in terms of their
> relative observable traffic characteristics (e.g., delay, loss).
Since RFC2474 & 2475, AQM behaviour and/or ECN marking behaviour has
become part of some Diffserv PHBs. E.g. WRED in AF. See any of the
tables in RFC4594 that have an AQM column.
The need for ECN marking behaviour (rather than just AQM behaviour in
general) as part of a PHB became necessary during the definition of PCN.
Jo Babiarz and Kwok Ho Chan were both authors of 4594 and of many of the
PCN specs, and proposed the term 'marking behaviour' as part of the PHB.
You will find ECN marking behaviours are central to, for instance, RFC5670.
As I've pointed out already, the transitions used by SCE were already in
the PCN baseline encoding spec [RFC6660], except only defined if
accompanied by a specific DSCP (which was subsequently standardized as
EF-ADMIT).
>
> I think that L4S therefore specifies such a PHB as it is defined
> in relation to the default PHB (as in the L4S arch draft
> "Classic service").
No. The L4S use of ECN is orthogonal to Diffserv, and can be associated
with more than one scheduling PHB in a queuing hierarchy. See
draft-briscoe-l4s-diffserv (which I would welcome you to review - Brian
Carpenter is also currently reviewing it).
However, you are certainly right in thinking that L4S associated with
the default PHB is by far and away the most important use-case for L4S.
All the other possible schemes in l4s-diffserv are only possibilities -
probably for corporate networks where proliferation of Diffserv models
is currently most prevalent.
>
>> 2/ The architectural intent of the ECN field
>>
>> For many years (long before we thought of L4S) I have been making sure
>> that ECN propagation through the layers supports the duality of ECN
>> behaviours as both a classifier (on the way down from L7/L4 to L3/2) and
>> as a return value (on the way back up).
>>
>> The architecture of ECN is determined by the valid codepoint
>> transitions. They are:
> I wouldn't say that it's determined solely by the transitions.
Correct. I didn't say that either.
>
>> 1. 00->11
>> 2. 10->11
>> 3. 01->11
>> 4. 10->01
>>
>> The first three were in RFC3168, but it did not preclude the fourth.
>> The fourth was first standardized in RFC6660 (which I co-authored). This
>> had to be isolated from the e2e use of ECN by inclusion of a DSCP as well.
>>
>> The relatively late addition of the fourth approach means that an
>> attempt to mark using the SCE approach (10->01) is more likely to find
>> that it gets reversed when the outer header is decapsulated, if the
>> decapsulator hasn't been updated to the latest RFC that catered for this
>> fourth transition (RFC6040, also co-authored by me).
>>
>> L4S follows the original RFC3168 approach
>> SCE uses the fourth
>>
>> So, SCE proposes to use /a/ correct approach, but it might not work.
> In case of nodes that implement RFC6040? I think that it would
> be useful to measure how many boxes out there actually do this
[BB] To measure this, you would need to have a box between the tunnel
endpoints to mark the outer, before you could check what the behaviour
of the decap was.
> (or how widespread is ECN usage actually, e.g., how many boxes
> actually set CE on congestion? MAMI results anyone?).
[BB] Brian Trammel re-ran ETHZ's ECN tests in Jan'19. Informally he told
me yesterday that he found about 13 CE marks (i.e. still hardly any).
But this might mean the links aren't loaded when he's looking. It's hard
to apply enough load while doing a large-scale measurement - takes far
too long per path.
Stuart Cheshire is helping me to try to get something meaningful out of
the data Apple is continually gathering. The data Padma presented at
MAPRG at IETF-98 was % of Apple devices tested that saw at least one CE
mark in 12 hours.
The hard problem is, once we find something CE-marking, we're interested
in knowing whether it's FQ (which protects flows from each other) or
single queue (which doesn't). Greg White, Jake Holland & I devised a
test for this between us. Tweak a congestion control so you have an
aggressive one. Run it in parallel with another regular CC between the
same two hosts. Then look for correlation of RTT movements. Like common
bottleneck detection in RMCAT.
>
>> Whereas L4S uses the original correct approach.
> Which might also not work...in case RFC3168 boxes set CE, so
> the L4S receivers/senders cannot know that the CE wasn't set
> by an L4S node.
Yes. That's a different point, but it's true.
>
>> 3a/ DualQ L4S AQMs
>> With the DualQ, the difference between the two queues is both in their
>> ECN marking behaviour and in their forwarding/scheduling behaviour.
>> However, whenever there's traffic in the classic queue the coupling
>> between the AQMs overrides the network scheduler. The coupling is solely
>> ECN behaviour not scheduling behaviour. So the primary difference
>> between the queues is in their ECN-marking behaviour.
>>
>> What do I mean by "the coupling overrides the network scheduler"? The
>> network scheduler certainly does give priority to L4S packets whenever
>> they arrive, but the coupling makes the L4S sources control how often
>> packets arrive. It's tough to reason about, because we haven't had a
>> mechanism like this before.
> Yes, the DualQ mechanism is actually nice, but what I particularly don't
> like is to fix the coupling into nodes in this way. If the congestion
> control behavior is different from your expectation it will not work
> (as already experienced with BBR) properly. This would ossify congestion
> control evolution and I see this a very big disadvantage of this
> approach.
The argument for using the square is to be compatible with the
worst-case classic traffic, which is Reno. The worst-case will remain
ossified for some many years yet. It doesn't mean that better CCs cannot
develop within Classic. And to a certain extent, they still have to
coexist with the worst case Classic... we'll see what the latest update
on BBRv2 is in a few day's time.
But importantly, a square relationship between flow rates is not
enforced by the network, it's encouraged for end-systems that have a
default behaviour. But, if the network policy becomes wrong in future,
end-systems can correct it.
>
>> 2b/ FQ L4S AQMs
>> If the AQM is implemented with per flow queues, the picture is clearer.
>> The only difference between the queues is in the ECN marking behaviour
>> of the different AQMs.
> This would at least avoid the baked-in coupling law problem...
Er, no. 1:1 is just as much a baked-in policy, and it really is baked-in
when the network is enforcing it.
That's why we developed the DualQ - we wanted to avoid the network
enforcing rigid fairness at every instant. It's much less ossified when
end-systems keep to it voluntarily.
Bob
>
> Regards
> Roland
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-25 1:34 ` Bob Briscoe
@ 2019-03-27 17:52 ` Alex Burr
0 siblings, 0 replies; 105+ messages in thread
From: Alex Burr @ 2019-03-27 17:52 UTC (permalink / raw)
To: Bob Briscoe; +Cc: tsvwg IETF list, bloat
[-- Attachment #1: Type: text/plain, Size: 5956 bytes --]
Bob,
Thanks, this is interesting. As you probably know this patent has come to
the attention of the Linux community and caused some concern:
https://lwn.net/Articles/784125/
so it's useful to know of a potential workaround.
Alex
On Mon, Mar 25, 2019 at 1:34 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
> Alex, inline...
>
> On 24/03/2019 21:15, alex.burr@ealdwulf.org.uk wrote:
> >
> > Hi Bob,
> >
> >
> > I note that all the non-dependent claims of US20170019343A1 (claims
> 1,14,22) seem to assume use of the proportional-integral controller (Note,
> I am not a lawyer, and especially not a patent lawyer).
> Yes, as I understand it, Nokia's intention with this filing was to cover
> use of the PI controller in particular, in combination with various
> other ideas.
>
> > In Appendix B of draft-briscoe-tsvwg-aqm-dualq-coupled, an alternate
> algorithm 'Curvy RED' seems to replace PI, but it is noted that 'the Curvy
> RED algorithm has not been maintained to the same degree as the DualPI2
> algorithm '.
> >
> > Can you comment on whether the Curvy RED algorithm could form a
> non-patent-encumbered dualq? In particular:
> > - Why wasn't curvy red further developed? Was it found to contain some
> deficiency? Are you intending to present it as an alternative?
> We just didn't develop it further, cos we were getting better results
> with PI2. However, I am aware of a hardware implementation based on
> Curvy RED going on at the moment, and you will see there have recently
> been review comments on that Curvy RED appendix on the list.
>
> So, even tho PI might be better, Curvy RED (or another AQM) might be
> preferable for other reasons that performance (e.g. ease of
> implementation, or similarity to an existing hardware implementation).
>
> And indeed, there's nothing to stop anyone using other AQMs, either to
> work round the IPR, or because they're preferable in their own right -
> the DualQ Coupled AQM is intentionally a framework into which you drop 2
> AQMs.
>
> > - Does Curvy RED actually completely replace PI?
> Yes.
> > - Can we have reasonable assurance that no patents will surface
> covering Curvy RED?
> Well, I developed the idea of Curvy RED and I / my employer (BT) did not
> file any IPR at the time. I got approval to publish a tech report
> jointly with Al-Lu. http://bobbriscoe.net/pubs.html#CRED-insights
>
> That was May 2015, so given nothing has surfaced by now, there can't be
> anything from that time from us (where us = Al-Lu & BT).
>
> Of course, I cannot guarantee that there is not another patent in the
> system from some other random company that my searches haven't found.
> There are large numbers of AQM patents. Also, I cannot guarantee that an
> implementer working now isn't filing patents around their
> implementation. All we can do is publish as much as possible as early as
> possible to try to keep some areas of the field open.
>
>
> Bob
> >
> > Thanks,
> > Alex
> >
> >
> > On Wednesday, March 20, 2019, 11:29:38 PM GMT, Bob Briscoe <
> ietf@bobbriscoe.net> wrote:
> >
> >
> >
> >
> >
> >
> > 1/ In 2016, I arranged for the hire of a patent attorney to undertake
> the unusual step of filing a third party observation with the European
> Patent Office. This went through Al-Lu's patent application claim by claim
> pointing to prior art and giving the patent examiner all the arguments to
> reject each claim. However, the examiner chose to take very little note of
> it, which was disappointing and costly for us. The main prior art is:
> > Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority
> Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
> > The guys named as inventors in AL-Lu's filing published a paper on PI2
> with me, in which we included a citation of this Gibbens paper as
> inspiration for the coupling. The Gibbens paper was already cited as
> background by other patents, so the EPO has it in their citation index.
> >
> > The coupling was also based on my prior research with Mirja before I
> started working with the guys from Al-Lu in the RITE European Collaborative
> project. we had to go through a few rejections, but Mirja and I finally got
> this work published in 2014 - still before the priority date of the Al-Lu
> patent application:
> > Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using
> Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom
> Workshop on Telecommunications Standards: From Research to Standards
> pp.583-588 (December 2014)
> >
> > 2/ The only claim that I could not find prior art for (in the original
> EU filing) was a very specific claim about using a square root for the
> coupling. The Linux implementation runs this the other way round so that it
> only has to do a squaring. So I figured we were safe from that.
> >
> > However, until just now, I had not noticed that Al-Lu has
> retrospectively re-written the claims in the US patent and in the EU patent
> application to claim this the other way round - as a squaring. And to claim
> the two random number trick. Both restructuring to use a squaring and the
> two random number trick were definitely my ideas (while working for BT in a
> collaboration with Al-Lu). I have emails to prove this (from memory they
> were actually both in the same email). This is important, because a patent
> has to be about mechanism, not algorithm.
> >
> > 3/ This is a positive development. It means this patent is on very shaky
> legal ground. I have been trying to put pressure on Nokia to license this
> royalty free. But now I see what they have done, I am going to have to get
> a different type of legal advice.
> >
> >
>
> --
> ________________________________________________________________
> Bob Briscoe http://bobbriscoe.net/
>
>
[-- Attachment #2: Type: text/html, Size: 6948 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 18:40 ` Mikael Abrahamsson
2019-03-23 19:11 ` Michael Welzl
@ 2019-03-23 21:04 ` Luca Muscariello
1 sibling, 0 replies; 105+ messages in thread
From: Luca Muscariello @ 2019-03-23 21:04 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Michael Welzl, Victor Hou, bloat
[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]
For enterprise networks interconnection to the cloud (public or private)
can be done with e2e DSCP marking.
This is just part of products available at AWS/Azure w/ or w/o their own
interconnection network (direct connect, express route)
and used regularly to ameliorate QoS of RTC applications such as WebEx,
Skype for business, WebRTC.
As I said, peering is key and cloud providers are doing that very well.
With the current widespread of SaaS and IaaS your app is likely running in
the cloud.
Even in case a branch office is connected using SD-WAN, DSCP is tunnelled
by it, and transparently for the application.
If working from home, the quality of peering of an SP to cloud providers is
key.
I'd say that the Cloud and the decline of transit networks have facilitated
DSCP e2e.
On Sat, Mar 23, 2019 at 7:40 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Sat, 23 Mar 2019, Luca Muscariello wrote:
>
> > It the app runs in the cloud and the cloud has direct peering links to
> > your branch office or SP most likely DSCP works.
>
> Do you have numbers to back this statement up? In my experience direct
> peering links has nothing to do with this, instead remaking is done
> equally at the customer edge and peering/transit edge respectively.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
>
[-- Attachment #2: Type: text/html, Size: 1833 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 17:48 ` Michael Welzl
2019-03-23 18:31 ` Luca Muscariello
@ 2019-03-23 19:55 ` Roland Bless
1 sibling, 0 replies; 105+ messages in thread
From: Roland Bless @ 2019-03-23 19:55 UTC (permalink / raw)
To: Michael Welzl; +Cc: bloat
Hi Michael,
thanks for the pointer. Looking forward to the talk...
Regards
Roland
On 23.03.19 at 18:48 Michael Welzl wrote:
> Hi,
>
> I’ll do what academics do and add our own data point, below:
>
>> On Mar 23, 2019, at 4:19 PM, Roland Bless <roland.bless@kit.edu
>> <mailto:roland.bless@kit.edu>> wrote:
>>
>> Hi Mikael,
>>
>> On 23.03.19 at 11:02 Mikael Abrahamsson wrote:
>>> On Sat, 23 Mar 2019, Roland Bless wrote:
>>>
>>>> I suggest to use an additional DSCP to mark L4S packets.
>>>
>>> DSCP doesn't work end-to-end on the Internet, so what you're suggesting
>>> isn't a workable solution.
>>
>> It's true that DSCPs may be remarked, but RFC 2474
>> already stated
>>
>> Packets received with an unrecognized codepoint SHOULD be forwarded
>> as if they were marked for the Default behavior (see Sec. 4), and
>> their codepoints should not be changed.
>>
>> The rtcweb WG also counts on e2e DSCPs.
>> After the LE PHB experience I would suggest to probably use
>> DSCP 5 which has got better chances to survive ToS bleaching (maybe
>> around 80%), if I understand
>> https://www.sciencedirect.com/science/article/pii/S0140366417312835
>> correctly. Diffserv behavior is usually configurable and can be changed
>> without replacing the network gear.
>
> Runa Barik, Michael Welzl, Ahmed Mustafa Elmokashfi, Thomas Dreibholz,
> Stein Gjessing: "Can WebRTC QoS Work? A DSCP Measurement Study", 30th
> International Teletraffic Congress (ITC 30), 3 - 7 September 2018,
> Vienna, Austria.
> https://itc-conference.org/_Resources/Persistent/780df4482d0fe80f6180f523ebb9482c6869e98b/Barik18ITC30.pdf
>
> We didn’t measure undefined codepoints though, only EF, AF42 and CS1.
> Table 2 compares bleaching for these codepoints with the results in the
> paper you point to; it’s quite similar.
> Well yes, there’s a significant amount of bleaching, we can’t deny that.
> But sometimes the codepoint survives, and it seems to survive
> surprisingly long into the path (fig.4 in our paper).
>
> In the MAPRG session in Prague, Runa Barik will give a talk about the
> measured delay impact of such opportunistic use of these DSCP values by
> an end system.
>
> Cheers,
> Michael
>
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 15:03 ` Jonathan Morton
@ 2019-03-23 19:52 ` Roland Bless
0 siblings, 0 replies; 105+ messages in thread
From: Roland Bless @ 2019-03-23 19:52 UTC (permalink / raw)
To: Jonathan Morton, Mikael Abrahamsson; +Cc: Victor Hou, bloat
Hi Jonathan,
On 23.03.19 at 16:03 Jonathan Morton wrote:
>> On 23 Mar, 2019, at 11:02 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Sat, 23 Mar 2019, Roland Bless wrote:
>>
>>> I suggest to use an additional DSCP to mark L4S packets.
>>
>> DSCP doesn't work end-to-end on the Internet, so what you're suggesting isn't a workable solution.
>
> An interesting question, in this context, is "precisely what happens if the DSCP is lost?"
>
> With a notional L4S using DSCP instead of ECT(1) as an identifier, that really is a serious problem; the L4S flow will flood out TCP-friendly flows *unless* its failsafe kicks in.
One possibility to avoid such problems could be to check for DSCP
remarking on connection setup (i.e., during TCP handshake) and then fall
back to classic behavior in case that the DSCP was modified.
> But with SCE, what happens is that the L4S flow gets treated the same as any other, if the bottleneck where the distinction matters is downstream of the DSCP corruption. The L4S flow reacts properly to CE in this scenario, because its been designed around SCE semantics, so it is TCP-friendly. Flow-isolating AQMs don't need to know the DSCP in the first place.
>
> So the worst case is when a single or dual-queue AQM bottleneck is involved, and the L4S flow is affected by queuing induced by other flows who aren't quite so enlightened. The damage is only to the L4S flow, and within reasonable bounds.
>
> This of course ignores the overwhelmingly most common situation on today's Internet, where the bottleneck queue is completely unmanaged. But then, losing the DSCP has no effect there.
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 17:16 ` Mikael Abrahamsson
@ 2019-03-23 19:45 ` Roland Bless
0 siblings, 0 replies; 105+ messages in thread
From: Roland Bless @ 2019-03-23 19:45 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Victor Hou, bloat
Hi Mikael,
On 23.03.19 at 18:16 Mikael Abrahamsson wrote:
> On Sat, 23 Mar 2019, Roland Bless wrote:
>
>> It's true that DSCPs may be remarked, but RFC 2474
>> already stated
>>
>> Packets received with an unrecognized codepoint SHOULD be forwarded
>> as if they were marked for the Default behavior (see Sec. 4), and
>> their codepoints should not be changed.
>
> https://mailman.nanog.org/pipermail/nanog/2015-May/075004.html
>
> https://www.nanog.org/mailinglist/mailarchives/old_archive/2005-05/msg00654.html
This is pretty sad. The correct answer to the first question
"does Internet trust IP DSCP marking?" should have been twofold:
a) don't trust already present markings on ingress
for your own supported PHBs (except default and LE PHBs :-)
unless you have agreed with the neighboring
DS domain.
b) Packets received with an unrecognized DSCP SHOULD be forwarded
as best effort and their DSCP should NOT be changed.
The BCP to unconditionally bleach (set to 0) is IMHO simply wrong: one
has to distinguish between treating as default PHB and overwriting the
DSCP. For internally supported DSCPs/PHBs one typically needs to bleach
(but e.g., not for LE), but for all unsupported DSCPs simply map them to
the default PHB.
It's true that Diffserv's major line of defense is the domain
boundary that needs to protect the domain's resources against
unauthorized use. So a domain that internally supports EF should not
honor incoming EF marked packets from untrusted/unadmitted sources, and
therefore must bleach them. For unsupported DSCPs though,
one could simply _map_ them to the default PHB while retaining the DSCP.
> Please note the dates, as in 4 and 14 years ago respectively.
>
> So please read those threads and then tell me that what you quoted above
> has bearing on reality.
It's clear that just setting everything to DSCP 0 is the safe option
(in case one has no full control over all equipment etc.),
but it has the mentioned drawback of limiting the future extensibility.
Since Diffserv requires a configurable mapping of DSCP to PHB
a consistent configuration should be possible, nevertheless.
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 18:40 ` Mikael Abrahamsson
@ 2019-03-23 19:11 ` Michael Welzl
2019-03-23 21:04 ` Luca Muscariello
1 sibling, 0 replies; 105+ messages in thread
From: Michael Welzl @ 2019-03-23 19:11 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Luca Muscariello, Victor Hou, bloat
Our paper has some data - again, fig. 4. That’s an AS level analysis, for the data that we had (which wasn’t huge - a couple thousand paths). For the majority of measurements, the DSCP survived more than 1 AS hop; in case of IPv6, the majority survived more than 2.
Cheers,
Michael
> On Mar 23, 2019, at 7:40 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Sat, 23 Mar 2019, Luca Muscariello wrote:
>
>> It the app runs in the cloud and the cloud has direct peering links to your branch office or SP most likely DSCP works.
>
> Do you have numbers to back this statement up? In my experience direct peering links has nothing to do with this, instead remaking is done equally at the customer edge and peering/transit edge respectively.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 18:31 ` Luca Muscariello
@ 2019-03-23 18:40 ` Mikael Abrahamsson
2019-03-23 19:11 ` Michael Welzl
2019-03-23 21:04 ` Luca Muscariello
0 siblings, 2 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-23 18:40 UTC (permalink / raw)
To: Luca Muscariello; +Cc: Michael Welzl, Victor Hou, bloat
On Sat, 23 Mar 2019, Luca Muscariello wrote:
> It the app runs in the cloud and the cloud has direct peering links to
> your branch office or SP most likely DSCP works.
Do you have numbers to back this statement up? In my experience direct
peering links has nothing to do with this, instead remaking is done
equally at the customer edge and peering/transit edge respectively.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 17:48 ` Michael Welzl
@ 2019-03-23 18:31 ` Luca Muscariello
2019-03-23 18:40 ` Mikael Abrahamsson
2019-03-23 19:55 ` Roland Bless
1 sibling, 1 reply; 105+ messages in thread
From: Luca Muscariello @ 2019-03-23 18:31 UTC (permalink / raw)
To: Michael Welzl; +Cc: Roland Bless, Victor Hou, bloat
[-- Attachment #1: Type: text/plain, Size: 2633 bytes --]
WebEx, WebRTC they use it.
QoS works. To answer the question in the title of Michael’s paper.
It the app runs in the cloud and the cloud has direct peering links to your
branch office or SP
most likely DSCP works.
And going back to Roland’s proposal, it seems the most natural approach
instead of hacking the semantics.
On Sat 23 Mar 2019 at 18:48, Michael Welzl <michawe@ifi.uio.no> wrote:
> Hi,
>
> I’ll do what academics do and add our own data point, below:
>
> On Mar 23, 2019, at 4:19 PM, Roland Bless <roland.bless@kit.edu> wrote:
>
> Hi Mikael,
>
> On 23.03.19 at 11:02 Mikael Abrahamsson wrote:
>
> On Sat, 23 Mar 2019, Roland Bless wrote:
>
> I suggest to use an additional DSCP to mark L4S packets.
>
>
> DSCP doesn't work end-to-end on the Internet, so what you're suggesting
> isn't a workable solution.
>
>
> It's true that DSCPs may be remarked, but RFC 2474
> already stated
>
> Packets received with an unrecognized codepoint SHOULD be forwarded
> as if they were marked for the Default behavior (see Sec. 4), and
> their codepoints should not be changed.
>
> The rtcweb WG also counts on e2e DSCPs.
> After the LE PHB experience I would suggest to probably use
> DSCP 5 which has got better chances to survive ToS bleaching (maybe
> around 80%), if I understand
> https://www.sciencedirect.com/science/article/pii/S0140366417312835
> correctly. Diffserv behavior is usually configurable and can be changed
> without replacing the network gear.
>
>
> Runa Barik, Michael Welzl, Ahmed Mustafa Elmokashfi, Thomas Dreibholz,
> Stein Gjessing: "Can WebRTC QoS Work? A DSCP Measurement Study", 30th
> International Teletraffic Congress (ITC 30), 3 - 7 September 2018, Vienna,
> Austria.
>
> https://itc-conference.org/_Resources/Persistent/780df4482d0fe80f6180f523ebb9482c6869e98b/Barik18ITC30.pdf
>
> We didn’t measure undefined codepoints though, only EF, AF42 and CS1.
> Table 2 compares bleaching for these codepoints with the results in the
> paper you point to; it’s quite similar.
> Well yes, there’s a significant amount of bleaching, we can’t deny that.
> But sometimes the codepoint survives, and it seems to survive surprisingly
> long into the path (fig.4 in our paper).
>
> In the MAPRG session in Prague, Runa Barik will give a talk about the
> measured delay impact of such opportunistic use of these DSCP values by an
> end system.
>
> Cheers,
> Michael
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
[-- Attachment #2: Type: text/html, Size: 3941 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 15:19 ` Roland Bless
2019-03-23 17:16 ` Mikael Abrahamsson
@ 2019-03-23 17:48 ` Michael Welzl
2019-03-23 18:31 ` Luca Muscariello
2019-03-23 19:55 ` Roland Bless
1 sibling, 2 replies; 105+ messages in thread
From: Michael Welzl @ 2019-03-23 17:48 UTC (permalink / raw)
To: Roland Bless; +Cc: Mikael Abrahamsson, Victor Hou, bloat
[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]
Hi,
I’ll do what academics do and add our own data point, below:
> On Mar 23, 2019, at 4:19 PM, Roland Bless <roland.bless@kit.edu> wrote:
>
> Hi Mikael,
>
> On 23.03.19 at 11:02 Mikael Abrahamsson wrote:
>> On Sat, 23 Mar 2019, Roland Bless wrote:
>>
>>> I suggest to use an additional DSCP to mark L4S packets.
>>
>> DSCP doesn't work end-to-end on the Internet, so what you're suggesting
>> isn't a workable solution.
>
> It's true that DSCPs may be remarked, but RFC 2474
> already stated
>
> Packets received with an unrecognized codepoint SHOULD be forwarded
> as if they were marked for the Default behavior (see Sec. 4), and
> their codepoints should not be changed.
>
> The rtcweb WG also counts on e2e DSCPs.
> After the LE PHB experience I would suggest to probably use
> DSCP 5 which has got better chances to survive ToS bleaching (maybe
> around 80%), if I understand
> https://www.sciencedirect.com/science/article/pii/S0140366417312835
> correctly. Diffserv behavior is usually configurable and can be changed
> without replacing the network gear.
Runa Barik, Michael Welzl, Ahmed Mustafa Elmokashfi, Thomas Dreibholz, Stein Gjessing: "Can WebRTC QoS Work? A DSCP Measurement Study", 30th International Teletraffic Congress (ITC 30), 3 - 7 September 2018, Vienna, Austria.
https://itc-conference.org/_Resources/Persistent/780df4482d0fe80f6180f523ebb9482c6869e98b/Barik18ITC30.pdf <https://itc-conference.org/_Resources/Persistent/780df4482d0fe80f6180f523ebb9482c6869e98b/Barik18ITC30.pdf>
We didn’t measure undefined codepoints though, only EF, AF42 and CS1. Table 2 compares bleaching for these codepoints with the results in the paper you point to; it’s quite similar.
Well yes, there’s a significant amount of bleaching, we can’t deny that. But sometimes the codepoint survives, and it seems to survive surprisingly long into the path (fig.4 in our paper).
In the MAPRG session in Prague, Runa Barik will give a talk about the measured delay impact of such opportunistic use of these DSCP values by an end system.
Cheers,
Michael
[-- Attachment #2: Type: text/html, Size: 3243 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 15:19 ` Roland Bless
@ 2019-03-23 17:16 ` Mikael Abrahamsson
2019-03-23 19:45 ` Roland Bless
2019-03-23 17:48 ` Michael Welzl
1 sibling, 1 reply; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-23 17:16 UTC (permalink / raw)
To: Roland Bless; +Cc: Victor Hou, bloat
On Sat, 23 Mar 2019, Roland Bless wrote:
> It's true that DSCPs may be remarked, but RFC 2474
> already stated
>
> Packets received with an unrecognized codepoint SHOULD be forwarded
> as if they were marked for the Default behavior (see Sec. 4), and
> their codepoints should not be changed.
https://mailman.nanog.org/pipermail/nanog/2015-May/075004.html
https://www.nanog.org/mailinglist/mailarchives/old_archive/2005-05/msg00654.html
Please note the dates, as in 4 and 14 years ago respectively.
So please read those threads and then tell me that what you quoted above
has bearing on reality.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 10:02 ` Mikael Abrahamsson
2019-03-23 15:03 ` Jonathan Morton
@ 2019-03-23 15:19 ` Roland Bless
2019-03-23 17:16 ` Mikael Abrahamsson
2019-03-23 17:48 ` Michael Welzl
1 sibling, 2 replies; 105+ messages in thread
From: Roland Bless @ 2019-03-23 15:19 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Victor Hou, bloat
Hi Mikael,
On 23.03.19 at 11:02 Mikael Abrahamsson wrote:
> On Sat, 23 Mar 2019, Roland Bless wrote:
>
>> I suggest to use an additional DSCP to mark L4S packets.
>
> DSCP doesn't work end-to-end on the Internet, so what you're suggesting
> isn't a workable solution.
It's true that DSCPs may be remarked, but RFC 2474
already stated
Packets received with an unrecognized codepoint SHOULD be forwarded
as if they were marked for the Default behavior (see Sec. 4), and
their codepoints should not be changed.
The rtcweb WG also counts on e2e DSCPs.
After the LE PHB experience I would suggest to probably use
DSCP 5 which has got better chances to survive ToS bleaching (maybe
around 80%), if I understand
https://www.sciencedirect.com/science/article/pii/S0140366417312835
correctly. Diffserv behavior is usually configurable and can be changed
without replacing the network gear.
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 10:02 ` Mikael Abrahamsson
@ 2019-03-23 15:03 ` Jonathan Morton
2019-03-23 19:52 ` Roland Bless
2019-03-23 15:19 ` Roland Bless
1 sibling, 1 reply; 105+ messages in thread
From: Jonathan Morton @ 2019-03-23 15:03 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: Roland Bless, Victor Hou, bloat
> On 23 Mar, 2019, at 11:02 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Sat, 23 Mar 2019, Roland Bless wrote:
>
>> I suggest to use an additional DSCP to mark L4S packets.
>
> DSCP doesn't work end-to-end on the Internet, so what you're suggesting isn't a workable solution.
An interesting question, in this context, is "precisely what happens if the DSCP is lost?"
With a notional L4S using DSCP instead of ECT(1) as an identifier, that really is a serious problem; the L4S flow will flood out TCP-friendly flows *unless* its failsafe kicks in.
But with SCE, what happens is that the L4S flow gets treated the same as any other, if the bottleneck where the distinction matters is downstream of the DSCP corruption. The L4S flow reacts properly to CE in this scenario, because its been designed around SCE semantics, so it is TCP-friendly. Flow-isolating AQMs don't need to know the DSCP in the first place.
So the worst case is when a single or dual-queue AQM bottleneck is involved, and the L4S flow is affected by queuing induced by other flows who aren't quite so enlightened. The damage is only to the L4S flow, and within reasonable bounds.
This of course ignores the overwhelmingly most common situation on today's Internet, where the bottleneck queue is completely unmanaged. But then, losing the DSCP has no effect there.
- Jonathan Morton
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 8:02 ` Roland Bless
2019-03-23 8:54 ` Luca Muscariello
@ 2019-03-23 10:02 ` Mikael Abrahamsson
2019-03-23 15:03 ` Jonathan Morton
2019-03-23 15:19 ` Roland Bless
1 sibling, 2 replies; 105+ messages in thread
From: Mikael Abrahamsson @ 2019-03-23 10:02 UTC (permalink / raw)
To: Roland Bless; +Cc: Victor Hou, bloat
On Sat, 23 Mar 2019, Roland Bless wrote:
> I suggest to use an additional DSCP to mark L4S packets.
DSCP doesn't work end-to-end on the Internet, so what you're suggesting
isn't a workable solution.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-23 8:02 ` Roland Bless
@ 2019-03-23 8:54 ` Luca Muscariello
2019-03-23 10:02 ` Mikael Abrahamsson
1 sibling, 0 replies; 105+ messages in thread
From: Luca Muscariello @ 2019-03-23 8:54 UTC (permalink / raw)
To: Roland Bless; +Cc: Victor Hou, bloat
[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]
+1
On Sat, Mar 23, 2019 at 9:02 AM Roland Bless <roland.bless@kit.edu> wrote:
> Hi,
>
> On 22.03.19 at 19:28 Victor Hou wrote:
>
> > Broadcom has been deeply involved in developing the Low Latency DOCSIS
> > cable industry specification that Bob Briscoe mentioned. We concur with
> > the L4S use of ECT(1). L4S can be implemented either in a dual-queue or
> > an fq implementation. SCE cannot be implemented with a dual-queue
> > implementation; the only way to support it is with an fq
> > implementation. An fq implementation is incompatible with the Low
> > Latency DOCSIS specification developed within the cable industry.
>
> I don't understand your rationale here.
> The basic SCE concept could be used for L4S as well.
> I suggest to use an additional DSCP to mark L4S packets.
> The L4S sender/receiver can simply react to the SCE
> markings the same way that they now react to CE, with
> the difference that it's safer to react to SCE, because
> the signal is unambiguous, whereas CE would be ambiguous
> (could be set by either classic AQM/ECN node or by
> an L4S node).
>
> Regards
> Roland
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
[-- Attachment #2: Type: text/html, Size: 1818 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
2019-03-22 18:28 [Bloat] [Ecn-sane] " Victor Hou
@ 2019-03-23 8:02 ` Roland Bless
2019-03-23 8:54 ` Luca Muscariello
2019-03-23 10:02 ` Mikael Abrahamsson
0 siblings, 2 replies; 105+ messages in thread
From: Roland Bless @ 2019-03-23 8:02 UTC (permalink / raw)
To: Victor Hou, bloat
Hi,
On 22.03.19 at 19:28 Victor Hou wrote:
> Broadcom has been deeply involved in developing the Low Latency DOCSIS
> cable industry specification that Bob Briscoe mentioned. We concur with
> the L4S use of ECT(1). L4S can be implemented either in a dual-queue or
> an fq implementation. SCE cannot be implemented with a dual-queue
> implementation; the only way to support it is with an fq
> implementation. An fq implementation is incompatible with the Low
> Latency DOCSIS specification developed within the cable industry.
I don't understand your rationale here.
The basic SCE concept could be used for L4S as well.
I suggest to use an additional DSCP to mark L4S packets.
The L4S sender/receiver can simply react to the SCE
markings the same way that they now react to CE, with
the difference that it's safer to react to SCE, because
the signal is unambiguous, whereas CE would be ambiguous
(could be set by either classic AQM/ECN node or by
an L4S node).
Regards
Roland
^ permalink raw reply [flat|nested] 105+ messages in thread
* [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
@ 2019-03-22 18:28 Victor Hou
2019-03-23 8:02 ` Roland Bless
0 siblings, 1 reply; 105+ messages in thread
From: Victor Hou @ 2019-03-22 18:28 UTC (permalink / raw)
To: bloat
[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]
On Mon Mar 18 21:06:57 EDT 2019, Bob Briscoe said:
>>>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired
me later in the year to help develop and specify it, along with support
for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now
being implemented.
o You can find the primary companies involved at the end of the White
Paper.
https://cablela.bs/low-latency-docsis-technology-overview-february-2019
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
>>>
Broadcom has been deeply involved in developing the Low Latency DOCSIS
cable industry specification that Bob Briscoe mentioned. We concur with
the L4S use of ECT(1). L4S can be implemented either in a dual-queue or an
fq implementation. SCE cannot be implemented with a dual-queue
implementation; the only way to support it is with an fq implementation.
An fq implementation is incompatible with the Low Latency DOCSIS
specification developed within the cable industry.
Victor
[-- Attachment #2: Type: text/html, Size: 3173 bytes --]
^ permalink raw reply [flat|nested] 105+ messages in thread
end of thread, other threads:[~2019-03-27 17:52 UTC | newest]
Thread overview: 105+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <AM0PR07MB48198660539171737E4CCAB1E0730@AM0PR07MB4819.eurprd07.prod.outlook.com>
[not found] ` <d91a6a71-5898-9571-2a02-0d9d83839615@bobbriscoe.net>
2019-03-15 10:46 ` [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104 Dave Taht
2019-03-15 13:01 ` Sebastian Moeller
2019-03-15 14:06 ` Dave Taht
2019-03-15 15:52 ` Sebastian Moeller
2019-03-15 17:01 ` [Bloat] [Ecn-sane] " David P. Reed
2019-03-15 17:45 ` Sebastian Moeller
2019-03-15 18:36 ` Mikael Abrahamsson
2019-03-15 19:23 ` Sebastian Moeller
2019-03-15 19:32 ` Jonathan Morton
2019-03-15 19:44 ` David P. Reed
2019-03-15 20:13 ` Jonathan Morton
2019-03-15 23:43 ` David P. Reed
2019-03-16 1:26 ` Jonathan Morton
2019-03-16 7:38 ` Sebastian Moeller
2019-03-16 18:56 ` Michael Richardson
2019-03-15 20:28 ` Jonathan Foulkes
2019-03-15 20:31 ` Dave Taht
2019-03-15 23:45 ` David P. Reed
2019-03-16 9:42 ` Michael Welzl
2019-03-16 10:08 ` Sebastian Moeller
2019-03-16 10:23 ` Nils Andreas Svee
2019-03-16 14:55 ` Jonathan Foulkes
2019-03-16 21:38 ` Holland, Jake
2019-03-16 21:57 ` Vint Cerf
2019-03-16 22:03 ` Dave Taht
2019-03-16 22:05 ` Holland, Jake
2019-03-17 18:07 ` David P. Reed
2019-03-17 18:05 ` Vint Cerf
2019-03-19 1:06 ` Bob Briscoe
2019-03-19 3:18 ` Dave Taht
2019-03-20 19:04 ` Holland, Jake
2019-03-20 19:58 ` Stephen Hemminger
2019-03-20 20:05 ` Holland, Jake
[not found] ` <5C9296E1.4010703@erg.abdn.ac.uk>
2019-03-20 20:00 ` [Bloat] [tsvwg] " Holland, Jake
2019-03-20 20:05 ` Jonathan Morton
2019-03-20 20:55 ` Greg White
2019-03-20 22:12 ` Sebastian Moeller
2019-03-20 22:31 ` Jonathan Morton
2019-03-20 22:56 ` Sebastian Moeller
2019-03-20 23:03 ` Jonathan Morton
2019-03-20 23:11 ` Holland, Jake
2019-03-20 23:28 ` Jonathan Morton
2019-03-21 8:15 ` [Bloat] [Ecn-sane] [tsvwg] " Mikael Abrahamsson
2019-03-21 8:31 ` Mikael Abrahamsson
2019-03-20 23:30 ` [Bloat] [tsvwg] [Ecn-sane] " Sebastian Moeller
2019-03-21 0:15 ` Holland, Jake
2019-03-21 0:41 ` Holland, Jake
2019-03-20 21:48 ` [Bloat] " Greg White
2019-03-20 21:56 ` Jonathan Morton
2019-03-20 22:38 ` Holland, Jake
2019-03-20 22:56 ` Greg White
2019-03-20 23:29 ` Bob Briscoe
2019-03-20 23:51 ` Jonathan Morton
2019-03-21 6:04 ` Bob Briscoe
2019-03-21 7:46 ` Jonathan Morton
2019-03-21 8:02 ` Bob Briscoe
2019-03-21 8:49 ` Bless, Roland (TM)
2019-03-21 13:24 ` Bob Briscoe
2019-03-22 12:53 ` Bless, Roland (TM)
2019-03-25 2:47 ` Bob Briscoe
2019-03-21 8:45 ` Sebastian Moeller
2019-03-24 20:15 ` alex.burr
2019-03-25 1:34 ` Bob Briscoe
2019-03-27 17:52 ` Alex Burr
2019-03-19 4:44 ` Greg White
2019-03-19 5:35 ` Jonathan Morton
2019-03-19 5:52 ` Greg White
2019-03-19 7:10 ` Jonathan Morton
2019-03-19 8:07 ` Sebastian Moeller
2019-03-19 8:50 ` Sebastian Moeller
2019-03-19 23:59 ` Dave Taht
2019-03-20 10:17 ` Sebastian Moeller
2019-03-16 22:03 ` Jonathan Morton
2019-03-16 22:09 ` Sebastian Moeller
2019-03-17 14:06 ` Mikael Abrahamsson
2019-03-17 17:37 ` Loganaden Velvindron
2019-03-17 17:40 ` Toke Høiland-Jørgensen
2019-03-17 17:44 ` Mikael Abrahamsson
2019-03-17 18:00 ` Dave Taht
2019-03-17 19:38 ` Rodney W. Grimes
2019-03-17 20:50 ` Luca Muscariello
2019-03-17 21:51 ` Toke Høiland-Jørgensen
2019-03-18 4:26 ` Mikael Abrahamsson
2019-03-16 4:04 ` Jonathan Morton
2019-03-16 4:51 ` Dave Taht
2019-03-15 18:07 ` Mikael Abrahamsson
2019-03-15 14:27 ` [Bloat] " Jonathan Morton
2019-03-15 14:44 ` Sebastian Moeller
2019-03-15 15:49 ` Jonathan Morton
2019-03-15 21:34 ` Wesley Eddy
2019-03-22 18:28 [Bloat] [Ecn-sane] " Victor Hou
2019-03-23 8:02 ` Roland Bless
2019-03-23 8:54 ` Luca Muscariello
2019-03-23 10:02 ` Mikael Abrahamsson
2019-03-23 15:03 ` Jonathan Morton
2019-03-23 19:52 ` Roland Bless
2019-03-23 15:19 ` Roland Bless
2019-03-23 17:16 ` Mikael Abrahamsson
2019-03-23 19:45 ` Roland Bless
2019-03-23 17:48 ` Michael Welzl
2019-03-23 18:31 ` Luca Muscariello
2019-03-23 18:40 ` Mikael Abrahamsson
2019-03-23 19:11 ` Michael Welzl
2019-03-23 21:04 ` Luca Muscariello
2019-03-23 19:55 ` Roland Bless
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox