* [Starlink] on using and debugging lossless congestion control (RFC3168 ECN)
@ 2021-06-21 8:41 Dave Taht
[not found] ` <20160.1624287984@localhost>
0 siblings, 1 reply; 5+ messages in thread
From: Dave Taht @ 2021-06-21 8:41 UTC (permalink / raw)
To: starlink
It came up recently that a lot of folk don't like the idea of dropping
packets as a congestion
control mechanism. Like loss or not... the fundamental paper on this
subject is Van Jacobson and Mike Karels:
"On congestion avoidance and control"
http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf
which is a really great read, IMHO. A classic. I like telling the
backstory a lot. "The Internet was down!"
Certainly as RTTs grow, the cost of recovering a lost packet also grows,
and ECN was designed to help out greatly at planet girdling RTTs for
interactive traffic.
Explicit congestion notification (ECN, RFC3168) has a long history
going back to the mid-80s
and I'd like to invite kk in for a talk about it. Sally Floyd (sadly
now deceased)
also wrote many cogent papers on it as the concept evolved that are
worth reading.
The RFC3168 mechanism is that: you mark packets as ecn-capable ECT(0)
in the IP header, and an ECN-capable aqm along the path can mark as
"CE", rather than drop, packets, when
the flow needs to be signaled to slow down (at the time, by half) ECN
was essentially co-designed with the RED algorithm back in the early
90s but took a decade longer to be standardized, and worse, when first
tried out over the broader internet, twiddling these bit crashed some
major routers so it was turned off and sometimes bleached out. More
active testing resumed in 2011...
anyway... RFC3168 defines that a mark should be treated the same as a
drop by the endpoint (but it's more complicated than that)
Both sch_cake and fq_codel enable RFC3168-style handling of ecn *by default*.
fq_codel is the nearly universal qdisc in most major linux distribution kernels.
FQ_codel for wifi also enables it by default but in a somewhat limited
fashion in older linux releases.[1]
All the implementations of these algorithms keep statistics. TCP_INFO
in linux, also, returns whether
the CE mark has been seen.
Wireshark can be used to tear apart captures and tcptrace -G/xplot.org
can also see marked packets.
Most endpoints will serve ECN-enabled requests. Enabling ecn support
on an endpoint in both directions
is a single sysctl. [0]
Apple products request ECN only sometimes nowadays however, there is a
sysctl to disable the heuristic for this
so you can have it on all the time.
Anyway...
I often use ECN marks merely to determine if an AQM is working
properly. Loss should then be
coming from something else. For a extreme example of this usage, see:
http://blog.cerowrt.org/post/crypto_fq_bug/
Having RFC3168 ecn enabled on your path and on your host machine is
the only way to get an A+ score on the dslreports bufferbloat test.
It.just.works with every major operating system on the planet if enabled.
Lastly.
The future of ECN debate over changes to RFC3168, L4S and SCE in the
ietf tsvwg working group has been the least fun debate of my entire
life, but I don't want to talk about that today. I have a small fix to
the RFC3168 compliant behavior of cubic that I am liking a lot in
simulation and long to test at scale.
I do encourage people to try out ECN as it exists today and see if
they can measure any
difference in behaviors induced. Years ago we put it into "mosh" for
example, so for interactive traffic
we do not lose those packets (and we put in a vastly more robust
response than RFC3168 - cutting the mosh frame rate from a max of
60fps to 1 or 2FPS on a CE)
[0] https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/
[1] Other linux mainline AQMs with ECN support include RED, codel, pie
& fq-pie, but off by default.
--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
Dave Täht CTO, TekLibre, LLC
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Starlink] on using and debugging lossless congestion control (RFC3168 ECN)
[not found] ` <20160.1624287984@localhost>
@ 2021-06-21 15:47 ` Dave Taht
2021-06-21 15:56 ` Dave Taht
0 siblings, 1 reply; 5+ messages in thread
From: Dave Taht @ 2021-06-21 15:47 UTC (permalink / raw)
To: Michael Richardson, starlink
On Mon, Jun 21, 2021 at 8:06 AM Michael Richardson <mcr@sandelman.ca> wrote:
>
> Dave Taht <dave.taht@gmail.com> wrote:
> > It came up recently that a lot of folk don't like the idea of dropping
> > packets as a congestion
> > control mechanism. Like loss or not... the fundamental paper on this
> > subject is Van Jacobson and Mike Karels:
>
> > "On congestion avoidance and control"
>
> > http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf
>
> I saved this post a long time ago.
> http://www.sandelman.ca/mcr/humour/vanj-enet-posting.txt
That was a *great* piece of history thx. And he referenced greg
chesson's work on XTP,
( https://grainger.illinois.edu/alumni/distinguished/9560 )
Greg was an early consultant to the bufferbloat project...
he sadly passed before we ever had a chance to jam together.
https://news.ycombinator.com/item?id=9795462
> There is another one about how he crashed Interop, which I don't know where I
> put.
More folk should have seen van's codel preso at ietf 84. It was the
very very early days of webrtc.
https://archive.org/details/video1_20191129
>
> --
> ] Never tell me the odds! | ipv6 mesh networks [
> ] Michael Richardson, Sandelman Software Works | IoT architect [
> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>
>
>
--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
Dave Täht CTO, TekLibre, LLC
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Starlink] on using and debugging lossless congestion control (RFC3168 ECN)
2021-06-21 15:47 ` Dave Taht
@ 2021-06-21 15:56 ` Dave Taht
2021-06-23 1:20 ` Doc Searls
0 siblings, 1 reply; 5+ messages in thread
From: Dave Taht @ 2021-06-21 15:56 UTC (permalink / raw)
To: Michael Richardson, starlink
On Mon, Jun 21, 2021 at 8:47 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Jun 21, 2021 at 8:06 AM Michael Richardson <mcr@sandelman.ca> wrote:
> >
> > Dave Taht <dave.taht@gmail.com> wrote:
> > > It came up recently that a lot of folk don't like the idea of dropping
> > > packets as a congestion
> > > control mechanism. Like loss or not... the fundamental paper on this
> > > subject is Van Jacobson and Mike Karels:
> >
> > > "On congestion avoidance and control"
> >
> > > http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf
> >
> > I saved this post a long time ago.
> > http://www.sandelman.ca/mcr/humour/vanj-enet-posting.txt
From that piece:
"funding agents looking for high speed,
next-generation networks may forget that research to make slow
things go fast sometimes makes fast things go faster".
Recently I dusted off fq_codel_fast in light of the cache misses
finally showing up
in a 100gbit dual bonded scenario over here:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=848ca9182a7d25bb54955c3aab9a3a2742bf9678
asked the author to give it a try.
> That was a *great* piece of history thx. And he referenced greg
> chesson's work on XTP,
> ( https://grainger.illinois.edu/alumni/distinguished/9560 )
>
> Greg was an early consultant to the bufferbloat project...
> he sadly passed before we ever had a chance to jam together.
>
> https://news.ycombinator.com/item?id=9795462
>
> > There is another one about how he crashed Interop, which I don't know where I
> > put.
>
> More folk should have seen van's codel preso at ietf 84. It was the
> very very early days of webrtc.
>
> https://archive.org/details/video1_20191129
> >
> > --
> > ] Never tell me the odds! | ipv6 mesh networks [
> > ] Michael Richardson, Sandelman Software Works | IoT architect [
> > ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
> >
> >
> >
>
>
> --
> Latest Podcast:
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>
> Dave Täht CTO, TekLibre, LLC
--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
Dave Täht CTO, TekLibre, LLC
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Starlink] on using and debugging lossless congestion control (RFC3168 ECN)
2021-06-21 15:56 ` Dave Taht
@ 2021-06-23 1:20 ` Doc Searls
2021-06-23 18:18 ` [Starlink] Q Dave Taht
0 siblings, 1 reply; 5+ messages in thread
From: Doc Searls @ 2021-06-23 1:20 UTC (permalink / raw)
To: Dave Taht; +Cc: Michael Richardson, starlink
[-- Attachment #1: Type: text/plain, Size: 3263 bytes --]
When you come on FLOSS Weekly, Jonathan here will be my co-host:
https://hackaday.com/2021/05/24/starlink-a-review-and-some-hacks/
Never mind that he doesn't know about bufferbloat on Starlink. He'll get it. And he's a great dude. You should loop him in. He can be helpful. So can his minions.
Doc
> On Jun 21, 2021, at 11:56 AM, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Jun 21, 2021 at 8:47 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
>>
>> On Mon, Jun 21, 2021 at 8:06 AM Michael Richardson <mcr@sandelman.ca> wrote:
>>>
>>> Dave Taht <dave.taht@gmail.com> wrote:
>>>> It came up recently that a lot of folk don't like the idea of dropping
>>>> packets as a congestion
>>>> control mechanism. Like loss or not... the fundamental paper on this
>>>> subject is Van Jacobson and Mike Karels:
>>>
>>>> "On congestion avoidance and control"
>>>
>>>> http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf
>>>
>>> I saved this post a long time ago.
>>> http://www.sandelman.ca/mcr/humour/vanj-enet-posting.txt
>
> From that piece:
>
> "funding agents looking for high speed,
> next-generation networks may forget that research to make slow
> things go fast sometimes makes fast things go faster".
>
> Recently I dusted off fq_codel_fast in light of the cache misses
> finally showing up
> in a 100gbit dual bonded scenario over here:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=848ca9182a7d25bb54955c3aab9a3a2742bf9678 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=848ca9182a7d25bb54955c3aab9a3a2742bf9678>
>
> asked the author to give it a try.
>
>
>
>
>
>> That was a *great* piece of history thx. And he referenced greg
>> chesson's work on XTP,
>> ( https://grainger.illinois.edu/alumni/distinguished/9560 )
>>
>> Greg was an early consultant to the bufferbloat project...
>> he sadly passed before we ever had a chance to jam together.
>>
>> https://news.ycombinator.com/item?id=9795462
>>
>>> There is another one about how he crashed Interop, which I don't know where I
>>> put.
>>
>> More folk should have seen van's codel preso at ietf 84. It was the
>> very very early days of webrtc.
>>
>> https://archive.org/details/video1_20191129
>>>
>>> --
>>> ] Never tell me the odds! | ipv6 mesh networks [
>>> ] Michael Richardson, Sandelman Software Works | IoT architect [
>>> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>>>
>>>
>>>
>>
>>
>> --
>> Latest Podcast:
>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>>
>> Dave Täht CTO, TekLibre, LLC
>
>
>
> --
> Latest Podcast:
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>
> Dave Täht CTO, TekLibre, LLC
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
[-- Attachment #2: Type: text/html, Size: 23876 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* [Starlink] Q
2021-06-23 1:20 ` Doc Searls
@ 2021-06-23 18:18 ` Dave Taht
0 siblings, 0 replies; 5+ messages in thread
From: Dave Taht @ 2021-06-23 18:18 UTC (permalink / raw)
To: Doc Searls; +Cc: Dave Taht, starlink
[-- Attachment #1: Type: text/plain, Size: 4823 bytes --]
dear doc:
great article, thx, he’s already most of the way to the right page. Can you introduce us off list?
one hope is we do failover better using the mwan3 tool in openwrt instead of relying on ping but the supplied statistics.
Another is to get the starwrt codebase up on the supplied router. We already have plenty of qcom IPQ8014 routers under test in openwrt mainline, which is the chipset the starlink router is based on….
… but most of our new work had shifted over the mt76 chipsets and the higher end armada platfoems and there are three large things involving aql, wifi retries, and cake left to make work on the ath10k derived stuff that I’d hoped would make the final build...
public work over here stopped a while back, but we can resume.
https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002
> On Jun 22, 2021, at 6:20 PM, Doc Searls <doc@searls.com> wrote:
>
> When you come on FLOSS Weekly, Jonathan here will be my co-host:
>
> https://hackaday.com/2021/05/24/starlink-a-review-and-some-hacks/ <https://hackaday.com/2021/05/24/starlink-a-review-and-some-hacks/>
>
> Never mind that he doesn't know about bufferbloat on Starlink. He'll get it. And he's a great dude. You should loop him in. He can be helpful. So can his minions.
>
> Doc
>
>> On Jun 21, 2021, at 11:56 AM, Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
>>
>> On Mon, Jun 21, 2021 at 8:47 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
>>>
>>> On Mon, Jun 21, 2021 at 8:06 AM Michael Richardson <mcr@sandelman.ca <mailto:mcr@sandelman.ca>> wrote:
>>>>
>>>> Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
>>>>> It came up recently that a lot of folk don't like the idea of dropping
>>>>> packets as a congestion
>>>>> control mechanism. Like loss or not... the fundamental paper on this
>>>>> subject is Van Jacobson and Mike Karels:
>>>>
>>>>> "On congestion avoidance and control"
>>>>
>>>>> http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf <http://web.stanford.edu/class/cs244/papers/CongestionControl.pdf>
>>>>
>>>> I saved this post a long time ago.
>>>> http://www.sandelman.ca/mcr/humour/vanj-enet-posting.txt <http://www.sandelman.ca/mcr/humour/vanj-enet-posting.txt>
>>
>> From that piece:
>>
>> "funding agents looking for high speed,
>> next-generation networks may forget that research to make slow
>> things go fast sometimes makes fast things go faster".
>>
>> Recently I dusted off fq_codel_fast in light of the cache misses
>> finally showing up
>> in a 100gbit dual bonded scenario over here:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=848ca9182a7d25bb54955c3aab9a3a2742bf9678 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=848ca9182a7d25bb54955c3aab9a3a2742bf9678>
>>
>> asked the author to give it a try.
>>
>>
>>
>>
>>
>>> That was a *great* piece of history thx. And he referenced greg
>>> chesson's work on XTP,
>>> ( https://grainger.illinois.edu/alumni/distinguished/9560 <https://grainger.illinois.edu/alumni/distinguished/9560> )
>>>
>>> Greg was an early consultant to the bufferbloat project...
>>> he sadly passed before we ever had a chance to jam together.
>>>
>>> https://news.ycombinator.com/item?id=9795462 <https://news.ycombinator.com/item?id=9795462>
>>>
>>>> There is another one about how he crashed Interop, which I don't know where I
>>>> put.
>>>
>>> More folk should have seen van's codel preso at ietf 84. It was the
>>> very very early days of webrtc.
>>>
>>> https://archive.org/details/video1_20191129
>>>>
>>>> --
>>>> ] Never tell me the odds! | ipv6 mesh networks [
>>>> ] Michael Richardson, Sandelman Software Works | IoT architect [
>>>> ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Latest Podcast:
>>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>>>
>>> Dave Täht CTO, TekLibre, LLC
>>
>>
>>
>> --
>> Latest Podcast:
>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>>
>> Dave Täht CTO, TekLibre, LLC
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
>> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
[-- Attachment #2: Type: text/html, Size: 25858 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-06-23 18:18 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-21 8:41 [Starlink] on using and debugging lossless congestion control (RFC3168 ECN) Dave Taht
[not found] ` <20160.1624287984@localhost>
2021-06-21 15:47 ` Dave Taht
2021-06-21 15:56 ` Dave Taht
2021-06-23 1:20 ` Doc Searls
2021-06-23 18:18 ` [Starlink] Q Dave Taht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox