[Bloat] Taxonomy of various sender-side TCPs

[Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

There's a lot of literature about how various TCP congestion-control algorithms compare to each other, but there's not much in the way of 10,000 foot overviews.  So this is an attempt to bring the most relevant data into one place.

I've sorted some common TCPs into groups in roughly ascending order of aggressiveness in the face of congestion, and considered how they might react to the introduction of ECN at the bottleneck.

1: Vegas.
Strategy:  Fill the pipe, not the buffers.
Reaction to congestion:  Backs off until RTT approaches baseline, ie. buffers empty.
Probable reaction to 1% ECN marking:  Very little (but it doesn't need to).

2: Illinois, Compound.
Strategy:  Fill the pipe quickly, then probe the buffer slowly to avoid being outcompeted.
Reaction to congestion:  Backs off firmly on packet loss, then quickly re-probes for the pipe.
Probable reaction to 1% ECN marking:  Window should stabilise near pipe length.

3: Reno (and subtle variations thereof).
Strategy:  Avoid congestion collapse in a simple, standardisable way.
Reaction to congestion:  Halves the window on packet loss.
Probable reaction to 1% ECN marking:  Window should stabilise near 100 packets.

4: Veno, Westwood+.
Strategy:  Distinguish congestion loss from random channel loss.
Reaction to congestion:  Identical to Reno.
Probable reaction to 1% ECN marking:  Identical to Reno.

5: CUBIC.
Strategy:  Fill pipe and all buffers to capacity.  This is the most aggressive widely-deployed TCP.
Reaction to congestion:  On packet loss, quickly re-probes for buffer capacity.
Probable reaction to 1% ECN marking:  UNKNOWN.

Currently, CUBIC is the default TCP send-side algorithm in Linux.  It seems likely that it will react correctly to ECN marking, but that a higher rate of marking may be needed to bring it down to a given buffering level.  From what I've read, SFB should be able to probe for the correct marking rate on a per-flow basis, which is nice.

On the subject of ECN, my impression is that YouTube currently doesn't enable it, but a one-man company I recently downloaded some stuff from does.  I wonder if there's any reliable data on how many of the most popular sites enable ECN if you ask for it.  Personally, I think IPv6 and ECN should probably go together - v6 gear is new or upgraded anyway so there shouldn't be any legacy problems.

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Stephen Hemminger]

On Fri, 11 Mar 2011 09:24:33 +0200
Jonathan Morton <chromatix99@gmail.com> wrote:

> There's a lot of literature about how various TCP congestion-control algorithms compare to each other, but there's not much in the way of 10,000 foot overviews.  So this is an attempt to bring the most relevant data into one place.

The literature often doesn't tell what is wrong with each one.

> 1: Vegas.
> Strategy:  Fill the pipe, not the buffers.
> Reaction to congestion:  Backs off until RTT approaches baseline, ie. buffers empty.
> Probable reaction to 1% ECN marking:  Very little (but it doesn't need to).
Weaknesess: Sensitive to variations in RTT and reverse path congestion; underperforms
when competing against loss based congestion control. Does not detect increase in
bandwidth ie recovers slowly from congestion clearing.


> 2: Illinois, Compound
(Add in Yeah, Veno, and TCP-NV )
> Strategy:  Fill the pipe quickly, then probe the buffer slowly to avoid being outcompeted.
> Reaction to congestion:  Backs off firmly on packet loss, then quickly re-probes for the pipe.
> Probable reaction to 1% ECN marking:  Window should stabilise near pipe length.
Weaknesses: Not widely tested. Microsoft disabled Compound.


> 3: Reno (and subtle variations thereof).
> Strategy:  Avoid congestion collapse in a simple, standardisable way.
> Reaction to congestion:  Halves the window on packet loss.
> Probable reaction to 1% ECN marking:  Window should stabilise near 100 packets.
Weaknesses: Unstable at higher speed and long delay links.


> 4: Westwood+.
> Strategy:  Distinguish congestion loss from random channel loss.
> Reaction to congestion:  Identical to Reno.
> Probable reaction to 1% ECN marking:  Identical to Reno.


> 5: CUBIC.
> Strategy:  Fill pipe and all buffers to capacity.  This is the most aggressive widely-deployed TCP.
> Reaction to congestion:  On packet loss, quickly re-probes for buffer capacity.
> Probable reaction to 1% ECN marking:  UNKNOWN.

Also any algorithm that uses RTT estimation ends up having to use higher (sub tick)
resolution time stamps. On many platforms this is cheap (TSC 0, HPET 1usec) but on
older hardware or other architectures it can be expensive (several usec's per packet).

> 
> Currently, CUBIC is the default TCP send-side algorithm in Linux.  It seems likely that it will react correctly to ECN marking, but that a higher rate of marking may be needed to bring it down to a given buffering level.  From what I've read, SFB should be able to probe for the correct marking rate on a per-flow basis, which is nice.
> 
> On the subject of ECN, my impression is that YouTube currently doesn't enable it, but a one-man company I recently downloaded some stuff from does.  I wonder if there's any reliable data on how many of the most popular sites enable ECN if you ask for it.  Personally, I think IPv6 and ECN should probably go together - v6 gear is new or upgraded anyway so there shouldn't be any legacy problems.

Even TCP window scaling is problematic. Many consumer bits of gear are seriously
broken. I have had to turn off WS on my laptop to deal with hotel and conference
wireless front ends.


-- 

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Täht]

There may be a new contender on the block, "TCP-fit", which is
"a novel congestion control algorithm targeting both emerging
wireless networks such as LTE, WiMax, Wi-Fi and HSPA and high speed long
delay (high BDP) networks. "

The knobs they are twisting seem to be appropriate for these sorts of
networks, and the results very interesting.

See paper at:

http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/papers/mobicom10_demo.pdf

And more details at:

http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/

(I've cc'd the authors in the hope they can tell us more about it.)

Stephen Hemminger <shemminger@vyatta.com> writes:

> On Fri, 11 Mar 2011 09:24:33 +0200
> Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> There's a lot of literature about how various TCP congestion-control algorithms compare to each other, but there's not much in the way of 10,000 foot overviews.  So this is an attempt to bring the most relevant data into one place.
>
> The literature often doesn't tell what is wrong with each one.
>
>> 1: Vegas.
>> Strategy:  Fill the pipe, not the buffers.
>> Reaction to congestion:  Backs off until RTT approaches baseline, ie. buffers empty.
>> Probable reaction to 1% ECN marking:  Very little (but it doesn't need to).
> Weaknesess: Sensitive to variations in RTT and reverse path congestion; underperforms
> when competing against loss based congestion control. Does not detect increase in
> bandwidth ie recovers slowly from congestion clearing.
>
>
>> 2: Illinois, Compound
> (Add in Yeah, Veno, and TCP-NV )
>> Strategy:  Fill the pipe quickly, then probe the buffer slowly to avoid being outcompeted.
>> Reaction to congestion:  Backs off firmly on packet loss, then quickly re-probes for the pipe.
>> Probable reaction to 1% ECN marking:  Window should stabilise near pipe length.
> Weaknesses: Not widely tested. Microsoft disabled Compound.
>
>
>> 3: Reno (and subtle variations thereof).
>> Strategy:  Avoid congestion collapse in a simple, standardisable way.
>> Reaction to congestion:  Halves the window on packet loss.
>> Probable reaction to 1% ECN marking:  Window should stabilise near 100 packets.
> Weaknesses: Unstable at higher speed and long delay links.
>
>
>> 4: Westwood+.
>> Strategy:  Distinguish congestion loss from random channel loss.
>> Reaction to congestion:  Identical to Reno.
>> Probable reaction to 1% ECN marking:  Identical to Reno.
>
>
>> 5: CUBIC.
>> Strategy:  Fill pipe and all buffers to capacity.  This is the most aggressive widely-deployed TCP.
>> Reaction to congestion:  On packet loss, quickly re-probes for buffer capacity.
>> Probable reaction to 1% ECN marking:  UNKNOWN.
>
> Also any algorithm that uses RTT estimation ends up having to use higher (sub tick)
> resolution time stamps. On many platforms this is cheap (TSC 0, HPET 1usec) but on
> older hardware or other architectures it can be expensive (several usec's per packet).
>
>> 
>> Currently, CUBIC is the default TCP send-side algorithm in Linux.  It seems likely that it will react correctly to ECN marking, but that a higher rate of marking may be needed to bring it down to a given buffering level.  From what I've read, SFB should be able to probe for the correct marking rate on a per-flow basis, which is nice.
>> 
>> On the subject of ECN, my impression is that YouTube currently doesn't enable it, but a one-man company I recently downloaded some stuff from does.  I wonder if there's any reliable data on how many of the most popular sites enable ECN if you ask for it.  Personally, I think IPv6 and ECN should probably go together - v6 gear is new or upgraded anyway so there shouldn't be any legacy problems.
>
> Even TCP window scaling is problematic. Many consumer bits of gear are seriously
> broken. I have had to turn off WS on my laptop to deal with hotel and conference
> wireless front ends.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 11 Mar, 2011, at 7:57 pm, Dave Täht wrote:

> There may be a new contender on the block, "TCP-fit", which is
> "a novel congestion control algorithm targeting both emerging
> wireless networks such as LTE, WiMax, Wi-Fi and HSPA and high speed long
> delay (high BDP) networks. "
> 
> The knobs they are twisting seem to be appropriate for these sorts of
> networks, and the results very interesting.
> 
> See paper at:
> 
> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/papers/mobicom10_demo.pdf
> 
> And more details at:
> 
> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/
> 
> (I've cc'd the authors in the hope they can tell us more about it.)

It would probably help if their DNS worked.  :-(

Actually, the public server at 4.2.2.2 can resolve them, but my home box can't.  An artefact of the Great Firewall perhaps?

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 11 Mar, 2011, at 7:57 pm, Dave Täht wrote:
>
>> There may be a new contender on the block, "TCP-fit", which is
>> "a novel congestion control algorithm targeting both emerging
>> wireless networks such as LTE, WiMax, Wi-Fi and HSPA and high speed long
>> delay (high BDP) networks. "
>> 
>> The knobs they are twisting seem to be appropriate for these sorts of
>> networks, and the results very interesting.
>> 
>> See paper at:
>> 
>> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/papers/mobicom10_demo.pdf
>> 
>> And more details at:
>> 
>> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/
>> 
>> (I've cc'd the authors in the hope they can tell us more about it.)
>
> It would probably help if their DNS worked.  :-(
>
> Actually, the public server at 4.2.2.2 can resolve them, but my home
> box can't.  An artefact of the Great Firewall perhaps?

I've just now mirrored that paper onto:

http://mirrors.bufferbloat.net/RelevantPapers/

They seem to have several other papers in the queue, as yet unpublished.

(If anyone needs other papers mirrored let me know)

>
>  - Jonathan
>

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Taxonomy of various sender-side TCPs

[Erica Han]

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

Dear all,

We are happy to provide a preprint of our papers (after Beijing woke up).

In addition, I was wondering is that possible to have a quick conference
call, so that we can understand a little bit more about you and your
project, and how we can help.

At the moment, I am located in LA, and the other three authors located in
Beijing.

Thanks.

Erica


On Fri, Mar 11, 2011 at 10:12 AM, Dave Täht <d@taht.net> wrote:

> Jonathan Morton <chromatix99@gmail.com> writes:
>
> > On 11 Mar, 2011, at 7:57 pm, Dave Täht wrote:
> >
> >> There may be a new contender on the block, "TCP-fit", which is
> >> "a novel congestion control algorithm targeting both emerging
> >> wireless networks such as LTE, WiMax, Wi-Fi and HSPA and high speed long
> >> delay (high BDP) networks. "
> >>
> >> The knobs they are twisting seem to be appropriate for these sorts of
> >> networks, and the results very interesting.
> >>
> >> See paper at:
> >>
> >>
> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/papers/mobicom10_demo.pdf
> >>
> >> And more details at:
> >>
> >> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/
> >>
> >> (I've cc'd the authors in the hope they can tell us more about it.)
> >
> > It would probably help if their DNS worked.  :-(
> >
> > Actually, the public server at 4.2.2.2 can resolve them, but my home
> > box can't.  An artefact of the Great Firewall perhaps?
>
> I've just now mirrored that paper onto:
>
> http://mirrors.bufferbloat.net/RelevantPapers/
>
> They seem to have several other papers in the queue, as yet unpublished.
>
> (If anyone needs other papers mirrored let me know)
>
> >
> >  - Jonathan
> >
>
> --
> Dave Taht
> http://nex-6.taht.net
>

[-- Attachment #2: Type: text/html, Size: 2813 bytes --]

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Hart]

On Fri, Mar 11, 2011 at 18:07 UTC, Jonathan Morton
<chromatix99@gmail.com> wrote:
>> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/papers/mobicom10_demo.pdf
>> http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/
>
> It would probably help if their DNS worked.  :-(
>
> Actually, the public server at 4.2.2.2 can resolve them, but my home box can't.  An artefact of the Great Firewall perhaps?

After digging into this a bit I think there's a connectivity problem
to the cs.tsinghua.edu.cn nameservers from some networks and not
others.  I can't resolve the name when I use my own local recursive
nameserver, but I can using any of the google 8.8.8.8, 8.8.4.4 or
level3 4.2.2.1, 4.2.2.2, public DNS.  With each of those, I can see
from TTLs they are able to reach dns1/2.cs.tsinghua.edu.cn when I
can't.

Turning to http://lg.he.net, I initiated traceroutes from San Jose,
Singapore, and Tokyo.  The asian routers eventually got through the *
* * dead zone to show last hops and destination responding, but the
core1.sjc2.he.net traceroute died after two hops while still in the
states, similar to what I see via sprintlink in Seattle.

The reachability problem for the DNS servers may be related to today's quake.

Cheers,
Dave Hart

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 11 Mar, 2011, at 9:10 pm, Dave Hart wrote:

> The reachability problem for the DNS servers may be related to today's quake.

I suppose that might be true.  I intuitively expect traffic to take the shortest geographic path, but there's a lot of Siberia in that direction from here, so the network might be sparse enough to be considered the "long" way around.

A traceroute to the IP I eventually obtained stops in "pacnet-*-sjo-*.telia.net" (San Jose?).  It does seem likely that this would connect to Japan rather than directly to China.

In any case, I'm reading the copy from the mirror.

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Erica Han]

[-- Attachment #1.1: Type: text/plain, Size: 1070 bytes --]

Jonathan, in case you still can't access the website, attached is a pdf
version. Perhaps you will find it interesting in our real-time testing
results in 192 cities over 43 countries.

Dave, early next week for a conference call could work. I will coordinate
with the Beijing team and see what's their availability looks like.

Thanks.

Erica



On Fri, Mar 11, 2011 at 11:25 AM, Jonathan Morton <chromatix99@gmail.com>wrote:

>
> On 11 Mar, 2011, at 9:10 pm, Dave Hart wrote:
>
> > The reachability problem for the DNS servers may be related to today's
> quake.
>
> I suppose that might be true.  I intuitively expect traffic to take the
> shortest geographic path, but there's a lot of Siberia in that direction
> from here, so the network might be sparse enough to be considered the "long"
> way around.
>
> A traceroute to the IP I eventually obtained stops in "pacnet-*-sjo-*.
> telia.net" (San Jose?).  It does seem likely that this would connect to
> Japan rather than directly to China.
>
> In any case, I'm reading the copy from the mirror.
>
>  - Jonathan
>
>

[-- Attachment #1.2: Type: text/html, Size: 1594 bytes --]

[-- Attachment #2: TCP-FIT.pdf --]
[-- Type: application/pdf, Size: 3319307 bytes --]

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 11 Mar, 2011, at 10:34 pm, Erica Han wrote:

> Jonathan, in case you still can't access the website, attached is a pdf version. Perhaps you will find it interesting in our real-time testing results in 192 cities over 43 countries. 

This does look intriguing, with very impressive throughput numbers compared to the other TCPs.

However, I don't see much detail on how it actually does this, and it would also be very valuable to compare the latency under load.  Is there a paper which describes the algorithm in detail?

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Stephen Hemminger]

On Fri, 11 Mar 2011 22:46:50 +0200
Jonathan Morton <chromatix99@gmail.com> wrote:

> 
> On 11 Mar, 2011, at 10:34 pm, Erica Han wrote:
> 
> > Jonathan, in case you still can't access the website, attached is a pdf version. Perhaps you will find it interesting in our real-time testing results in 192 cities over 43 countries. 
> 
> This does look intriguing, with very impressive throughput numbers compared to the other TCPs.
> 
> However, I don't see much detail on how it actually does this, and it would also be very valuable to compare the latency under load.  Is there a paper which describes the algorithm in detail?
> 
>  - Jonathan

I bugged then at Linux Plumber's to submit it to get into the upstream
kernel, but seems to be stuck in internal testing?

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 12 Mar, 2011, at 12:20 am, Stephen Hemminger wrote:

> I bugged then at Linux Plumber's to submit it to get into the upstream
> kernel, but seems to be stuck in internal testing?

It appears that they want to get a patent on it.  Hence they do have to keep their cards quite close for the moment, if they don't want to compromise that process.

Of course I have opinions on software patents - happily they are not a massive problem in Finland - but merely being patented isn't a complete deal-breaker for an algorithm to go into Linux, provided it has an appropriately liberal licensing scheme (ie. GPL compatible).

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Stephen Hemminger]

On Sat, 12 Mar 2011 02:13:20 +0200
Jonathan Morton <chromatix99@gmail.com> wrote:

> 
> On 12 Mar, 2011, at 12:20 am, Stephen Hemminger wrote:
> 
> > I bugged then at Linux Plumber's to submit it to get into the upstream
> > kernel, but seems to be stuck in internal testing?
> 
> It appears that they want to get a patent on it.  Hence they do have to keep their cards quite close for the moment, if they don't want to compromise that process.
> 
> Of course I have opinions on software patents - happily they are not a massive problem in Finland - but merely being patented isn't a complete deal-breaker for an algorithm to go into Linux, provided it has an appropriately liberal licensing scheme (ie. GPL compatible).
> 
>  - Jonathan
> 

Never mind then, it sounds like another TCP FAST!

-- 

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 11 Mar, 2011, at 7:21 pm, Stephen Hemminger wrote:

>> 2: Illinois, Compound
> (Add in Yeah, Veno, and TCP-NV )
>> Strategy:  Fill the pipe quickly, then probe the buffer slowly to avoid being outcompeted.

I separated Veno to put it with Westwood, because there is almost no difference between Veno and Westwood, and they do not advance more rapidly than Reno in the uncongested regime.  I consider those two more aggressive than Reno only because they back off less if they believe the link is not congested, and that belief has a chance of being incorrect.

Meanwhile I considered Illinois and Compound to be less aggressive than Reno because they explicitly advance less rapidly in the congested regime, and back off at least as effectively as Reno does (well, the Linux implementation of Compound - it seems M$ screwed up their version).

I haven't read up on Yeah and NV (or TCP-fit) yet, so I can't classify them.  I'm also aware that a number of other aggressive TCPs designed for LFN throughput exist, but I can't be bothered enumerating them because they are useless - obsoleted by CUBIC in their intended application, and too aggressive for the wider Internet.

I do agree that TCPs which attempt to probe for minRTT have a weakness in this area, and Vegas is particularly weak because it relies very heavily on minRTT and is also very timid.  If the whole Internet used Vegas this wouldn't be a problem, but it gets outcompeted badly by literally everything else, especially the extra-aggressive TCPs like CUBIC.  So simply changing the send-side TCP congestion algorithm is insufficient to solve the whole problem, though some classes of users are seeing benefits from using Vegas despite it's flaws.

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 11 Mar, 2011, at 7:21 pm, Stephen Hemminger wrote:
>
>>> 2: Illinois, Compound
>> (Add in Yeah, Veno, and TCP-NV )
>>> Strategy: Fill the pipe quickly, then probe the buffer slowly to
>>> avoid being outcompeted.
>
> I separated Veno to put it with Westwood, because there is almost no
> difference between Veno and Westwood, and they do not advance more
> rapidly than Reno in the uncongested regime.  I consider those two
> more aggressive than Reno only because they back off less if they
> believe the link is not congested, and that belief has a chance of
> being incorrect.
>
> Meanwhile I considered Illinois and Compound to be less aggressive
> than Reno because they explicitly advance less rapidly in the
> congested regime, and back off at least as effectively as Reno does
> (well, the Linux implementation of Compound - it seems M$ screwed up
> their version).
>
> I haven't read up on Yeah and NV (or TCP-fit) yet, so I can't classify
> them.  I'm also aware that a number of other aggressive TCPs designed
> for LFN throughput exist, but I can't be bothered enumerating them
> because they are useless - obsoleted by CUBIC in their intended
> application, and too aggressive for the wider Internet.
>
> I do agree that TCPs which attempt to probe for minRTT have a weakness
> in this area, and Vegas is particularly weak because it relies very
> heavily on minRTT and is also very timid.  If the whole Internet used
> Vegas this wouldn't be a problem, but it gets outcompeted badly by
> literally everything else, especially the extra-aggressive TCPs like
> CUBIC.  So simply changing the send-side TCP congestion algorithm is
> insufficient to solve the whole problem, though some classes of users
> are seeing benefits from using Vegas despite it's flaws.

Several freeswitch (VOIP) folk said they were using Vegas for the TCP
side of their servers due to it's lower impact for command and control
applications while lots of RTP streams were running. One claimed that
Vegas is what google uses, but that's unconfirmed. Discussion here
(including recording)

http://www.bufferbloat.net/projects/bloat/wiki/Bufferbloat_and_Freeswitch_Conference_Call_March_9

Also, for those new to the list, there was a huge LEDBAT thread here
earlier and interest in embedding that into the kernel.

https://lists.bufferbloat.net/pipermail/bloat/2011-February/000025.html

And split-tcp solutions were discussed here:

https://lists.bufferbloat.net/pipermail/bloat/2011-February/000068.html

>
>  - Jonathan
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Täht]

Stephen Hemminger <shemminger@vyatta.com> writes:

> On Fri, 11 Mar 2011 09:24:33 +0200
> Jonathan Morton <chromatix99@gmail.com> wrote:
>> Currently, CUBIC is the default TCP send-side algorithm in Linux.  It
>> seems likely that it will react correctly to ECN marking, but that a
>> higher rate of marking may be needed to bring it down to a given
>> buffering level.  From what I've read, SFB should be able to probe
>> for the correct marking rate on a per-flow basis, which is nice.
>> 
>> On the subject of ECN, my impression is that YouTube currently
>> doesn't enable it, but a one-man company I recently downloaded some
>> stuff from does.  I wonder if there's any reliable data on how many
>> of the most popular sites enable ECN if you ask for it.  Personally,
>> I think IPv6 and ECN should probably go together - v6 gear is new or
>> upgraded anyway so there shouldn't be any legacy problems.

I agree, but lack data. What TCP algorithms are available in the IPv6
stack on Linux? I know SFB works with both ipv4 and ipv6...

ECN has been enabled on kernel org for 8+ years.

>
> Even TCP window scaling is problematic. Many consumer bits of gear are seriously
> broken. I have had to turn off WS on my laptop to deal with hotel and conference
> wireless front ends.

Good tip.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Taxonomy of various sender-side TCPs

[Jonathan Morton]

On 11 Mar, 2011, at 8:05 pm, Dave Täht wrote:

>> On the subject of ECN, my impression is that YouTube currently
>> doesn't enable it, but a one-man company I recently downloaded some
>> stuff from does.  I wonder if there's any reliable data on how many
>> of the most popular sites enable ECN if you ask for it.  Personally,
>> I think IPv6 and ECN should probably go together - v6 gear is new or
>> upgraded anyway so there shouldn't be any legacy problems.
> 
> I agree, but lack data. What TCP algorithms are available in the IPv6
> stack on Linux? I know SFB works with both ipv4 and ipv6...

AFAIK, the TCP congestion algorithms are all implemented independently of IP version, so even the exotic ones are available in v6 already.  I don't think they need to look at the addresses or port numbers.

The qdiscs are also relatively protocol-independent in themselves, I think.  Classifiers are mostly what need to examine the addresses and so on.  In particular, I'm pretty sure SFQ works, and that *does* depend on flow identification, so there is no reason why a competent implementation of nRED or SFB wouldn't.

> ECN has been enabled on kernel org for 8+ years.

Which is great, except that ECN is still not universally enabled client-side, so many problems with random ISPs are still masked until someone starts doing something unusual.

More relevantly, what is the situation with Windows (XP/Vista/7), MacOS X, and the prominent mobile OSes (Android, iOS)?  Do they attempt ECN negotiation already?  What happens if they encounter a black hole while doing so?

 - Jonathan

Re: [Bloat] Taxonomy of various sender-side TCPs

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 11 Mar, 2011, at 8:05 pm, Dave Täht wrote:
>
>>> On the subject of ECN, my impression is that YouTube currently
>>> doesn't enable it, but a one-man company I recently downloaded some
>>> stuff from does.  I wonder if there's any reliable data on how many
>>> of the most popular sites enable ECN if you ask for it.  Personally,
>>> I think IPv6 and ECN should probably go together - v6 gear is new or
>>> upgraded anyway so there shouldn't be any legacy problems.
>> 
>> I agree, but lack data. What TCP algorithms are available in the IPv6
>> stack on Linux? I know SFB works with both ipv4 and ipv6...
>
> AFAIK, the TCP congestion algorithms are all implemented independently
> of IP version, so even the exotic ones are available in v6 already.  I
> don't think they need to look at the addresses or port numbers.

So the

echo vegas > /proc/sys/net/ipv4/tcp_congestion_control

knob applies to both v4 and v6?

I have been thinking that we needed per-device TCP/IP algorithms. An
internal interface differs in requirements from an external gateway,
which in turn differs from several different kinds of wireless
interfaces.

>
> The qdiscs are also relatively protocol-independent in themselves, I
> think.  Classifiers are mostly what need to examine the addresses and
> so on.  

Since the state of ipv6 and encapsulated traffic shaping is so poor
(non-existent), I've been working on pulling together a good classifier
in my Cruft repository, when I have time. I hope we can gather more
shaper people together to work on improving matters for
bufferbloat/ECN/advanced shapers in the coming weeks, now that we have
the debloat-testing tree to work against and work ongoing in openwrt.

> In particular, I'm pretty sure SFQ works, and that *does*
> depend on flow identification, so there is no reason why a competent
> implementation of nRED or SFB wouldn't.

Testing... We really could use a good example tc implementation of SFB
and HTB at minimum to be playing with.

>> ECN has been enabled on kernel org for 8+ years.
>
> Which is great, except that ECN is still not universally enabled
> client-side, so many problems with random ISPs are still masked until
> someone starts doing something unusual.

Lots O Testing needed... If you haven't already seen the recent MIT
study on ECN, it's here:

http://mirrors.bufferbloat.net/Talks/AIMS2011/bauer-ecn-aims-2011.pdf

>
> More relevantly, what is the situation with Windows (XP/Vista/7),
> MacOS X, and the prominent mobile OSes (Android, iOS)?  Do they
> attempt ECN negotiation already?  What happens if they encounter a
> black hole while doing so?

I've experienced some issues with the Iphone and ECN, but I have so many
other balls/kernels/tools/drivers in the air regarding wireless that I
have not been able to track it down.
>
>  - Jonathan
>

-- 
Dave Taht
http://nex-6.taht.net

[Bloat] FW: [Iccrg] Fwd: Taxonomy of various sender-side TCPs

[Narasimha Reddy]

Our work on PERT might interest this thread.

http://www.ece.tamu.edu/~reddy/papers/sigcomm07.pdf

PERT is a sender side modification to convert TCP into employing a delay-based response.
It has been designed now to be able to co-exist with TCP:
http://www.ece.tamu.edu/~reddy/papers/iwqos2008.pdf

You can find a comparative performance of some of these protocols in Kiran Kotla's thesis (particularly the buffer buildup part might interest this group, Chapter 7, starting on page46):
http://cegroup.ece.tamu.edu/techpubs/2008/TAMU-ECE-2008-02.pdf

You can find PERT's video delivery performance results at:
http://www.ece.tamu.edu/~reddy/papers/pfldnet10.pdf

Regards,
Reddy

-----Original Message-----
From: iccrg-bounces@cs.ucl.ac.uk [mailto:iccrg-bounces@cs.ucl.ac.uk] On Behalf Of Lars Eggert
Sent: Saturday, March 12, 2011 2:11 AM
To: tcpm@ietf.org Extensions; iccrg@cs.ucl.ac.uk list
Subject: [Iccrg] Fwd: [Bloat] Taxonomy of various sender-side TCPs

FYI, there is an active thread on the bufferbloat list in response to this email that some of you might want to be aware of or join.

Begin forwarded message:

> From: Jonathan Morton <chromatix99@gmail.com>
> Date: March 11, 2011 9:24:33 GMT+02:00
> To: <bloat@lists.bufferbloat.net>
> Subject: [Bloat] Taxonomy of various sender-side TCPs
> 
> There's a lot of literature about how various TCP congestion-control algorithms compare to each other, but there's not much in the way of 10,000 foot overviews.  So this is an attempt to bring the most relevant data into one place.
> 
> I've sorted some common TCPs into groups in roughly ascending order of aggressiveness in the face of congestion, and considered how they might react to the introduction of ECN at the bottleneck.
> 
> 1: Vegas.
> Strategy:  Fill the pipe, not the buffers.
> Reaction to congestion:  Backs off until RTT approaches baseline, ie. buffers empty.
> Probable reaction to 1% ECN marking:  Very little (but it doesn't need to).
> 
> 2: Illinois, Compound.
> Strategy:  Fill the pipe quickly, then probe the buffer slowly to avoid being outcompeted.
> Reaction to congestion:  Backs off firmly on packet loss, then quickly re-probes for the pipe.
> Probable reaction to 1% ECN marking:  Window should stabilise near pipe length.
> 
> 3: Reno (and subtle variations thereof).
> Strategy:  Avoid congestion collapse in a simple, standardisable way.
> Reaction to congestion:  Halves the window on packet loss.
> Probable reaction to 1% ECN marking:  Window should stabilise near 100 packets.
> 
> 4: Veno, Westwood+.
> Strategy:  Distinguish congestion loss from random channel loss.
> Reaction to congestion:  Identical to Reno.
> Probable reaction to 1% ECN marking:  Identical to Reno.
> 
> 5: CUBIC.
> Strategy:  Fill pipe and all buffers to capacity.  This is the most aggressive widely-deployed TCP.
> Reaction to congestion:  On packet loss, quickly re-probes for buffer capacity.
> Probable reaction to 1% ECN marking:  UNKNOWN.
> 
> Currently, CUBIC is the default TCP send-side algorithm in Linux.  It seems likely that it will react correctly to ECN marking, but that a higher rate of marking may be needed to bring it down to a given buffering level.  From what I've read, SFB should be able to probe for the correct marking rate on a per-flow basis, which is nice.
> 
> On the subject of ECN, my impression is that YouTube currently doesn't enable it, but a one-man company I recently downloaded some stuff from does.  I wonder if there's any reliable data on how many of the most popular sites enable ECN if you ask for it.  Personally, I think IPv6 and ECN should probably go together - v6 gear is new or upgraded anyway so there shouldn't be any legacy problems.
> 
> - Jonathan
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Taxonomy of various sender-side TCPs

[Daniel Baluta]

Hi Jonathan,

On Fri, Mar 11, 2011 at 9:24 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> There's a lot of literature about how various TCP congestion-control algorithms compare to each other, but there's not much in the way of 10,000 foot overviews.  So this is an attempt to bring the most relevant data into one place.

Could you save this info on somewhere like a wiki page?

thanks,
Daniel.