[Bloat] a flood of Bufferbloat-related papers

[Bloat] a flood of Bufferbloat-related papers

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

I sat down on my vacation last week thinking I would write up a review
of progress since the bufferbloat effort began back in January, 2011. In
particular, I was interested in discovering to what extent we'd made the
cross-over not just into other OSes besides Linux (e.g - BSD, windows)
but into academia.

I figured I'd just write up a *short* wiki page on the interesting
bufferbloat related papers referenced via google and google scholar.

I was astounded (and delighted) to discover that google scholar had 30+
references, and digging into the later results (google page 2 and
beyond) was fascinating.

http://scholar.google.fr/scholar?start=10&q=bufferbloat&hl=fr&as_sdt=0,5&as_vis=1


I confess that I have not had time to read most of these yet! I've got
"FIFO Service with Differentiated Queueing" loaded on my kindle as I
write, as well as "on the impact of congestion control for concurrent
multipath transfer".

In the future I would certainly appreciate the authors of bufferbloat
related/referencing papers to mention them on this mailing list *as*
they are published!

Also a heads-up on work in progress would be helpful, both to avoid
unnecessary duplication of work and to find potential collaborators.

Rather than writing up that list of interesting papers, which google
scholar is doing so well for us already, I'm going to instead start
creating a list of papers and research I'd like to see. I'd appreciate
suggestions...

On my list already would be "an analysis of the effects of broken sack
processing on linux 2.4.24-3.1", of which I *think* I've captured
multiple examples of in the raw traces I've been collecting for
months... (so if anyone is interested in the raw data, I can provide)

I also note that TCP-fit, which was discussed on this mailing list a few
months ago, has a new paper due out this month (if it isn't out already)
in Infocom, Shanghai, October, 2011. I would love to have more
information on this algorithm because I simply cannot believe the
results, as much as I would like to believe them.
http://media.cs.tsinghua.edu.cn/~multimedia/tcp-fit/


-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 204 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] a flood of Bufferbloat-related papers

[Steinar H. Gunderson]

On Tue, Oct 11, 2011 at 12:27:17PM +0200, David Täht wrote:
> On my list already would be "an analysis of the effects of broken sack
> processing on linux 2.4.24-3.1", of which I *think* I've captured
> multiple examples of in the raw traces I've been collecting for
> months... (so if anyone is interested in the raw data, I can provide)

Do you have any more information? The only thing I could find online was that
there were SACK issues that were supposed to be fixed by 2.6.16; nothing
about a fix in 3.1 or post-3.1.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] a flood of Bufferbloat-related papers

[Eric Dumazet]

Le mardi 11 octobre 2011 à 13:58 +0200, Steinar H. Gunderson a écrit :
> On Tue, Oct 11, 2011 at 12:27:17PM +0200, David Täht wrote:
> > On my list already would be "an analysis of the effects of broken sack
> > processing on linux 2.4.24-3.1", of which I *think* I've captured
> > multiple examples of in the raw traces I've been collecting for
> > months... (so if anyone is interested in the raw data, I can provide)
> 
> Do you have any more information? The only thing I could find online was that
> there were SACK issues that were supposed to be fixed by 2.6.16; nothing
> about a fix in 3.1 or post-3.1.
> 

Of course, instead of discussing all linux stuff privately, it would be
good to discuss with linux network maintainers.

Or is the goal is to provide nice papers only ?

commit f779b2d60ab95c17f1e025778ed0df3ec2f05d75
Author: Zheng Yan <zheng.z.yan@intel.com>
Date:   Sun Sep 18 22:37:34 2011 -0400

    tcp: fix validation of D-SACK
    
    D-SACK is allowed to reside below snd_una. But the corresponding check
    in tcp_is_sackblock_valid() is the exact opposite. It looks like a typo.
    
    Signed-off-by: Zheng Yan <zheng.z.yan@intel.com>
    Acked-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index ea0d218..21fab3e 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -1124,7 +1124,7 @@ static int tcp_is_sackblock_valid(struct tcp_sock *tp, int is_dsack,
 		return 0;
 
 	/* ...Then it's D-SACK, and must reside below snd_una completely */
-	if (!after(end_seq, tp->snd_una))
+	if (after(end_seq, tp->snd_una))
 		return 0;
 
 	if (!before(start_seq, tp->undo_marker))

Re: [Bloat] a flood of Bufferbloat-related papers

[David Täht]

[-- Attachment #1.1: Type: text/plain, Size: 1295 bytes --]

On Tue, Oct 11, 2011 at 12:27:17PM +0200, David Täht wrote:

> > On my list already would be "an analysis of the effects of broken sack
> > processing on linux 2.4.24-3.1", of which I *think* I've captured
> > multiple examples of in the raw traces I've been collecting for
> > months... (so if anyone is interested in the raw data, I can provide)
Do you have any more information? The only thing I could find online was that
there were SACK issues that were supposed to be fixed by 2.6.16; nothing
about a fix in 3.1 or post-3.1.


This was a typo on my part - 2.6.24-3.1, not 2.4


The specific commit that concerned me was this one. I had seen multiple
cases of odd behavior (basically tcp sacking like crazy until the cwr
collapses or a tcp reset) that I *think* this fix explains.



commit f779b2d60ab95c17f1e025778ed0df3ec2f05d75
Author: Zheng Yan <zheng.z.yan@intel.com>
Date:   Sun Sep 18 22:37:34 2011 -0400

    tcp: fix validation of D-SACK
   
    D-SACK is allowed to reside below snd_una. But the corresponding check
    in tcp_is_sackblock_valid() is the exact opposite. It looks like a typo.
   
    Signed-off-by: Zheng Yan <zheng.z.yan@intel.com>
    Acked-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>



-- 
Dave Täht


[-- Attachment #1.2: Type: text/html, Size: 2502 bytes --]

[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] a flood of Bufferbloat-related papers

[Eric Dumazet]

Le mardi 11 octobre 2011 à 14:13 +0200, Eric Dumazet a écrit :
> Le mardi 11 octobre 2011 à 13:58 +0200, Steinar H. Gunderson a écrit :
> > On Tue, Oct 11, 2011 at 12:27:17PM +0200, David Täht wrote:
> > > On my list already would be "an analysis of the effects of broken sack
> > > processing on linux 2.4.24-3.1", of which I *think* I've captured
> > > multiple examples of in the raw traces I've been collecting for
> > > months... (so if anyone is interested in the raw data, I can provide)
> > 
> > Do you have any more information? The only thing I could find online was that
> > there were SACK issues that were supposed to be fixed by 2.6.16; nothing
> > about a fix in 3.1 or post-3.1.
> > 
> 
> Of course, instead of discussing all linux stuff privately, it would be
> good to discuss with linux network maintainers.
> 
> Or is the goal is to provide nice papers only ?
> 
> commit f779b2d60ab95c17f1e025778ed0df3ec2f05d75
> Author: Zheng Yan <zheng.z.yan@intel.com>
> Date:   Sun Sep 18 22:37:34 2011 -0400

Another fix (scheduled for linux-3.2 available in net-next tree :)

[ And some work is currently in progress to support RFC 5562 ]

commit 7a269ffad72f3604b8982fa09c387670e0d2ee14
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Thu Sep 22 20:02:19 2011 +0000

    tcp: ECN blackhole should not force quickack mode
    
    While playing with a new ADSL box at home, I discovered that ECN
    blackhole can trigger suboptimal quickack mode on linux : We send one
    ACK for each incoming data frame, without any delay and eventual
    piggyback.
    
    This is because TCP_ECN_check_ce() considers that if no ECT is seen on a
    segment, this is because this segment was a retransmit.
    
    Refine this heuristic and apply it only if we seen ECT in a previous
    segment, to detect ECN blackhole at IP level.
    
    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    CC: Jamal Hadi Salim <jhs@mojatatu.com>
    CC: Jerry Chu <hkchu@google.com>
    CC: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
    CC: Jim Gettys <jg@freedesktop.org>
    CC: Dave Taht <dave.taht@gmail.com>
    Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/include/net/tcp.h b/include/net/tcp.h
index f357bef..702aefc 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -356,6 +356,7 @@ static inline void tcp_dec_quickack_mode(struct sock *sk,
 #define	TCP_ECN_OK		1
 #define	TCP_ECN_QUEUE_CWR	2
 #define	TCP_ECN_DEMAND_CWR	4
+#define	TCP_ECN_SEEN		8
 
 static __inline__ void
 TCP_ECN_create_request(struct request_sock *req, struct tcphdr *th)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index a5d01b1..5a4408c 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -217,16 +217,25 @@ static inline void TCP_ECN_withdraw_cwr(struct tcp_sock *tp)
 	tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
 }
 
-static inline void TCP_ECN_check_ce(struct tcp_sock *tp, struct sk_buff *skb)
+static inline void TCP_ECN_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
 {
-	if (tp->ecn_flags & TCP_ECN_OK) {
-		if (INET_ECN_is_ce(TCP_SKB_CB(skb)->flags))
-			tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
+	if (!(tp->ecn_flags & TCP_ECN_OK))
+		return;
+
+	switch (TCP_SKB_CB(skb)->flags & INET_ECN_MASK) {
+	case INET_ECN_NOT_ECT:
 		/* Funny extension: if ECT is not set on a segment,
-		 * it is surely retransmit. It is not in ECN RFC,
-		 * but Linux follows this rule. */
-		else if (INET_ECN_is_not_ect((TCP_SKB_CB(skb)->flags)))
+		 * and we already seen ECT on a previous segment,
+		 * it is probably a retransmit.
+		 */
+		if (tp->ecn_flags & TCP_ECN_SEEN)
 			tcp_enter_quickack_mode((struct sock *)tp);
+		break;
+	case INET_ECN_CE:
+		tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
+		/* fallinto */
+	default:
+		tp->ecn_flags |= TCP_ECN_SEEN;
 	}
 }
 

Re: [Bloat] a flood of Bufferbloat-related papers

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 4023 bytes --]

On 10/11/2011 02:13 PM, Eric Dumazet wrote:
> Le mardi 11 octobre 2011 à 13:58 +0200, Steinar H. Gunderson a écrit :
>> On Tue, Oct 11, 2011 at 12:27:17PM +0200, David Täht wrote:
>>> On my list already would be "an analysis of the effects of broken sack
>>> processing on linux 2.4.24-3.1", of which I *think* I've captured
>>> multiple examples of in the raw traces I've been collecting for
>>> months... (so if anyone is interested in the raw data, I can provide)
>> Do you have any more information? The only thing I could find online was that
>> there were SACK issues that were supposed to be fixed by 2.6.16; nothing
>> about a fix in 3.1 or post-3.1.
>>
> Of course, instead of discussing all linux stuff privately, it would be
> good to discuss with linux network maintainers.
>
> Or is the goal is to provide nice papers only ?

hmm?

0) I know there are enough linux network maintainers on the bloat list
for me to not consider anything typed here private! If you would prefer
that the previous message had been sent more broadly (netdev?) please
feel free to forward.

1) Lord, no, I'm far more interested in Linux working well, and having a
short diagnose, fix, test cycle, than having nice papers. Staying within
2 kernel release cycles is fastly superior to dozens.

Having nice papers helps, however, particularly on problems that have
existed a long time, and can be analyzed - in particular - as to the
effect on what other datasets that occurred during this period. On my
bad days I think we're going to have to re-run the last decade's worth
of experiments on ECN, congestion control, AQMs, and wireless across
every protocol that has ever been experimented with until the
bufferbloat problem is thoroughly licked.

on my good days... well, I don't have a lot of good days

2) My problem with publishing data collected from various tcp traces
I've been collecting is that of preserving privacy and anonymity to the
end user from where I collected the traces. For example, I got a very
interesting trace today showing very interesting ipv6 sack-related
behavior from a friend of mine's home over free.fr - but I'd be very
reluctant to make that widely available without permission.

Now, I don't have that restriction on data collected from more public
places, or in my own tests of cerowrt - but I have to say that I have
mostly only seen lots of sacks over the LFN, where my data sets are more
limited, at present.

If you are interested in collecting your own data set,
jupiter.lab.bufferbloat.net is a cerowrt router located in bloatlab 1 at
ISC.org in san francisco.

http://www.bufferbloat.net/projects/cerowrt/wiki/BloatLab_1

and is running linux 3.04, with netperf 2.5 enabled for both IPv6 and
ipv4, with all the cerowrt related configuration changes to date. High
on my list is measuring lfn performance with the vastly decreased amount
of buffering presently in that system both for single and multiple
streams - and then move on to analyzing the newer AQMs now in Linux.

io.lab.bufferbloat.net is an x86 netsonar box, running 2.6.16 - netperf
2.5 as well - configured as per the shipping linux defaults.

I hope fairly soon, to update another router (europa) to the latest and
greatest tcp/sack code now that the kernel.org problems are settling
down. It is my hope that this is the last tcp-related issue that can
skew the results - and for all I know, without analysis, may not skew
the results...

We are still playing with nuttcp and iperf, and there are several
bufferbloat related changes in the next version of netperf's svn tree
that need to settle down some.

If there is more I can do in providing test facilities and valuable
feedback to the networking maintainers, please let me know.

4) I personally find sack very interesting in the case of wireless as it
could be used very effectively (or so I think) in reducing the amount of
needed packet aggregation in wireless-n - and thus retries and
buffering, and I'm glad to see people working on improving it.

-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

[Bloat] the observed sack oddity

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

I have put up screenshots of tcptrace -G and xplot.org's analysis of my
most recent capture of an ipv6 connection over freebox (wired) from a
debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
kernel (the aformentioned netsonar box) in redwood city, ca, both
running tcp cubic, over ipv6.

at first glance, it looked a lot like a classic bufferbloat trace, with
complete with periodic collapses and cubic increasing the window.

Then I noticed the sacks.

I also have a trace taken between the debloat-testing box and a cerowrt
box (running linux 3.04), both running westwood, which shows
considerably less undesirable behavior, and I will repeat that test with
cubic this evening and post additional data tomorrow morning.

-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] the observed sack oddity

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Of course, it would help if I posted a link to the screenshots!

http://www.teklibre.com/~d/sackoddity/

On 10/11/2011 03:09 PM, David Täht wrote:
> I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> most recent capture of an ipv6 connection over freebox (wired) from a
> debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> kernel (the aformentioned netsonar box) in redwood city, ca, both
> running tcp cubic, over ipv6.
>
> at first glance, it looked a lot like a classic bufferbloat trace, with
> complete with periodic collapses and cubic increasing the window.
>
> Then I noticed the sacks.
>
> I also have a trace taken between the debloat-testing box and a cerowrt
> box (running linux 3.04), both running westwood, which shows
> considerably less undesirable behavior, and I will repeat that test with
> cubic this evening and post additional data tomorrow morning.
>


-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] the observed sack oddity

[Eric Dumazet]

Le mardi 11 octobre 2011 à 15:11 +0200, David Täht a écrit :
> Of course, it would help if I posted a link to the screenshots!
> 
> http://www.teklibre.com/~d/sackoddity/
> 
> On 10/11/2011 03:09 PM, David Täht wrote:
> > I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> > most recent capture of an ipv6 connection over freebox (wired) from a
> > debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> > kernel (the aformentioned netsonar box) in redwood city, ca, both
> > running tcp cubic, over ipv6.
> >
> > at first glance, it looked a lot like a classic bufferbloat trace, with
> > complete with periodic collapses and cubic increasing the window.
> >
> > Then I noticed the sacks.
> >
> > I also have a trace taken between the debloat-testing box and a cerowrt
> > box (running linux 3.04), both running westwood, which shows
> > considerably less undesirable behavior, and I will repeat that test with
> > cubic this evening and post additional data tomorrow morning.


Who is the sender ?

Is some WIFI involved ?

"netstat -s" on the receiver would help

This might be a truesize mismatch.

Re: [Bloat] the observed sack oddity

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On 10/11/2011 03:33 PM, Eric Dumazet wrote:
> Le mardi 11 octobre 2011 à 15:11 +0200, David Täht a écrit :
>> Of course, it would help if I posted a link to the screenshots!
>>
>> http://www.teklibre.com/~d/sackoddity/
>>
>> On 10/11/2011 03:09 PM, David Täht wrote:
>>> I have put up screenshots of tcptrace -G and xplot.org's analysis of my
>>> most recent capture of an ipv6 connection over freebox (wired) from a
>>> debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
>>> kernel (the aformentioned netsonar box) in redwood city, ca, both
>>> running tcp cubic, over ipv6.
>>>
>>> at first glance, it looked a lot like a classic bufferbloat trace, with
>>> complete with periodic collapses and cubic increasing the window.
>>>
>>> Then I noticed the sacks.
>>>
>>> I also have a trace taken between the debloat-testing box and a cerowrt
>>> box (running linux 3.04), both running westwood, which shows
>>> considerably less undesirable behavior, and I will repeat that test with
>>> cubic this evening and post additional data tomorrow morning.
>
> Who is the sender ?

The laptop was the netperf initiator, running 3.1.0-rc4-dbt23 #1 SMP
PREEMPT. This is basically 3.1-rc4 + the very few debloat-testing
patches which are all specific to wifi. I can easily test against
2.6.38, and with a little work, against linux git head, on the laptop.

sysctl on the laptop side is:

net.ipv4.tcp_ecn=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1

net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.ipv4.tcp_rmem = 4096 87380 2097152
net.ipv4.tcp_wmem = 4096 65536 2097152

on the netsonar side remotely (2.6.16), it is whatever the defaults are.

Updating a server in bloatlab #1 in California is a little harder, as
you might imagine, but I should be able to get that done by next week.


> Is some WIFI involved ?

No.

> "netstat -s" on the receiver would help
>
I will rerun the tests.

> This might be a truesize mismatch.
>
>
I saw those patches being discussed but do not understand the implications.



-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] the observed sack oddity

[Justin McCann]

On Tue, Oct 11, 2011 at 9:09 AM, David Täht <dave.taht@gmail.com> wrote:
> I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> most recent capture of an ipv6 connection over freebox (wired) from a
> debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> kernel (the aformentioned netsonar box) in redwood city, ca, both
> running tcp cubic, over ipv6.
>
> at first glance, it looked a lot like a classic bufferbloat trace, with
> complete with periodic collapses and cubic increasing the window.
>
> Then I noticed the sacks.

It took me a bit to figure out what was odd about the SACKs. I see:

- At the beginning, the sender is ramping up faster than the rate the
ACKs are coming in (white vs. green). I'd expect them both to have
near the same slope (and change in slope) if slow start wasn't going
overboard. Is this difference in slope pretty common in bufferbloat
traces?

- Sender marks the first CWR shortly after the first SACKs come in.
That looks like some sort of AQM kicked in and dropped packets at the
head (or middle) instead of drop-tail, so you get the SACKs. Since the
RTT is so large (France <-> CA), the retransmissions take a while to
get to the receiver (and the ACKs returned), so the green line stays
flat. But since packets are getting through (as seen by the SACKs),
the window keeps advancing.

Now I see the problem: Why doesn't the receiver cover the whole
missing range in each SACK, while the green line isn't advancing
(green == next seqno)? It does in the right hand side of
http://www.teklibre.com/~d/sackoddity/freeboxipv6zoomtosackwindow.png
but not at the beginning. If it was a dropped ACK, any one of the
later SACKs would have advanced the acknowledged window, but they
didn't--- the receiver isn't properly tracking the entire SACK
window(s).

Does that catch it?

     Justin

Re: [Bloat] the observed sack oddity

[Justin McCann]

On Tue, Oct 11, 2011 at 9:09 AM, David Täht <dave.taht@gmail.com> wrote:
> I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> most recent capture of an ipv6 connection over freebox (wired) from a
> debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> kernel (the aformentioned netsonar box) in redwood city, ca, both
> running tcp cubic, over ipv6.
>
> at first glance, it looked a lot like a classic bufferbloat trace, with
> complete with periodic collapses and cubic increasing the window.

A side question on the outstanding data graph:
  http://www.teklibre.com/~d/sackoddity/freeboxoutstandingdata.png
compared with the sequence number graph:
  http://www.teklibre.com/~d/sackoddity/freeboxipv6seq.png

Looking at
  http://www.tcptrace.org/manual/node15_tf.html
the red line is supposed to approximate the sender's congestion window.

If you line up the segment timeline and the outstanding data timeline,
it seems like the (red) instantaneous outstanding data line doesn't
take SACKed segments into account, and just uses the normal
acknowledgment number. So, once the early missing segment gets
acknowledged, you see a sharp dropoff in the red line---that last ACK
covers all of the already-received and SACKed segments as well.

I guess the red line includes packets in the network, plus held up in
the receiver's buffer. Would it be useful to have a line that
estimates what is currently in the network that doesn't include the
SACKed packets (i.e. the ones in the receive buffer that can't make it
to the application)?

     Justin

Re: [Bloat] a flood of Bufferbloat-related papers

[Lawrence Stewart]

Hi David,

On 10/11/11 21:27, David Täht wrote:
> I sat down on my vacation last week thinking I would write up a review
> of progress since the bufferbloat effort began back in January, 2011. In

I think we should exercise a bit of caution in relation to saying things 
like "the bufferbloat effort started in Jan 2011" - the pre-2011 
literature alone (although not using the term "bufferbloat" 
specifically) extensively covers the issues, symptoms and a myriad of 
solutions for the problem.

> particular, I was interested in discovering to what extent we'd made the
> cross-over not just into other OSes besides Linux (e.g - BSD, windows)
> but into academia.

FWIW, I'm a FreeBSD kernel developer and a PhD student working on 
transport layer congestion control. I've been involved in doing a lot of 
experimental work related to bufferbloat (we refer to it as "collateral 
damage" in our papers) using FreeBSD and Linux.

> In the future I would certainly appreciate the authors of bufferbloat
> related/referencing papers to mention them on this mailing list *as*
> they are published!

Here are a few pointers to relevant papers done by people at the 
research centre where I'm studying:

"A rough comparison of NewReno, CUBIC, Vegas and ‘CAIA Delay Gradient’ 
TCP (v0.1)": http://caia.swin.edu.au/reports/110729A/CAIA-TR-110729A.pdf

"Revisiting TCP Congestion Control using Delay Gradients": 
http://www.springerlink.com/content/mq50134631115076/

"Multimedia-unfriendly TCP Congestion Control and Home Gateway Queue 
Management": http://dx.doi.org/10.1145/1943552.1943558

"Improved coexistence and loss tolerance for delay based TCP congestion 
control": http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5735714&tag=1

"Collateral Damage: The Impact of Optimised TCP Variants on Real-Time 
Traffic Latency in Consumer Broadband Environments": 
http://dx.doi.org/10.1007/978-3-642-01399-7_31


Off the top of my head, you should also check out the work done by Nick 
McKeown's group (although their focus has been on core routers): 
http://yuba.stanford.edu/buffersizing/

There's plenty of other relevant work that I can't think of specifically 
right now, but it's out there.

Cheers,
Lawrence

Re: [Bloat] a flood of Bufferbloat-related papers

[David Täht]

[-- Attachment #1: Type: text/plain, Size: 3169 bytes --]

On 10/12/2011 09:49 AM, Lawrence Stewart wrote:
> Hi David,
>
> On 10/11/11 21:27, David Täht wrote:
>> I sat down on my vacation last week thinking I would write up a review
>> of progress since the bufferbloat effort began back in January, 2011. In
>
> I think we should exercise a bit of caution in relation to saying
> things like "the bufferbloat effort started in Jan 2011" - the
> pre-2011 literature alone (although not using the term "bufferbloat"
> specifically) extensively covers the issues, symptoms and a myriad of
> solutions for the problem.
>

You are correct, I should have said 'bufferbloat.net effort', which got
off the ground in late january, 2011.

We had started on a collection of older papers on the wiki, starting
with rfc970 and working forward. Filling in the blanks between that (and
yes, we can go back further) and the present day is going to take some work!

>> particular, I was interested in discovering to what extent we'd made the
>> cross-over not just into other OSes besides Linux (e.g - BSD, windows)
>> but into academia.
>
> FWIW, I'm a FreeBSD kernel developer and a PhD student working on
> transport layer congestion control. I've been involved in doing a lot
> of experimental work related to bufferbloat (we refer to it as
> "collateral damage" in our papers) using FreeBSD and Linux.
>

More keywords to google for...

>> In the future I would certainly appreciate the authors of bufferbloat
>> related/referencing papers to mention them on this mailing list *as*
>> they are published!
>
> Here are a few pointers to relevant papers done by people at the
> research centre where I'm studying:
>
> "A rough comparison of NewReno, CUBIC, Vegas and ‘CAIA Delay Gradient’
> TCP (v0.1)": http://caia.swin.edu.au/reports/110729A/CAIA-TR-110729A.pdf
>
> "Revisiting TCP Congestion Control using Delay Gradients":
> http://www.springerlink.com/content/mq50134631115076/
>
> "Multimedia-unfriendly TCP Congestion Control and Home Gateway Queue
> Management": http://dx.doi.org/10.1145/1943552.1943558
>
> "Improved coexistence and loss tolerance for delay based TCP
> congestion control":
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5735714&tag=1
>
> "Collateral Damage: The Impact of Optimised TCP Variants on Real-Time
> Traffic Latency in Consumer Broadband Environments":
> http://dx.doi.org/10.1007/978-3-642-01399-7_31
>
>

I was aware of 2 of these, but not the others. I would very much like to
read the last in particular, but it doesn't appear to be publicly
available from a quick google.

(And thx for putting the refs into the wiki format!)

> Off the top of my head, you should also check out the work done by
> Nick McKeown's group (although their focus has been on core routers):
> http://yuba.stanford.edu/buffersizing/
>

I've read much of their work, and much of it does apply, but home
routers are different in significant ways that I hope to be talking
about at lincs in paris, oct 19...


> There's plenty of other relevant work that I can't think of
> specifically right now, but it's out there.
>

Thanks for the pointers... (all, keep 'em coming)

> Cheers,
> Lawrence


-- 
Dave Täht


[-- Attachment #2: dave_taht.vcf --]
[-- Type: text/x-vcard, Size: 214 bytes --]

begin:vcard
fn;quoted-printable:Dave T=C3=A4ht
n;quoted-printable:T=C3=A4ht;Dave
email;internet:dave.taht@gmail.com
tel;home:1-239-829-5608
tel;cell:0638645374
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [Bloat] the observed sack oddity

[Jan Ceuleers]

On 10/11/2011 03:09 PM, David Täht wrote:
> I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> most recent capture of an ipv6 connection over freebox (wired) from a
> debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> kernel (the aformentioned netsonar box) in redwood city, ca, both
> running tcp cubic, over ipv6.
>
David, Eric,

Since the both of you are in France, would it be an idea to get together?

Cheers, Jan

Re: [Bloat] the observed sack oddity

[Eric Dumazet]

Le jeudi 13 octobre 2011 à 20:23 +0200, Jan Ceuleers a écrit :
> On 10/11/2011 03:09 PM, David Täht wrote:
> > I have put up screenshots of tcptrace -G and xplot.org's analysis of my
> > most recent capture of an ipv6 connection over freebox (wired) from a
> > debloat-testing kernel here in france (pre 3.1 by a lot) to a 2.6.16
> > kernel (the aformentioned netsonar box) in redwood city, ca, both
> > running tcp cubic, over ipv6.
> >
> David, Eric,
> 
> Since the both of you are in France, would it be an idea to get together?
> 

Hello Jan

I actually met David two days ago in Paris, since I went there for a
business trip. It was a great time !

By the way, I suggest continuing this discussion privately ;)

See you
Eric