[Bloat] ECN & AQM Hall of Fame?

[Bloat] ECN & AQM Hall of Fame?

[Richard Scheffenegger]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

Hi,

wasn't there talk about getting a Hall of Fame for networks / operators, which are using and actively supporting AQM and ECN in their administrative domain?

I just looked at some traces to troubleshoot SMTP with GMX (german freemail/hosting provider), and noticed that they are in fact negotiating for ECN on their mail servers:

0.885436 IP home.58492 > mx0.gmx.net.smtp: SWE 1849574907:1849574907(0) win 5840 <mss 1460,sackOK,timestamp 1988551314 0,nop,wscale 1>
0.885581 IP mx0.gmx.net.smtp > home.58492: SE 2571500293:2571500293(0) ack 1849574908 win 5792 <mss 1460,sackOK,timestamp 113816731 1988551314,nop,wscale 7>

;)

Regards,
   Richard



[-- Attachment #2: Type: text/html, Size: 1583 bytes --]

Re: [Bloat] ECN & AQM Hall of Fame?

[Jim Gettys]

On 01/30/2011 05:40 PM, Richard Scheffenegger wrote:
> Hi,
>
> wasn't there talk about getting a Hall of Fame for networks / operators, which are using and actively supporting AQM and ECN in their administrative domain?
>
> I just looked at some traces to troubleshoot SMTP with GMX (german freemail/hosting provider), and noticed that they are in fact negotiating for ECN on their mail servers:
>
> 0.885436 IP home.58492>  mx0.gmx.net.smtp: SWE 1849574907:1849574907(0) win 5840<mss 1460,sackOK,timestamp 1988551314 0,nop,wscale 1>
> 0.885581 IP mx0.gmx.net.smtp>  home.58492: SE 2571500293:2571500293(0) ack 1849574908 win 5792<mss 1460,sackOK,timestamp 113816731 1988551314,nop,wscale 7>
>
> ;)
>
You just talked: so why not start one?  That's what wiki's are for...
		Best Regards,
			- Jim

Re: [Bloat] ECN & AQM Hall of Fame?

[Dave Täht]

> On 01/30/2011 05:40 PM, Richard Scheffenegger wrote:
>> Hi,
>>
>> wasn't there talk about getting a Hall of Fame for networks / operators, which are using and actively supporting AQM and ECN in their administrative domain?
>>
>> I just looked at some traces to troubleshoot SMTP with GMX (german freemail/hosting provider), and noticed that they are in fact negotiating for ECN on their mail servers:
>>

One data collection method I've been thinking about was using hooks on
network chains (netfilter) to gather statistics on the actual incidence
of ECN and SACK/DSACK.

I have found a lot of places that negotiate ECN but have rarely seen ECN
actually get used on the path.

We'd need an ECN hall of fame and a separate SACK related one too.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] ECN & AQM Hall of Fame?

[Jim Gettys]

On 01/30/2011 08:03 PM, Dave Täht wrote:
>
>> On 01/30/2011 05:40 PM, Richard Scheffenegger wrote:
>>> Hi,
>>>
>>> wasn't there talk about getting a Hall of Fame for networks / operators, which are using and actively supporting AQM and ECN in their administrative domain?
>>>
>>> I just looked at some traces to troubleshoot SMTP with GMX (german freemail/hosting provider), and noticed that they are in fact negotiating for ECN on their mail servers:
>>>
>
> One data collection method I've been thinking about was using hooks on
> network chains (netfilter) to gather statistics on the actual incidence
> of ECN and SACK/DSACK.
>
> I have found a lot of places that negotiate ECN but have rarely seen ECN
> actually get used on the path.
>
> We'd need an ECN hall of fame and a separate SACK related one too.
>
Steve Bauer has been performing a very extensive survey of ECN on the 
Alexa top sites; he should have results in the next few weeks, for the 
CAIDA workshop.

Talk Title: A survey of the current state of ECN support in servers, 
clients, and routers

Talk Abstract: Explicit congestion notification (ECN) is a key building 
block in a number of ongoing standardization efforts [Conex] and 
research projects [Alizadeh]. This paper therefore sought to survey the 
current state of ECN support on the Internet, updating and extending a 
similar survey from 2008 [Langley]. (There are a number of reasons to 
suspect the state of ECN support may have changed including 'server-side 
ECN' becoming the default on recent versions of the Linux kernel.) In 
the process of conducting our survey we discovered that some routers 
incorrectly handle the ECN bits in the IP header, namely clearing the 
ECT bit. Given that this is a new impediment to using ECN, we sought to 
carefully measure exactly where this problem occurred. While measuring 
ECN support to web servers is straightforward, we developed novel 
active/passive hybrid approaches for collecting similar measurements of 
paths to clients. This is important because even if some servers support 
ECN, if the paths to clients contain impediments to ECN usage (which we 
show they do) the incremental deployment of ECN is harmed.

In short, ECN is no longer blocked much, but very few are actually 
turning it on.

News when it's available...
			- Jim

[Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Florian Lohoff]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

On Sun, Jan 30, 2011 at 11:40:05PM +0100, Richard Scheffenegger wrote:
> Hi,
> 
> wasn't there talk about getting a Hall of Fame for networks / operators,
> which are using and actively supporting AQM and ECN in their administrative
> domain?
> 

The problem is not only in the IP Path.

In the old days i was used to be able to work interactive on a Serial
connection with 33600 Bit/s shared with 10 Dialup users.

After reading Jims blog i started to dig in our Network and found
huge buffers on the Huawei MA5600 DSLAMs which are Ethernet based
DSLAMs. These DSLAMs are basically invisible to the user as they
are only in the L2 PPPoE path for the customer.

I found the Buffer to be roughly 1MByte per Line. With a 1MBit/s DSL 
line this is roughly 10s worth of Buffer which i can observe on my line
in the real world.

There is no way of tuning this buffer based on the speed or even shrink
it so its difficult to fix.

ANCP + a shaper on the BRAS would be a solution which is basically how i solved
it for me - As i have a fixed rate 1MBit/s DSL i attached a 1MBit/s rate
limiter to my profile on the BRAS which immediatly fixed the problem for me.

And the Hall of Shame probably is a good idea - but wont fix it. The
ISPs are not aware of the problem as testing happens on empty lines with
30cm of wire. I sent the link to Jims blog to certain people and the DSLAM
guys immediatly promised to contact Huawei about the non tunable buffersize.

Flo
-- 
Florian Lohoff                                                 f@zz.de
                 Professionell gesehen bin ich zu haben ....

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Jim Gettys]

On 01/31/2011 03:41 AM, Florian Lohoff wrote:
> On Sun, Jan 30, 2011 at 11:40:05PM +0100, Richard Scheffenegger wrote:
>> Hi,
>>
>> wasn't there talk about getting a Hall of Fame for networks / operators,
>> which are using and actively supporting AQM and ECN in their administrative
>> domain?
>>
>
> The problem is not only in the IP Path.
>
> In the old days i was used to be able to work interactive on a Serial
> connection with 33600 Bit/s shared with 10 Dialup users.
>
> After reading Jims blog i started to dig in our Network and found
> huge buffers on the Huawei MA5600 DSLAMs which are Ethernet based
> DSLAMs. These DSLAMs are basically invisible to the user as they
> are only in the L2 PPPoE path for the customer.
>
> I found the Buffer to be roughly 1MByte per Line. With a 1MBit/s DSL
> line this is roughly 10s worth of Buffer which i can observe on my line
> in the real world.

Yes, Dave Clark first ran into bufferbloat on a DSLAM he runs, and it 
was 6 seconds.

Even a gigabit ethernet switch needs to have buffering (which will work 
out to 8ms, worst case).


>
> There is no way of tuning this buffer based on the speed or even shrink
> it so its difficult to fix.

DOCSIS has the same issue: it's not currently tunable for the speed, and 
different people are provisioned greatly different bandwidth.

>
> ANCP + a shaper on the BRAS would be a solution which is basically how i solved
> it for me - As i have a fixed rate 1MBit/s DSL i attached a 1MBit/s rate
> limiter to my profile on the BRAS which immediatly fixed the problem for me.
>
> And the Hall of Shame probably is a good idea - but wont fix it. The
> ISPs are not aware of the problem as testing happens on empty lines with
> 30cm of wire. I sent the link to Jims blog to certain people and the DSLAM
> guys immediatly promised to contact Huawei about the non tunable buffersize.
>

Yes, please do.  And yes, it is the lack of "latency under load" tests 
that mean that most won't see it in typical testing.

That we're also trying to fix, by getting the formal tests done by 
government agencies and testing services to add such a test.  I think 
we're likely to succeed for speedtest.net and for the FCC broadband 
test.  Those who are helping other government's agencies out with 
similar tests should also be educated.

I believe in the meanwhile, and because having the tests for development 
lab and corporate environments is also really necessary and those tests 
typically aren't available in those environments, we should try to build 
some better tests here and make them available to everyone.  This is 
part of the purpose of bufferbloat.net.

And please, please make sure everyone treads gently: we're all bozos on 
this glass bus, and don't pick up stones.  Everyone (myself included) 
has been making the same class of mistakes, again and again; so do be 
gentle.  The problem is truly counter intuitive to most people.

I will be presenting at the IETF transport area meeting in Prague.
			Best regards,
				- Jim

Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Richard Scheffenegger]

There is currently a IPPM Draft in the works, talking about TCP Performance 
measurement methodology. It talks a little bit about measuring throughput 
during steady state and using in-band latency measurements (instead of ICMP 
pings in parallel):

http://tools.ietf.org/html/draft-ietf-ippm-tcp-throughput-tm-11


As of yet, there is no detailed discussion section about buffer bloat in 
that doc (and it may be out-of-scope to discuss this effect in there very 
extensively), but it might be good to have more knowledgable people review 
that doc as well, and give some feedback to the authors.

After all, doing meaningful measurements is obviously the very first step 
necessary to reveal this to a larger audience and make people aware. Hinting 
at buffer bloat in a standard doc that talks about TCP performance 
measurment seems to be a good idea!

Best regards,
   Richard


BTW, I found this legacy document, where the authors boldly claim that more 
buffers are always better for 802.11 networks, to circumvent costly TCP 
congestion control decisions....
http://csl.snu.ac.kr/~ecpark/papers/TCP_WLAN_TMC08.pdf

Also, this paper hints that more buffers are better:
http://www.ieee-infocom.org/2004/Papers/16_1.PDF
(I didn't spot a mentioning, that a packet switches network with infinite 
buffering will deliver zero goodput under congestion, as has been prooven in 
the dawn of time :). Here, the authors seem to be concerned only with work 
conservation on each individual router interface - i.e. that no interfaces 
is ever idle....). But at least: "The important aspect of end-to-end packet 
delay evaluation taking into consideration retransmissions of lost packets 
is beyond the scope of this paper."


Ah, so many unexploited paths to go (tweaking ACK windowsizes by the network 
in 802.11, for example, to give detailed congestion/network goodput feedback 
to the tcp sender)....

Richard


----- Original Message ----- 
From: "Jim Gettys" <jg@freedesktop.org>
To: <bloat@lists.bufferbloat.net>
Sent: Monday, January 31, 2011 3:16 PM
Subject: Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?


> On 01/31/2011 03:41 AM, Florian Lohoff wrote:
>> On Sun, Jan 30, 2011 at 11:40:05PM +0100, Richard Scheffenegger wrote:
>>> Hi,
>>>
>>> wasn't there talk about getting a Hall of Fame for networks / operators,
>>> which are using and actively supporting AQM and ECN in their 
>>> administrative
>>> domain?
>>>
>>
>> The problem is not only in the IP Path.
>>
>> In the old days i was used to be able to work interactive on a Serial
>> connection with 33600 Bit/s shared with 10 Dialup users.
>>
>> After reading Jims blog i started to dig in our Network and found
>> huge buffers on the Huawei MA5600 DSLAMs which are Ethernet based
>> DSLAMs. These DSLAMs are basically invisible to the user as they
>> are only in the L2 PPPoE path for the customer.
>>
>> I found the Buffer to be roughly 1MByte per Line. With a 1MBit/s DSL
>> line this is roughly 10s worth of Buffer which i can observe on my line
>> in the real world.
>
> Yes, Dave Clark first ran into bufferbloat on a DSLAM he runs, and it was 
> 6 seconds.
>
> Even a gigabit ethernet switch needs to have buffering (which will work 
> out to 8ms, worst case).
>
>
>>
>> There is no way of tuning this buffer based on the speed or even shrink
>> it so its difficult to fix.
>
> DOCSIS has the same issue: it's not currently tunable for the speed, and 
> different people are provisioned greatly different bandwidth.
>
>>
>> ANCP + a shaper on the BRAS would be a solution which is basically how i 
>> solved
>> it for me - As i have a fixed rate 1MBit/s DSL i attached a 1MBit/s rate
>> limiter to my profile on the BRAS which immediatly fixed the problem for 
>> me.
>>
>> And the Hall of Shame probably is a good idea - but wont fix it. The
>> ISPs are not aware of the problem as testing happens on empty lines with
>> 30cm of wire. I sent the link to Jims blog to certain people and the 
>> DSLAM
>> guys immediatly promised to contact Huawei about the non tunable 
>> buffersize.
>>
>
> Yes, please do.  And yes, it is the lack of "latency under load" tests 
> that mean that most won't see it in typical testing.
>
> That we're also trying to fix, by getting the formal tests done by 
> government agencies and testing services to add such a test.  I think 
> we're likely to succeed for speedtest.net and for the FCC broadband test. 
> Those who are helping other government's agencies out with similar tests 
> should also be educated.
>
> I believe in the meanwhile, and because having the tests for development 
> lab and corporate environments is also really necessary and those tests 
> typically aren't available in those environments, we should try to build 
> some better tests here and make them available to everyone.  This is part 
> of the purpose of bufferbloat.net.
>
> And please, please make sure everyone treads gently: we're all bozos on 
> this glass bus, and don't pick up stones.  Everyone (myself included) has 
> been making the same class of mistakes, again and again; so do be gentle. 
> The problem is truly counter intuitive to most people.
>
> I will be presenting at the IETF transport area meeting in Prague.
> Best regards,
> - Jim
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat 

Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Dave Täht]

"Richard Scheffenegger" <rscheff@gmx.at> writes:

> BTW, I found this legacy document, where the authors boldly claim that
> more buffers are always better for 802.11 networks, to circumvent
> costly TCP congestion control decisions....
> http://csl.snu.ac.kr/~ecpark/papers/TCP_WLAN_TMC08.pdf


Citing Section 3.2

"Effect of the Maximum Congestion Window Size on Fairness and
Utilization"

 "Based on the observation of asymmetric behavior of TCP congestion
 control shown in Figs. 2 and 4, we can infer that the unfairness
 problem can be alleviated by preventing packet loss from occurring. We
 can avoid packet loss due to buffer overflow by either making the
 buffer size, B, sufficiently large or by restricting the maximum
 congestion window size, Wmax . In this section, we study the effect of
 Wmax on fairness and aggregate throughput. We set B = 50 packets and
 Wmax = 10..80 packets."

I would love it if they could re-run their simulation setting "B"
according to the buffer sizes for wireless devices we are now seeing in
the field, which are in the 128..1500 packet range (not counting
retries!), under poor radio conditions.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Jim Gettys]

On 01/31/2011 12:35 PM, Dave Täht wrote:
> "Richard Scheffenegger"<rscheff@gmx.at>  writes:
>
>> BTW, I found this legacy document, where the authors boldly claim that
>> more buffers are always better for 802.11 networks, to circumvent
>> costly TCP congestion control decisions....
>> http://csl.snu.ac.kr/~ecpark/papers/TCP_WLAN_TMC08.pdf
>
>
> Citing Section 3.2
>
> "Effect of the Maximum Congestion Window Size on Fairness and
> Utilization"
>
>   "Based on the observation of asymmetric behavior of TCP congestion
>   control shown in Figs. 2 and 4, we can infer that the unfairness
>   problem can be alleviated by preventing packet loss from occurring. We
>   can avoid packet loss due to buffer overflow by either making the
>   buffer size, B, sufficiently large or by restricting the maximum
>   congestion window size, Wmax . In this section, we study the effect of
>   Wmax on fairness and aggregate throughput. We set B = 50 packets and
>   Wmax = 10..80 packets."
>
> I would love it if they could re-run their simulation setting "B"
> according to the buffer sizes for wireless devices we are now seeing in
> the field, which are in the 128..1500 packet range (not counting
> retries!), under poor radio conditions.
>
Note that lossy wireless networks behave much better than "clean" 
variable bandwidth wireless networks; packet drops due to such losses at 
least cause TCP to back off sometimes....
			- Jim

Re: [Bloat] Bloat on Layer 2 Was: ECN & AQM Hall of Fame?

[Dave Taht]

On Mon, Jan 31, 2011 at 12:41 AM, Florian Lohoff <f@zz.de> wrote:
> On Sun, Jan 30, 2011 at 11:40:05PM +0100, Richard Scheffenegger wrote:
>> Hi,
>>
>> wasn't there talk about getting a Hall of Fame for networks / operators,
>> which are using and actively supporting AQM and ECN in their administrative
>> domain?
>>


I like the hall of fame/shame idea more and more.

... as long as we can get towards benchmarks that can back it up.

I stumbled across these guys this morning, has anyone seen their stuff?

http://etherealmind.com/gnodal-ethernet-fabric/




-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net