[Bloat] AQM deployment status?

[Bloat] AQM deployment status?

[Naeem Khademi]

[-- Attachment #1: Type: text/plain, Size: 271 bytes --]

Hi

Sorry if I'm asking a redundant question here, but any citeable information
on AQM's deployment on the Internet? RED, ARED, BLUE, (FQ_)CoDel's
deployment either at the edge or core? I'm curious to know how much of it
has made it into real deployment.

Regards,
Naeem

[-- Attachment #2: Type: text/html, Size: 385 bytes --]

Re: [Bloat] [aqm] AQM deployment status?

[LOCHIN Emmanuel]

Le Mardi 24 Septembre 2013 19:21 CEST, Naeem Khademi <naeem.khademi@gmail.com> a écrit:
 +1

Emmanuel

> Hi
>
> Sorry if I'm asking a redundant question here, but any citeable information
> on AQM's deployment on the Internet? RED, ARED, BLUE, (FQ_)CoDel's
> deployment either at the edge or core? I'm curious to know how much of it
> has made it into real deployment.
>
> Regards,
> Naeem

Re: [Bloat] [aqm] AQM deployment status?

[Akhtar, Shahid (Shahid)]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Hi Naeem,

Most routers/switches/access equipment support RFC 2309, which is a description of RED. It is not fully clear how often RED is actually turned on in today's networks, the below research looked at RED deployment on access networks and concluded that only a fraction of the access lines use RED (section 4.3.2).

http://research.microsoft.com/en-us/um/people/ssaroiu/publications/imc/2007/imc2007-dischinger.pdf

I am not aware of any more recent version of the same tests.

Best regards,

-Shahid.

________________________________
From: aqm-bounces@ietf.org [mailto:aqm-bounces@ietf.org] On Behalf Of Naeem Khademi
Sent: Tuesday, September 24, 2013 12:21 PM
To: aqm@ietf.org; bloat; iccrg@irtf.org
Subject: [aqm] AQM deployment status?

Hi

Sorry if I'm asking a redundant question here, but any citeable information on AQM's deployment on the Internet? RED, ARED, BLUE, (FQ_)CoDel's deployment either at the edge or core? I'm curious to know how much of it has made it into real deployment.

Regards,
Naeem

[-- Attachment #2: Type: text/html, Size: 2789 bytes --]

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Eggert, Lars]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Hi,

On Sep 25, 2013, at 15:20, "Akhtar, Shahid (Shahid)" <shahid.akhtar@alcatel-lucent.com> wrote:
> Most routers/switches/access equipment support RFC 2309, which is a description of RED.

I've heard that statement from many different people. I wonder if it is actually true. Is there any hard data on this?

(From experience with the - small - selection of datacenter gear I have access to, almost none has usable RED/ECN support. Even in cases where the silicon is claimed to support it, the firmware doesn't allow configuration.)

Lars

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 313 bytes --]

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Steinar H. Gunderson]

On Wed, Sep 25, 2013 at 02:31:21PM +0000, Eggert, Lars wrote:
>> Most routers/switches/access equipment support RFC 2309, which is a
>> description of RED.
> I've heard that statement from many different people. I wonder if it is
> actually true. Is there any hard data on this?

I don't have hard data, but my soft data is that generally, devices branded
as switches (including L3 switches) do not support it. Routers might.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Akhtar, Shahid (Shahid)]

Hi Steinar and Lars,

Please see below examples of support for RED/WRED from switches (from ALU and Cisco websites, search for RED or WRED in document): 

ALU 7450:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&ved=0CDMQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2F7450ESS_HS_MDA_ds.pdf&ei=pS5DUrIZiI70BLyYgYgK&usg=AFQjCNGS8uB-0Ele3TpDIRqbL4p1_kd2DQ&sig2=v-AgGJMpcZjrpjP2oZIHrA&bvm=bv.53077864,d.eWU

ALU Omniswitch:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CDkQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2Fent_OS6855_datasheet_en.pdf&ei=9i1DUsXlOo788QTU94HQAQ&usg=AFQjCNFSXSEpkFjSihK1a0pkSHhItU8sZQ&sig2=3fitEZLZZI7Dmf0H2T3QbQ&bvm=bv.53077864,d.eWU

Cisco Catalyst 6000/6500:  

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html

Cisco Catalyst 4500:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps4324/white_paper_c11-539588.html

Best regards,

-Shahid.

-----Original Message-----
From: Steinar H. Gunderson [mailto:sgunderson@bigfoot.com] 
Sent: Wednesday, September 25, 2013 9:35 AM
To: Eggert, Lars
Cc: Akhtar, Shahid (Shahid); bloat; iccrg@irtf.org; aqm@ietf.org
Subject: Re: [Bloat] [iccrg] [aqm] AQM deployment status?

On Wed, Sep 25, 2013 at 02:31:21PM +0000, Eggert, Lars wrote:
>> Most routers/switches/access equipment support RFC 2309, which is a 
>> description of RED.
> I've heard that statement from many different people. I wonder if it 
> is actually true. Is there any hard data on this?

I don't have hard data, but my soft data is that generally, devices branded as switches (including L3 switches) do not support it. Routers might.

/* Steinar */
--
Homepage: http://www.sesse.net/

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Naeem Khademi]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

Thanks Shahid

Although interesting to know that (W)RED has made it into some hardware (in
your RE to Lars' point), my question is more about "deployment" at the edge
or the core, whether it's being used or not?

Cheers,
Naeem


On Wed, Sep 25, 2013 at 8:51 PM, Akhtar, Shahid (Shahid) <
shahid.akhtar@alcatel-lucent.com> wrote:

> Hi Steinar and Lars,
>
> Please see below examples of support for RED/WRED from switches (from ALU
> and Cisco websites, search for RED or WRED in document):
>
> ALU 7450:
>
>
> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&ved=0CDMQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2F7450ESS_HS_MDA_ds.pdf&ei=pS5DUrIZiI70BLyYgYgK&usg=AFQjCNGS8uB-0Ele3TpDIRqbL4p1_kd2DQ&sig2=v-AgGJMpcZjrpjP2oZIHrA&bvm=bv.53077864,d.eWU
>
> ALU Omniswitch:
>
>
> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CDkQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2Fent_OS6855_datasheet_en.pdf&ei=9i1DUsXlOo788QTU94HQAQ&usg=AFQjCNFSXSEpkFjSihK1a0pkSHhItU8sZQ&sig2=3fitEZLZZI7Dmf0H2T3QbQ&bvm=bv.53077864,d.eWU
>
> Cisco Catalyst 6000/6500:
>
>
> http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html
>
> Cisco Catalyst 4500:
>
>
> http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps4324/white_paper_c11-539588.html
>
> Best regards,
>
> -Shahid.
>
> -----Original Message-----
> From: Steinar H. Gunderson [mailto:sgunderson@bigfoot.com]
> Sent: Wednesday, September 25, 2013 9:35 AM
> To: Eggert, Lars
> Cc: Akhtar, Shahid (Shahid); bloat; iccrg@irtf.org; aqm@ietf.org
> Subject: Re: [Bloat] [iccrg] [aqm] AQM deployment status?
>
> On Wed, Sep 25, 2013 at 02:31:21PM +0000, Eggert, Lars wrote:
> >> Most routers/switches/access equipment support RFC 2309, which is a
> >> description of RED.
> > I've heard that statement from many different people. I wonder if it
> > is actually true. Is there any hard data on this?
>
> I don't have hard data, but my soft data is that generally, devices
> branded as switches (including L3 switches) do not support it. Routers
> might.
>
> /* Steinar */
> --
> Homepage: http://www.sesse.net/
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>

[-- Attachment #2: Type: text/html, Size: 4606 bytes --]

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Akhtar, Shahid (Shahid)]

[-- Attachment #1: Type: text/plain, Size: 3184 bytes --]

Naeem,

If you look into the document I initially sent

http://research.microsoft.com/en-us/um/people/ssaroiu/publications/imc/2007/imc2007-dischinger.pdf

They show that 26% of DSL lines have RED actually deployed - and tuned on!

They did not detect any RED on the cable side.

-Shahid.
________________________________
From: Naeem Khademi [mailto:naeem.khademi@gmail.com]
Sent: Wednesday, September 25, 2013 2:20 PM
To: Akhtar, Shahid (Shahid)
Cc: Steinar H. Gunderson; Eggert, Lars; iccrg@irtf.org; aqm@ietf.org; bloat
Subject: Re: [aqm] [Bloat] [iccrg] AQM deployment status?

Thanks Shahid

Although interesting to know that (W)RED has made it into some hardware (in your RE to Lars' point), my question is more about "deployment" at the edge or the core, whether it's being used or not?

Cheers,
Naeem


On Wed, Sep 25, 2013 at 8:51 PM, Akhtar, Shahid (Shahid) <shahid.akhtar@alcatel-lucent.com<mailto:shahid.akhtar@alcatel-lucent.com>> wrote:
Hi Steinar and Lars,

Please see below examples of support for RED/WRED from switches (from ALU and Cisco websites, search for RED or WRED in document):

ALU 7450:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&ved=0CDMQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2F7450ESS_HS_MDA_ds.pdf&ei=pS5DUrIZiI70BLyYgYgK&usg=AFQjCNGS8uB-0Ele3TpDIRqbL4p1_kd2DQ&sig2=v-AgGJMpcZjrpjP2oZIHrA&bvm=bv.53077864,d.eWU

ALU Omniswitch:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CDkQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2Fent_OS6855_datasheet_en.pdf&ei=9i1DUsXlOo788QTU94HQAQ&usg=AFQjCNFSXSEpkFjSihK1a0pkSHhItU8sZQ&sig2=3fitEZLZZI7Dmf0H2T3QbQ&bvm=bv.53077864,d.eWU

Cisco Catalyst 6000/6500:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html

Cisco Catalyst 4500:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps4324/white_paper_c11-539588.html

Best regards,

-Shahid.

-----Original Message-----
From: Steinar H. Gunderson [mailto:sgunderson@bigfoot.com<mailto:sgunderson@bigfoot.com>]
Sent: Wednesday, September 25, 2013 9:35 AM
To: Eggert, Lars
Cc: Akhtar, Shahid (Shahid); bloat; iccrg@irtf.org<mailto:iccrg@irtf.org>; aqm@ietf.org<mailto:aqm@ietf.org>
Subject: Re: [Bloat] [iccrg] [aqm] AQM deployment status?

On Wed, Sep 25, 2013 at 02:31:21PM +0000, Eggert, Lars wrote:
>> Most routers/switches/access equipment support RFC 2309, which is a
>> description of RED.
> I've heard that statement from many different people. I wonder if it
> is actually true. Is there any hard data on this?

I don't have hard data, but my soft data is that generally, devices branded as switches (including L3 switches) do not support it. Routers might.

/* Steinar */
--
Homepage: http://www.sesse.net/
_______________________________________________
aqm mailing list
aqm@ietf.org<mailto:aqm@ietf.org>
https://www.ietf.org/mailman/listinfo/aqm


[-- Attachment #2: Type: text/html, Size: 7062 bytes --]

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Mikael Abrahamsson]

On Wed, 25 Sep 2013, Akhtar, Shahid (Shahid) wrote:

> Please see below examples of support for RED/WRED from switches (from ALU and Cisco websites, search for RED or WRED in document):

I'd venture to claim that putting RED on a device with a few milliseconds 
worth of buffer depth is pretty much useless (for instance the Cisco 6704 
or 6724 linecards). So most datacenter equipment doesn't have this, or if 
they do, it's not that usable even if they had (who cares about drop 
probability when taildrop is being done at 4-5 ms buffer depth?).

For higher end platforms, for instance all cisco CPU based routers (for 
some value of "all") can be configured with RED, fair-queue or similar, 
but they come with FIFO as default. This has been the same way since at 
least the mid 90ties as far as I know, long way back to cisco 1600 device 
etc.

Higher end Cisco equipment such as ASR9k, 12000, CRS etc, all support 
WRED, and here it makes sense since they all have ~50ms worth of buffering 
or more. They also come with FIFO as default setting.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Scheffenegger, Richard]

HI Mikael,

[chair-hat off]

> I'd venture to claim that putting RED on a device with a few milliseconds
> worth of buffer depth is pretty much useless (for instance the Cisco 6704
> or 6724 linecards). So most datacenter equipment doesn't have this, or if
> they do, it's not that usable even if they had (who cares about drop
> probability when taildrop is being done at 4-5 ms buffer depth?).

I'd state that people operating datacenters with request-response type data exchange via TCP do care a lot about the microscopic drop distribution. Typically, a tail-drop queue has the unfortunate property of losing the more critical packets of such an exchange, leading to excessive "transaction" (higher level protocol) delays due to lengthy TCP loss recovery.

Agreed, that is not the regular "internet" use case, but the same gear is being deployed there...

Richard Scheffenegger



> -----Original Message-----
> From: iccrg-bounces@irtf.org [mailto:iccrg-bounces@irtf.org] On Behalf Of
> Mikael Abrahamsson
> Sent: Mittwoch, 25. September 2013 21:25
> To: iccrg@irtf.org
> Cc: aqm@ietf.org; bloat
> Subject: Re: [iccrg] [aqm] [Bloat] AQM deployment status?
> 
> On Wed, 25 Sep 2013, Akhtar, Shahid (Shahid) wrote:
> 
> > Please see below examples of support for RED/WRED from switches (from
> ALU and Cisco websites, search for RED or WRED in document):
> 
> I'd venture to claim that putting RED on a device with a few milliseconds
> worth of buffer depth is pretty much useless (for instance the Cisco 6704
> or 6724 linecards). So most datacenter equipment doesn't have this, or if
> they do, it's not that usable even if they had (who cares about drop
> probability when taildrop is being done at 4-5 ms buffer depth?).
> 
> For higher end platforms, for instance all cisco CPU based routers (for
> some value of "all") can be configured with RED, fair-queue or similar,
> but they come with FIFO as default. This has been the same way since at
> least the mid 90ties as far as I know, long way back to cisco 1600 device
> etc.
> 
> Higher end Cisco equipment such as ASR9k, 12000, CRS etc, all support
> WRED, and here it makes sense since they all have ~50ms worth of buffering
> or more. They also come with FIFO as default setting.
> 
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Mikael Abrahamsson]

On Thu, 26 Sep 2013, Scheffenegger, Richard wrote:

> I'd state that people operating datacenters with request-response type 
> data exchange via TCP do care a lot about the microscopic drop 
> distribution. Typically, a tail-drop queue has the unfortunate property 
> of losing the more critical packets of such an exchange, leading to 
> excessive "transaction" (higher level protocol) delays due to lengthy 
> TCP loss recovery.

Do you know of simulations or similar that investigates this? I would like 
to understand why having RED on a 3ms buffer depth device makes a 
difference and how much difference it makes.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Eggert, Lars]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

On Sep 27, 2013, at 4:59, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> Do you know of simulations or similar that investigates this? I would like to understand why having RED on a 3ms buffer depth device makes a difference and how much difference it makes.

Because saving 3ms on a path with microsecond latency is a big deal?

Lars

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 273 bytes --]

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Mikael Abrahamsson]

On Fri, 27 Sep 2013, Eggert, Lars wrote:

> On Sep 27, 2013, at 4:59, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> Do you know of simulations or similar that investigates this? I would like to understand why having RED on a 3ms buffer depth device makes a difference and how much difference it makes.
>
> Because saving 3ms on a path with microsecond latency is a big deal?

Yes, of course. I am not a datacenter guy so this is not a problem I face, 
because most of packets in my environment travel tens or hundreds of 
milliseconds.

So in datacenter one wants to start marking ECN on packets very soon into 
buffer depth, hoping sender will get feedback and throttle back the speed 
way before one gets taildrops?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Eggert, Lars]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

On Sep 28, 2013, at 9:01, Mikael Abrahamsson <swmike@swm.pp.se>
 wrote:
> So in datacenter one wants to start marking ECN on packets very soon into buffer depth, hoping sender will get feedback and throttle back the speed way before one gets taildrops?

Yep. There have been a bunch of papers on datacenter TCP variants recently (look through the last 2-3 years of SIGCOMM papers, all online.)

Lars

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 273 bytes --]

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Bob Briscoe]

Mikael,

The shallow marking theshold is not the most significant feature of 
the AQM in DCTCP. More importantly, it does not delay congestion 
signals by averaging them over a period equivalent to a worst-case 
RTT (about 100ms), like all other AQMs do (RED, PIE, CoDel etc).

The shallow marking threshold certainly keeps standing queuing delay 
low. However, that's only under long-running constant conditions. 
During dynamics, not waiting a few hundred msec to respond to a 
change in the queue is what keeps the queuing delay predictably low. 
Dynamics are the norm, not constant conditions.

For instance, the AQM in DCTCP signals to a flow in slow-start as 
soon as it crosses the threshold. So a flow with 20ms RTT will get 
the signal in 20ms, and a flow with 3ms RTT will get the signal in 
3ms (e.g. to or from a CDN cache). Whereas all other AQMs will delay 
all signals for of the order of 100ms, even if the flow has a much 
shorter RTT (because all these other AQMs don't know the RTT of each 
flow, so they use a nominal worst-case RTT).

The source can use the signal as soon as it arrives (which it does at 
the end of slow-start), or it can smooth the signal itself (which it 
does once it's in congestion avoidance phase). But it can smooth the 
signal over its own RTT, rather than guessing a worst-case RTT.

CoDel taught us that the best line-rate auto-tuning an AQM can do is 
to use service time. DCTCP teaches us that the best RTT auto-tuning 
an AQM can do is /not/ to try to guess the RTT in the first place. 
Instead it is best to defer anything to do with RTT to the end-system.

We should learn from both lessons.



Bob

At 08:04 28/09/2013, Eggert, Lars wrote:
>Content-Language: en-US
>Content-Type: multipart/signed;
>         boundary="Apple-Mail=_19C08185-C4A7-46F5-925E-BA32C00EB99A";
>         protocol="application/pgp-signature"; micalg=pgp-sha1
>
>On Sep 28, 2013, at 9:01, Mikael Abrahamsson <swmike@swm.pp.se>
>  wrote:
> > So in datacenter one wants to start marking ECN on packets very 
> soon into buffer depth, hoping sender will get feedback and 
> throttle back the speed way before one gets taildrops?
>
>Yep. There have been a bunch of papers on datacenter TCP variants 
>recently (look through the last 2-3 years of SIGCOMM papers, all online.)
>
>Lars
>
>
>_______________________________________________
>aqm mailing list
>aqm@ietf.org
>https://www.ietf.org/mailman/listinfo/aqm

________________________________________________________________
Bob Briscoe,                                                  BT 

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Mikael Abrahamsson]

On Sun, 29 Sep 2013, Bob Briscoe wrote:

> The shallow marking threshold certainly keeps standing queuing delay 
> low. However, that's only under long-running constant conditions. During 
> dynamics, not waiting a few hundred msec to respond to a change in the 
> queue is what keeps the queuing delay predictably low. Dynamics are the 
> norm, not constant conditions.

Well, my original question was in the context of a 3-5ms tail drop queue 
(such as are frequently available in lower end switches). My understanding 
from earlier experience is that TCP will severely saw-tooth under these 
conditions and only way I could see marking as being valuable was if the 
RTT was less than buffer depth (or at least very low).

There was a discussion earlier on bloat-l regarding what the impact of a 
10ms CoDel queuing scheme will have on 200ms RTT existing non-ECN TCP 
performance. Deep queues were invented to handle this specific use-case.

One way would absolutely be for TCP to not send a lot of packets 
back-to-back but instead pace them out at the rate that actually makes the 
TCP rate be exact in a millisecond resolution instead of as it is today, 
perhaps tens or hundreds of milliseconds. I believe some implementations 
already do this.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Scheffenegger, Richard]

Hi Mikael,

the papers around DCTCP have probably the information you are looking for (map/reduce type workloads, leading to incast issues - momentary filling of queues; loosing the final segments of a TCP session [1], which is likely with drop tail queue management policy, is well known to result in timely retransmission timeout-type recoveries. Tweaking OS time granularities down to recover in few milliseconds on paths that usually have a few dozen microseconds delay is often not good enough, but even that is not for cautious.

Apparently, this is seen important and valuable enough to motivate the unilateral deployment of DCTCP with Windows 8 server, even though the modification in the switches is then still necessary.

Richard Scheffenegger

[1] Google has presented the liklihood of packet loss per segment vs. burst size; Figure 3 in this paper http://plus.url.google.com/url?sa=z&n=1380288903097&url=http%3A%2F%2Fstatic.googleusercontent.com%2Fexternal_content%2Funtrusted_dlcp%2Fresearch.google.com%2Fen%2F%2Fpubs%2Farchive%2F41217.pdf&usg=lEiaJfELSWQoGNsDNsbQgbSO9RQ.



> -----Original Message-----
> From: Mikael Abrahamsson [mailto:swmike@swm.pp.se]
> Sent: Freitag, 27. September 2013 04:59
> To: Scheffenegger, Richard
> Cc: iccrg@irtf.org; aqm@ietf.org; bloat
> Subject: RE: [iccrg] [aqm] [Bloat] AQM deployment status?
> 
> On Thu, 26 Sep 2013, Scheffenegger, Richard wrote:
> 
> > I'd state that people operating datacenters with request-response type
> > data exchange via TCP do care a lot about the microscopic drop
> > distribution. Typically, a tail-drop queue has the unfortunate
> > property of losing the more critical packets of such an exchange,
> > leading to excessive "transaction" (higher level protocol) delays due
> > to lengthy TCP loss recovery.
> 
> Do you know of simulations or similar that investigates this? I would like
> to understand why having RED on a 3ms buffer depth device makes a
> difference and how much difference it makes.
> 
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Mikael Abrahamsson]

On Fri, 27 Sep 2013, Scheffenegger, Richard wrote:

> Hi Mikael,
>
> the papers around DCTCP have probably the information you are looking 
> for (map/reduce type workloads, leading to incast issues - momentary 
> filling of queues; loosing the final segments of a TCP session [1], 
> which is likely with drop tail queue management policy, is well known to 
> result in timely retransmission timeout-type recoveries. Tweaking OS 
> time granularities down to recover in few milliseconds on paths that 
> usually have a few dozen microseconds delay is often not good enough, 
> but even that is not for cautious.

Ok, so this makes sense within a data center with ECN, but I fail to see 
any substantial upside deploying this in an environment where RTT is in 
the tens of ms because TCP won't have time to react.

I seem to remember reading about enhancement to TCP where packets are 
actually sent out with spacing between packets instead of in line-rate 
bursts. If everybody started doing that then I would imagine there would 
be less tail-drop in large RTT environments with large TCP windows?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [aqm] [iccrg] AQM deployment status?

[Fred Baker (fred)]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]


On Sep 25, 2013, at 12:24 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> For higher end platforms, for instance all cisco CPU based routers (for some value of "all") can be configured with RED, fair-queue or similar, but they come with FIFO as default. This has been the same way since at least the mid 90ties as far as I know, long way back to cisco 1600 device etc.
> 
> Higher end Cisco equipment such as ASR9k, 12000, CRS etc, all support WRED, and here it makes sense since they all have ~50ms worth of buffering or more. They also come with FIFO as default setting.

Yes. There are two reasons that we don't deploy RED by default. 

One is that we operate on a principle we call the "principle of least surprise"; we try to not change our customer's configurations by changing the defaults, and at the time I wrote the RED/WRED code the default was FIFO. Yes, that was quite some time ago, and one could imagine newer equipment coming out with different defaults on deployment, but that's not what the various business units do. 

The other is this matter of configuration. In the late 1990's, Cisco employed Van and Kathy, and asked them to figure out autoconfiguration values. That didn't work out. I don't say that as a slam on V+K; it's simply a fact. They started by recommending an alternative algorithm called RED-Lite (don't bother googling that; you get discothèques), and are now recommending CoDel. Now, there is probably a simplistic value that one could throw in, such as setting max-threshold to the queue memory allocated to the queue and min-threshold to some function logarithmically related to the bit rate that serves as an estimator of the number of bit-times of delay to tune to. That would probably be better than nothing, but it would be nice to have some science behind it.

That brings us back to the auto-tuning discussion.

To my way of thinking, a simplistic algorithm such as that logarithmic approach or a lookup table that starts from (perhaps) bit rate and neighbor ping RTT can come up with an initial set of parameters that the algorithm's own processes can modify to ambient traffic behavior. Kathy's simulations in the study I mentioned suggested that a 1.5 MBPS link might do well with min-threshold at 30 ms (e.g., ~4 MTU-sized messages or a larger number of smaller messages), and rates higher at some single digit number of ms, decreasing as the speed increased. 

CoDel suggests a flat 5 ms value (at 1.5 MBPS, less than a single MTU, and at one gigabit, 83 12K bit messages). I could imagine the initialization algorithm selecting 5 ms above a certain speed, and the equivalent of 4 MTU intervals at lower line speeds where 4*MTU exceeds 5 ms. I could imagine a related algorithm then adjusting that initial value to interact well with Google's IW=10 behavior or whatever else it finds on the link. 

PIE would probably start with a similar set of values, and tune its mark/drop interval to reach a value that works well with ambient traffic.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [Bloat] [iccrg] [aqm] AQM deployment status?

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]


Eggert, Lars <lars@netapp.com> wrote:
    > I've heard that statement [that RED is supported] from many different
    > people. I wonder if it is
    > actually true. Is there any hard data on this?

The issue I've had is that while RED is supported on a routing device,
and I can enable it on a per-VLAN basis, that does me little good, because
there are bottlenecks in layer-2 devices (ethernet, but also DSL,GPON, etc.)
which are out of my control.

What I would like is a standard layer-3 way to measure bandwidth across
my layer-3 hop, and autoconfigure RED (or CoDel) to the measured bandwidth.
My understanding is that some equipment can use 802.1ad CCP and/or Loopback
frames to measure bandwidth across bridged layer-2 ethernet networks.

I'm unclear if this is a feature of that equipment, or a part of the
standard.  My understanding is that the math involved is pretty
straightforward, but getting it right in code can involve dealing with a
number of numerical issues which makes it slightly non-trivial to
implement. (And you need good low-latency time stamping of packets)

--
Michael Richardson <mcr+IETF@sandelman.ca>, Sandelman Software Works



[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]