[Bloat] fq_codel, high bandwidth, and delays

[Bloat] fq_codel, high bandwidth, and delays

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

List,

In talking with a friend over the weekend that moves data around for the
national labs (on links at rates like 10Gbps), we ended up having a rather
interesting discussion about just how radically different the problem
spaces are vs. what he's seen in the bufferbloat community.

They have few flows, long lived, and are trying to push >1Gbps per flow,
across the continent (or from Europe to the US), with inherent delays on
the order of 100ms.  TCP under these conditions is, from his reports,
incredibly fragile, where even a tiny packet error rate stops TCP for
saturating the link (since it can't tell the difference between congestion
and a non-congestion-related dropped packet).

And suddenly the "every packet is precious" mode of thought becomes crystal
clear.

Clearly they are trying to solve different problems, yet they do have
congestion events, when new flows are added to the network.

Has anyone used fq_codel (or it's friends) in scenarios like this?  fq is
fairly new (2 years?) and I can't find much about it and high bandwidth
links in my searches.

Given that their problems aren't those that fq is trying to solve, I
wouldn't expect it, but curious to see if anyone has any research on it.

Thanks,
Aaron

[-- Attachment #2: Type: text/html, Size: 1482 bytes --]

Re: [Bloat] fq_codel, high bandwidth, and delays

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 1662 bytes --]

On Mon, 7 Jul 2014, Aaron Wood wrote:

>
> In talking with a friend over the weekend that moves data around for the
> national labs (on links at rates like 10Gbps), we ended up having a rather
> interesting discussion about just how radically different the problem
> spaces are vs. what he's seen in the bufferbloat community.
>
> They have few flows, long lived, and are trying to push >1Gbps per flow,
> across the continent (or from Europe to the US), with inherent delays on
> the order of 100ms.  TCP under these conditions is, from his reports,
> incredibly fragile, where even a tiny packet error rate stops TCP for
> saturating the link (since it can't tell the difference between congestion
> and a non-congestion-related dropped packet).
>
> And suddenly the "every packet is precious" mode of thought becomes crystal
> clear.

Part of the answer for them is "don't do that" :-)

There are many tools around to move massive amounts of data through multiple 
parallel TCP connections so that if one connection gets a lost packet, it 
doesn't kill the entire flow, only the one connection slows down (if it's true 
congestion, then all the flows will be slowed)

David Lang

> Clearly they are trying to solve different problems, yet they do have
> congestion events, when new flows are added to the network.
>
> Has anyone used fq_codel (or it's friends) in scenarios like this?  fq is
> fairly new (2 years?) and I can't find much about it and high bandwidth
> links in my searches.
>
> Given that their problems aren't those that fq is trying to solve, I
> wouldn't expect it, but curious to see if anyone has any research on it.
>
> Thanks,
> Aaron
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 140 bytes --]

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fq_codel, high bandwidth, and delays

[Neil Davies]

Aaron

We did the analysis and optimisation issue with CERN about 10 years ago. The ATLAS team has the answers. 

Your friend needs to investigate the pattern of the packet loss, not just its rate [dynamics of queues is an interesting issue, INRIA tech reports are a good start]. Are they sharing this bandwidth (i.e how stationary are the data transport properties). All that helps you work out the optimal window size (probably just below the delay bandwidth product for the link). Making a single TCP session saturate such links is always hard, as for a handful - much easier - as long as they don't create their own self-syncing DOS attack [at the key constrained/variable resource point on the path].

Neil

On 8 Jul 2014, at 05:10, Aaron Wood <woody77@gmail.com> wrote:

> List,
> 
> In talking with a friend over the weekend that moves data around for the national labs (on links at rates like 10Gbps), we ended up having a rather interesting discussion about just how radically different the problem spaces are vs. what he's seen in the bufferbloat community.
> 
> They have few flows, long lived, and are trying to push >1Gbps per flow, across the continent (or from Europe to the US), with inherent delays on the order of 100ms.  TCP under these conditions is, from his reports, incredibly fragile, where even a tiny packet error rate stops TCP for saturating the link (since it can't tell the difference between congestion and a non-congestion-related dropped packet).
> 
> And suddenly the "every packet is precious" mode of thought becomes crystal clear.
> 
> Clearly they are trying to solve different problems, yet they do have congestion events, when new flows are added to the network.
> 
> Has anyone used fq_codel (or it's friends) in scenarios like this?  fq is fairly new (2 years?) and I can't find much about it and high bandwidth links in my searches.
> 
> Given that their problems aren't those that fq is trying to solve, I wouldn't expect it, but curious to see if anyone has any research on it.
> 
> Thanks,
> Aaron
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fq_codel, high bandwidth, and delays

[David Collier-Brown]

David Lang <david@lang.hm> wrote:

> Part of the answer for them is "don't do that" :-)
> 
> There are many tools around to move massive amounts of data through multiple 
> parallel TCP connections so that if one connection gets a lost packet, it 
> doesn't kill the entire flow, only the one connection slows down (if it's true 
> congestion, then all the flows will be slowed)
> 
> David Lang

Can you suggest one that also allows the total bandwidth to be limited?

I recollect similar problem distributing "films" to early digital
theatres, and their fix for low throughput needed an anti-fix for
induced congestion (;-))

--dave
-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain

Re: [Bloat] fq_codel, high bandwidth, and delays

[David Lang]

On Tue, 8 Jul 2014, David Collier-Brown wrote:

> Date: Tue, 08 Jul 2014 15:43:40 -0400
> David Lang <david@lang.hm> wrote:
>
>> Part of the answer for them is "don't do that" :-)
>>
>> There are many tools around to move massive amounts of data through multiple
>> parallel TCP connections so that if one connection gets a lost packet, it
>> doesn't kill the entire flow, only the one connection slows down (if it's true
>> congestion, then all the flows will be slowed)
>>
>> David Lang
>
> Can you suggest one that also allows the total bandwidth to be limited?

take a look at bbcp, it lets you tune a lot of things, including total bandwidth

http://www.slac.stanford.edu/~abh/bbcp/

David Lang