[Bloat] Steam's download CDNs - breaking bufferbloat and inbound policers

[Bloat] Steam's download CDNs - breaking bufferbloat and inbound policers

[cloneman]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

Hi,

Apologies in advance if this is the wrong place to ask.

I'm looking for any comments on Steam's game distribution download system -
specifically how it defeats any bufferbloat management system I've used.

It seems to push past inbound policers, exceeding them by about 40%. That
is to say, you must police steam traffic to half your line rate, then
enough capacity will remain to avoid packet loss, latency, etc. Obviously
this is too much bandwidth to reserve for practical use.

Without any inbound control, you can expect very heavy packet loss and
jitter. With fq_codel or sfq and taking the usual recommended 15% off the
table, you get improved, but still unacceptable performance in your small
flows / ping etc.

The behavior can be observed by downloading any free game on their
platform. I'm trying to figure out how they've accomplished this and how to
mitigate this behavior. It operates with 20 http connections
simultaneously, which is normally not an issue (20 multiple web downloads
perform well under fq_codel and 15% reserve bandwidth)

Note: in my testing cable and vdsl below 100mbit were vulnerable to this
behavior, while fiber was immune.

Basically there are edge application cases on the internet that like to
push too many bytes down a line. I would like to see some discussion or
testing of this issue.

I haven't tried tweaking any of the parameters / latency targets in
fq_codel.

[-- Attachment #2: Type: text/html, Size: 2566 bytes --]

Re: [Bloat] Steam's download CDNs - breaking bufferbloat and inbound policers

[Jonathan Morton]

> On 27 Apr, 2017, at 07:19, cloneman <bufferbloat@flamingpc.com> wrote:
> 
> I'm looking for any comments on Steam's game distribution download system - specifically how it defeats any bufferbloat management system I've used.  
> 
> It seems to push past inbound policers, exceeding them by about 40%. That is to say, you must police steam traffic to half your line rate, then enough capacity will remain to avoid packet loss, latency, etc. Obviously this is too much bandwidth to reserve for practical use.
> 
> Without any inbound control, you can expect very heavy packet loss and jitter. With fq_codel or sfq and taking the usual recommended 15% off the table, you get improved, but still unacceptable performance in your small flows / ping etc. 
> 
> The behavior can be observed by downloading any free game on their platform. I'm trying to figure out how they've accomplished this and how to mitigate this behavior. It operates with 20 http connections simultaneously, which is normally not an issue (20 multiple web downloads perform well under fq_codel and 15% reserve bandwidth) 

Have you tried using Cake in its new ingress mode?  That counts dropped packets against the shaper, ensuring that the load they already imposed on the link upstream of the shaper is accounted for.  This is probably impossible to implement with a separate shaper and AQM (eg. HTB + fq_codel).

I’ve also found that Steam responds well to ECN, at least on my local instances, so you should turn ECN on fully on your end-hosts.  I would dearly love to see ECN on by default, but so far only Apple (of all vendors!) has shown that level of courage.

 - Jonathan Morton

Re: [Bloat] Steam's download CDNs - breaking bufferbloat and inbound policers

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu, 27 Apr 2017 at 06:19 cloneman <bufferbloat@flamingpc.com> wrote:

> The behavior can be observed by downloading any free game on their
> platform. I'm trying to figure out how they've accomplished this and how to
> mitigate this behavior. It operates with 20 http connections
> simultaneously, which is normally not an issue (20 multiple web downloads
> perform well under fq_codel and 15% reserve bandwidth)
>

I think it starts new connections quite often as it hops content delivery
nodes to find the fastest ones, perhaps this contributes to the problem?

[-- Attachment #2: Type: text/html, Size: 918 bytes --]