Re: [Codel] [aqm] codel with low shape rates

Re: [Codel] [aqm] codel with low shape rates

[Dave Täht]

On 1/18/16 2:11 PM, Jeff Weeks wrote:
> Hello all,
> 
> I'm wondering if there's some data on Codel with low shape rates?
> 
> In particular, I'm talking about in the kbps ranges.

We recently did some testing of several codel variants at very low rates
(2mbit/384kbit asymmetric). One flent dataset is here:

http://snapon.cs.kau.se/~d/384k/

There are a couple others. I didn't take the time to create graphs and
collate results before leaving for christmas vacation, the closest
I came to that w/graphs was this post:

https://lists.bufferbloat.net/pipermail/cake/2016-January/001808.html

pie was miserable, also. (it has a 10k estimator needed)

> 
> In my investigation, it seems as though the algorithm can't control latency as effectively.
> 
> For one, the algorithm requires 2 packets in the queue to operate, but at a low shape rate, it's highly likely that all packets in the queue will have high latency, so 'count' will begin to ramp up... but conversely, it's also likely that we drain the queue, and then leave the drop state (and wont re-enter it for at least an interval's worth of time (100ms)).

This is not quite true - it's bytes, not packets. An MTU's worth of
bytes is the std codel limit before switching off.

Additionally, when htb is used, there is at least an htb quantum's worth
of packets that queue also. (the "cake" variant does not have this
problem. After I published the bcake vs cake results above, the cake
code got improved)

> Effectively, we get an oscillation of "in drop state, out of drop state", and we're not in the drop state for long enough to ramp up count, because the queue itself (proportional to the shape rate) isn't very big.

I don't see this, we generally end up with a persistently >1 packet
queue with two or more flows going. I see is a worse problem where the
basic attributes of keeping a connection up and things like slow start,
and with multiple flows, result in count climbing excessively high,
especially with ecn in use.

More research at sub 2mbit speeds is needed. Am pretty happy with things
in the 4mbit to 40gbit range.

> Are there way to combat this, or am I misinterpreting a portion of the algorithm?

The simplest way to get decent results is to set target slightly more
than the MTU at the speed you are running at. e.g. 1Mbit = target 13ms.
This is essentially what the sqm-scripts and cake do - and the results
are still less than fully desirable.

(I would like very much to have an aqm that was knobless at these
speeds. Most of the work on trying to improve matters at these rates is
in the multitude of "cake" variants)

> 
> Thanks,
> Jeff
> 
> 
> ________________________________
> /dev/jeff_weeks.x2936
> Sandvine Incorporated
> 
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
> 

Re: [Codel] [aqm] codel with low shape rates

[Agarwal, Anil]

[-- Attachment #1: Type: text/plain, Size: 5063 bytes --]

Dave, Jeff,



A quick simulation of Codel with a single queue, with -

               link rate  = 100 kbps,

               packet size = 500 bytes,

               traffic rate = 110 kbps, CBR, unresponsive



shows queue size stabilizes at 1-3 packets.

Count remains at 1.

Queueing delays are 50-86 ms, but that is understandable, given that the packet transmission time is 40 ms.

The algorithm changes state frequently (every ~400 ms), but that is how it works.



Jeff - what scenario are you studying?



Anil



-----Original Message-----
From: Codel [mailto:codel-bounces@lists.bufferbloat.net] On Behalf Of Dave Täht
Sent: Monday, January 18, 2016 5:58 PM
To: Jeff Weeks; aqm@ietf.org
Cc: cake@lists.bufferbloat.net; codel@lists.bufferbloat.net
Subject: Re: [Codel] [aqm] codel with low shape rates







On 1/18/16 2:11 PM, Jeff Weeks wrote:

> Hello all,

>

> I'm wondering if there's some data on Codel with low shape rates?

>

> In particular, I'm talking about in the kbps ranges.



We recently did some testing of several codel variants at very low rates (2mbit/384kbit asymmetric). One flent dataset is here:



https://urldefense.proofpoint.com/v2/url?u=http-3A__snapon.cs.kau.se_-7Ed_384k_&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=dlQhigd0CU9PuDq2KSsG-73u2A01K4KEagZVPJzuYc8&e=



There are a couple others. I didn't take the time to create graphs and collate results before leaving for christmas vacation, the closest I came to that w/graphs was this post:



https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_pipermail_cake_2016-2DJanuary_001808.html&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=sF1-9PiJpAY41luL-EjJxf2SFPSNNEkILZ0AizH9PL0&e=



pie was miserable, also. (it has a 10k estimator needed)



>

> In my investigation, it seems as though the algorithm can't control latency as effectively.

>

> For one, the algorithm requires 2 packets in the queue to operate, but at a low shape rate, it's highly likely that all packets in the queue will have high latency, so 'count' will begin to ramp up... but conversely, it's also likely that we drain the queue, and then leave the drop state (and wont re-enter it for at least an interval's worth of time (100ms)).



This is not quite true - it's bytes, not packets. An MTU's worth of bytes is the std codel limit before switching off.



Additionally, when htb is used, there is at least an htb quantum's worth of packets that queue also. (the "cake" variant does not have this problem. After I published the bcake vs cake results above, the cake code got improved)



> Effectively, we get an oscillation of "in drop state, out of drop state", and we're not in the drop state for long enough to ramp up count, because the queue itself (proportional to the shape rate) isn't very big.



I don't see this, we generally end up with a persistently >1 packet queue with two or more flows going. I see is a worse problem where the basic attributes of keeping a connection up and things like slow start, and with multiple flows, result in count climbing excessively high, especially with ecn in use.



More research at sub 2mbit speeds is needed. Am pretty happy with things in the 4mbit to 40gbit range.



> Are there way to combat this, or am I misinterpreting a portion of the algorithm?



The simplest way to get decent results is to set target slightly more than the MTU at the speed you are running at. e.g. 1Mbit = target 13ms.

This is essentially what the sqm-scripts and cake do - and the results are still less than fully desirable.



(I would like very much to have an aqm that was knobless at these speeds. Most of the work on trying to improve matters at these rates is in the multitude of "cake" variants)



>

> Thanks,

> Jeff

>

>

> ________________________________

> /dev/jeff_weeks.x2936

> Sandvine Incorporated

>

> _______________________________________________

> aqm mailing list

> aqm@ietf.org<mailto:aqm@ietf.org>

> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail

> man_listinfo_aqm&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fP

> k&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3a

> efVdgD3p1luGVD6HQAWZt7A&s=pDBHTaIvFZbGIz6xTyl2PwKyBA7KXP-nL5w-nIfT4gc&

> e=

>

_______________________________________________

Codel mailing list

Codel@lists.bufferbloat.net<mailto:Codel@lists.bufferbloat.net>

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_codel&d=BQIGaQ&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=MfiDsixBXXTDIMhuHe3aefVdgD3p1luGVD6HQAWZt7A&s=aQhUguCaMy2FpF_zUnTrqGCflO0G-FVhW2L8LT9NyDs&e=

[-- Attachment #2: Type: text/html, Size: 31161 bytes --]

Re: [Codel] [aqm] codel with low shape rates

[Jonathan Morton]

> On 19 Jan, 2016, at 19:32, Jeff Weeks <jweeks@sandvine.com> wrote:
> 
> Dave, What does HTB stand for?  I'm not familiar with it.

Hierarchical Token Bucket.  It’s one of the two most commonly used shapers on Linux, the other being HFSC.

> Perhaps this is an area where fq_codel would perform better?

Very likely.  My experience is that flow isolation (which is what fq_codel adds over plain codel) gives the single biggest improvement to perceived traffic quality of all the techniques I’ve seen and used.  Codel also works significantly better when it only needs to signal to one flow at a time.

If you plan to try out HTB+fq_codel or HFSC+fq_codel, then I would also suggest trying out Cake.  It’s slightly more involved to set up since it’s not yet upstream, but it’s a no-knobs-needed solution using even more sophisticated and comprehensive algorithms than fq_codel.  I’ve just finished adding “triple isolation” to it, and I plan to make it the default in the near future, unless someone turns up a major problem.

- Jonathan Morton

Re: [Codel] [aqm] codel with low shape rates

[Jonathan Morton]

> On 19 Jan, 2016, at 15:43, Agarwal, Anil <Anil.Agarwal@viasat.com> wrote:
> 
> (I would like very much to have an aqm that was knobless at these speeds. Most of the work on trying to improve matters at these rates is in the multitude of "cake" variants)

Codel is pretty close to being self-tuning in terms of link capacity, as long as it’s on the bottleneck and only needs to deal with one flow per instance.  The main trouble is when the inherent serialisation delay becomes comparable to the time constants assumed for the total estimated RTT.  On very slow links, the a-priori RTT estimate needs to be increased to account for that.

Cake has an advantage here in knowing the link capacity, via its built-in shaper.  This is how it knows what fine-tuning to apply.  It’s hard for a “pure” AQM qdisc to gather that information automatically.

With that self-tuning, Cake works usably well down to 64Kbps.  At that level, typical modern TCPs (especially CUBIC) saturate the link continuously from the outset, so Codel ends up constantly signalling them to slow down.  With ECN on, this is harmless.

 - Jonathan Morton