[Bloat] Exploring the potential of codel, fq_codel, and qfq

[dave taht]

On 05/15/2012 11:09 PM, Eric Dumazet wrote:
> On Wed, 2012-05-16 at 08:55 +0300, Jonathan Morton wrote:
>> On 9 May, 2012, at 4:04 pm, Kevin Gross wrote:
>>
>>>  From the paper (figure 7), you can see that CoDel still leaves spikes of buffer occupancy when network conditions change. These will still be disruptive to real-time traffic. Many networks that need QoS now will still need QoS. Networks that do not have QoS will be much more usable with CoDel.
>> Combining AQM with FQ certainly seems like a good idea to me.  I haven't had a chance to try the implementation of fq_codel which already exists yet, but it's compiled and just waiting for me to get around to it.  If it works, then it should be an excellent default.
>
> fq_codel has no priority concepts. It's only Fair. But what is
> Fairness ?
>
> In the presence of elephant flows, you get more packets drops from
> elephant flows than from thin ones.
The overall effects of fq_codel's implementation on top of codel's idea 
of good queue vs bad queue
has yet to be evaluated. I'd like to see people explore iw10 and TCP 
mice, for example on an apache benchmark emulating a typical web site, 
as well as typical accesses of a web site, as well as TCP_RR stats 
against saturating forward and reverse traffic, with and without ecn, at 
rates ranging from 1Mbit to 40GigE.

I'm hoping to get some of that said myself, but it's a lot to do, so I 
hope more people jump on it.

Then there's wireless.

All that said, I am loving what I see of fq_codel so far. I think in 
general fq is superior to anything a human fiddling with priorities can do.

Also I have hooked up qfq to codel in the debloat script, which might 
provide some useful comparisons to fq_codel, and also offers the 
possibility of using qfq to hash on device, and fq_codel to fq within 
flows to a given device, thus yielding 1/d behavior.

But I am throwing ENOTIME, ENOBRAINCELLS, and ENOENERGY errors a lot 
personally, right now. I'd planned to be on vacation last week and 
really need some time in a yurt far away from anything electronic.

I am looking forward very much towards published results, analysis, and 
benchmarks from everyone else. Certainly I intend to get to where I can 
do CDF plots of what I reference below over large
sample periods, etc, at some point... but for your low latency pleasure:

50 netperf -l 120 -t TCP_STREAM, then 1 netperf -l 30 -t TCP_RR, ecn on, 
100Mbit ethernet, 3 hops
1 machine has codel, the other does not (so many kernels, so little time)

I note that QFQ is WAY more cpu/memory overhead and I haven't captured 
or analysed the streams.

                                                  TCP_RR    PING

PFIFO_FAST on both, no load:      2440.42   .26 to .59
PFIFO_FAST to PFIFO_FAST:              8.13     121ms
CODEL to PFIFO_FAST:                    27.56     Initial impulse of 
117ms, settles down to about 25ms
FQ_CODEL to PFIFO_FAST:             272.76    1.2ms, no spike
QFQ+CODEL to PFIFO_FAST:        1002.39     .7ish, no spike
QFQ+FQ_CODEL to PFIFO_FAST  1020.62     .95ms no spike (we are down at 
statistical noise levels)

I haven't tried codel to codel or fq_codel to fq_codel much yet.

I note also that I get best results when also applying fq_codel as an 
ingress qdisc. With that
on it's hard to get any latencies in excess of 5ms in any direction. 
Without it, traffic in the incoming direction (via TCP_MAERTS on 
netperf) cracks 30ms worth of delay and 110 TCP_RR vs pfifo_fast.

   in debloat is script that does that using a three tier pfifo_fast 
superset. Using ifb is rocket science, as is tc, however. It would be 
very interesting to come up with a simpler way of using codel or 
fq_codel on ingress, perhaps with tbf.

I just checked in the latest revision of debloat and the simple_qos and ingres scripts to the deBloat repository
on github.


  >If the elephant flow is the 'high priority' traffic, dropping packets
>from it is not what you wanted.

>pfifo_fast is able to have strict priority.


Which can lead to starvation of all flows, in pfifo_fast's case. Two 
possible compromises are
using a 3 tier htb bucket to emulate pfifo_fasts behavior with the 
addition of codel for sane
packet drop, or to wedge some interesting behavior into the extra 15 
bits in the codel vars.


> If you want Codel and strict
> priority, you can use prio + codel  (or prio + fq_codel)

I'm not fond of strict prioritization.

I am unsure of how useful mq_prio is at this point.

>
>
> _______________________________________________
> Codel mailing list
> Codel at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/codel