[Bloat-ietf] [aqm] [FQ_CoDel] "Official" version

[Dave Taht]

On Mon, Feb 3, 2014 at 7:57 AM, Jim Gettys <jg at freedesktop.org> wrote:
>
>
> On Mon, Feb 3, 2014 at 9:30 AM, Nicolas KUHN
> <nicolas.kuhn at telecom-bretagne.eu> wrote:
>>
>> Dear all,
>>
>> We would like to test FQ_CoDel and we wonder which version to evaluate.
>
>
> fq_codel has been in Linux since about Linux 3.5, IIRC.  For casual
> experiments, a current Linux distro will have it available at the invocation
> of the "tc" command.  But...

Improvements as of 3.12 is the ability to set a default qdisc across the
entire system via a single sysctl.conf option.

> You should generally be testing the latest Linux release (not that I think
> there has been much change in fq_codel, as demonstrated in:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/?id=refs%2Ftags%2Fv3.14-rc1&qt=grep&q=fq_codel

However there were some improvements to the jhash algorithm that
landed in 3.8 and later to improve ipv6 hashing, and to more deeply
inspect gre and 6in4 tunnels.

> For serious bench marking getting results from an old system can make your
> results easily invalid since there has been a large number of improvements
> in Linux for bufferbloat everywhere over the last three years, and the queue
> discipline is only part of the equation.  For example, many ethernet drivers
> in SOC's found in home routers may still lack even BQL support.

I note the sanest results are more consistently obtained using a SQM
system like what's in cero, which leverages htb, thus moving control
to htb and away from the drivers deeper in the stack.

that said, htb also underwent improvements in 3.11 to make it faster, and
there are some "dis-improvements" the current SQM system for cero that
speed it up for the low powered hardware it's aimed at, above 20Mbit.
Faster systems don't need those hacks...

(and htb can be improved further, or replaced with hfsc.)

> You should be testing the fq_codel queue discipline: while Dave Taht has
> played with a number of variants on that theme, nothing to date has shown
> itself (by testing) with enough better results to be worth merging upstream
> into kernel.org.

nfq_codel presently does mildly better at very low bandwidths (<2Mbit)
as it is more sfq-like. Actually everything below 4Mbit is kind of
finicy with everything (aqm, sfq, codel, fq_codel).

I have been thinking of bringing back a patch from 3.5 in some form to
make maxpacket handling saner at lower bandwidths (at a price in latency)

> Pie finally entered kernel.org Linux a month or so ago.

I have been backporting that as far back as 3.8.


>
> Note that there a large number of "traps" for the unwary in this
> benchmarking/testing business, into which multiple groups have come to bad
> ends causing questionable results to confuse everyone. For example, the
> details of the underlying device driver can matter a lot (actually,
> hugely!), and netem has to be used with exquisite care just to name a couple
> of the big traps.

netem was hugely buggy prior to 3.11. I still only run it on a dedicated box
to insert delay and watch it carefully.

> Fundamentally, you need to realize that the queue discipline in Linux (where
> you can apply aqm) does *not* have control of all of the buffering in the
> system; device drivers and even the hardware itself can do "interesting"
> things to you.  And a bunch of changes have taken place in the Linux TCP
> stack to remove unneeded buffering since around Linux 3.2.  At one point I
> think the list was up to a dozen or more significant improvements  in the
> Linux networking stack, independent of fq_codel.

Yep.

I look forward to hardware that can run this stuff directly.

> We've come to learn that WiFi in particular needs a serious rewrite, and
> before venturing far, you'd be wise to get in touch with us further.  ATM,
> the ath9k driver is least problematic, but even there, aggregation causes
> large amounts of buffering (32 packets!).
>
> Dave Taht and I tried to outline what you should be thinking about when
> bench marking in:
>
> http://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

Don't be too scared...

> Pretty much all of these issues apply to anything you do, not just fq_codel.
>
> I think that page overs most of the problems  though of course, Linux is a
> moving target so I'm sure there will be future problems :-(.
>
>
>>
>> Does anyone know where we could get the "official" version / source code
>> of FQ_CoDel ?
>> (linux and/or NS2 if possible)
>
>
> www.kernel.org for the source.

patches and backports to older kernel versions are available as part
of the kernel-backports package.

(YMMV, as we note lots of other changes in the stack elsewhere
 matter quite a lot)

>
> You can use git to see the exact revision history of the code.
>
> I forget off hand where to find the simulation code...  There is a link
> somewhere in the bufferbloat/codel wiki
>
> http://www.bufferbloat.net/projects/codel/wiki/Wiki

ns2 and ns3 versions of the patches have been rotting in a repo on github.
Hope someone makes it a google summer of code project to get these mainlined.

> Help from people to get the wiki up to date gratefully appreciated!
>
> Jim
>
>
>>
>> Thanks a lot,
>>
>> Kind regards,
>>
>> Nicolas
>>
>> _______________________________________________
>> aqm mailing list
>> aqm at ietf.org
>> https://www.ietf.org/mailman/listinfo/aqm
>
>
>
> _______________________________________________
> aqm mailing list
> aqm at ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html