[Codel] better testing, linux 3.6.1, cerowrt credits, other stuff

[Codel] better testing, linux 3.6.1, cerowrt credits, other stuff

[Dave Taht]

Some notes:

Moving rapidly up on my own list of priorities is developing at least
a spec for better tests. Help?

Openwrt is nearing a stable release on linux 3.3.8, which includes a
backport of fq_codel. 150+ platforms, no waiting.

The current "sugarland" version of cerowrt contains versions of codel
and fq_codel based on the initial linux code and the current ns2 code.

Linux 3.6.1 + 1 patch is looking like a useful release for host based
benchmarking of fq_codel and related technologies on x86, arm, etc.

A 3.6 feature, TCP small queues, is a win - even with pfifo_fast,
particularly with bidirectional utilization.
Wireless-n remains a nightmare.

FQ_Codel testing on CPE and home gateways is expanding. We've
identified two different ADSL devices that can be suitably hacked on,
in particular. I hope we can collect a variety of good comparison data
on a variety of hardware in the coming month. I hope to expand the
bloatlab systems to allow more live testing, over the next two months.

If you (and/or your company) is supporting your efforts to beat the
bloat, please let me know soon so I can update the cerowrt credits
file before its next release, which will (hopefully) be based on the
final openwrt sources.

Current draft here:

http://cero2.bufferbloat.net/cerowrt/credits.html

We're coming up on the 2 year anniversary of jg accidentally leaking
his first bufferbloat blog posts onto the internet, and while we've
come a long way, much work remains.
Still, looking back, I'm pretty happy with where we are today.

-- 
Dave Täht

Please help beat bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

Re: [Codel] better testing, linux 3.6.1, cerowrt credits, other stuff

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

Dave Taht <dave.taht@gmail.com> writes:

> Moving rapidly up on my own list of priorities is developing at least
> a spec for better tests. Help?

This is probably not what you meant, but I thought I'd use this occasion
to post it anyway:

I am doing a project about bufferbloat at my university (I'm a master
student), which involves testing for bufferbloat and testing various
qdisc's performance in a controlled environment. I've been doing my
tests by running concurrent netperf instances (so far TCP_STREAM,
TCP_MAERTS and TCP_RR) and collecting the results. To make this a bit
easier I've written a small Python script to automate running the
various netperf instances and collecting the results. I thought I'd
share it here in case anyone else thinks it's useful. It's quite crude,
but works slightly better than a shell script for my purposes. :)

The code is available on github:
https://github.com/tohojo/netperf-wrapper

Oh, and in case you have any testing you'd like me to do, I'd be happy
to incorporate it. I can pretty much organise the project any way I
want, and it would be cool to do something that's useful for things
other than satisfying my own curiosity. ;)

Cheers,
-Toke

-- 
Toke Høiland-Jørgensen
toke@toke.dk

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [Codel] better testing, linux 3.6.1, cerowrt credits, other stuff

[Rick Jones]

Toke -

The script looks reasonable.  Certainly cleaner than any Python I've yet 
written :) I might be a little worried about skew error though (assuming 
I've not mis-read the script and example ini file).  That is why I use 
the "demo mode" of netperf in 
http://www.netperf.org/svn/netperf2/trunk/doc/examples/bloat.sh  though 
it does make the post-processing rather more involved.

I see you are running the TCP_RR test for less time than the 
TCP_STREAM/TCP_MAERTS test.  What do you then do to show the latency 
without the bulk transfer load?

You may find 
http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Omni-Output-Selection 
in 2.5.0 and 2.6.0 helpful when it comes to getting netperf to emit 
specific measurements.

I was thinking of trying to write a version of bloat.sh in python but 
before I did I wanted to know if python was sufficiently available in 
most folks bufferbloat testing environments.  I figure in 
"full-featured" *nix systems that isn't an issue, but what about in the 
routers?

happy benchmarking,

rick jones
here are some links concerning demo mode:
http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-_002d_002denable_002ddemo

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dD_002c-Global-22

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Dave Taht]

On Tue, Oct 9, 2012 at 9:45 AM, Rick Jones <rick.jones2@hp.com> wrote:
> Toke -
>
> The script looks reasonable.  Certainly cleaner than any Python I've yet
> written :) I might be a little worried about skew error though (assuming
> I've not mis-read the script and example ini file).  That is why I use the
> "demo mode" of netperf in
> http://www.netperf.org/svn/netperf2/trunk/doc/examples/bloat.sh  though it
> does make the post-processing rather more involved.

In my case I originally had a preference for writing wrappers in lua
(as this is the highest
level language available on the router itself) but rapidly ran into
snags with the overall lack of additional library support and my own
lack of lua expertise. Python is a much easier language to deal with.

That said, there is a start at lua wrappers as well as various
post-processing scripts for things like gnuplot in my deBloat repo on
github. The last time I got serious about getting something more
conventionally publishable is in
https://github.com/dtaht/deBloat/tree/master/test/results

Some stuff that drives gnuplot and does cdf:

https://github.com/dtaht/deBloat/tree/master/test/sfqs

My most current series of ad-hoc tests are at:
https://github.com/dtaht/deBloat/tree/master/test/multiple_codels
and I'm painfully aware of how ad-hoc they are, presently. There's
multiple flooding tests, as well as multiple setup
scripts for htb and hfsc, in there... my primary purpose in this test
series was to determine codel vs ns2_codel characteristics.

Initial results pointed at the ns2_codel derived versions managing a
queue depth better than the linux codel versions...

But: It became obvious fast that long RTT tests were needed, which
I've been trying to establish the infrastructure to do, and the big
flooding tests therein seems to point to a "horizontal standing queue"
issue that I wrote efq_codel to start to resolve, as well as issues
where the queue size is between 0 and 1 and low bandwidths. And that
TCP small queues mess up (improve) the behavior of netperf,
invalidating data not collected with that on....

(as well as tso/gso/ufo/gro really mucking with things, so I generally
turn them off with the debloat tool)

I'm tickled Toke is also dumping stuff into org-mode, which makes
publishing content much easier from emacs.

I will gladly start collecting tests and test results into that repo
if that helps, and grant commit privs. Or start over. There's
tons of cruft in there. Good commit logs, tho...

> I see you are running the TCP_RR test for less time than the
> TCP_STREAM/TCP_MAERTS test.  What do you then do to show the latency without
> the bulk transfer load?

We seem to be converging on this as a decent test, as this is similar
to tests that blogic and myself run.

In my case I had established baseline latency once using TCP_RR, and
didn't feel the need to re-establish the number on every test.

I also have a voip-like test that uses fping on a 10ms interval
instead of TCP_RR. This is useful in wireless especially, as that data
is crazy noisy. I find CDF plots very useful of data like that.

Coherently tracking certain other variables (such as the actual tc
qdisc line and sampling tc -s statistics across an interval) across
tests is also needed.

Another very useful test is TCP_STREAM + TCP_RR while running the
google chrome benchmarks.

Obviously standardization on easily repeatable tests is desperately needed!!

>
> You may find
> http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Omni-Output-Selection
> in 2.5.0 and 2.6.0 helpful when it comes to getting netperf to emit specific
> measurements.
>
> I was thinking of trying to write a version of bloat.sh in python but before
> I did I wanted to know if python was sufficiently available in most folks
> bufferbloat testing environments.  I figure in "full-featured" *nix systems
> that isn't an issue, but what about in the routers?
>
> happy benchmarking,
>
> rick jones
> here are some links concerning demo mode:
> http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Using-_002d_002denable_002ddemo
>
> http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#index-g_t_002dD_002c-Global-22
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Rick Jones]

On 10/09/2012 12:35 PM, Toke Høiland-Jørgensen wrote:
> Rick Jones <rick.jones2@hp.com> writes:
>
> Hi Rick
>
> Thanks for your feedback.
>
>> The script looks reasonable. Certainly cleaner than any Python I've
>> yet written :) I might be a little worried about skew error though
>> (assuming I've not mis-read the script and example ini file). That is
>> why I use the "demo mode" of netperf in
>> http://www.netperf.org/svn/netperf2/trunk/doc/examples/bloat.sh though
>> it does make the post-processing rather more involved.
>
> Ah, I see. I'll look into that. As far as I can tell from that script,
> you're basically running it with the demo mode enabled, and graphing the
> results with each line as a data point?

Mostly.  The smallest step size in rrdtool is one second, so I create 
rrd's with that as the step size, but I try for sub-second samples from 
the interval results to mitigate (or try to) some of rrdtools averaging.

> There's a comment about using negative values for -D to increase
> accuracy at a cost of performance. Is this cost significant? And if it
> is, would there be any reason why it wouldn't be possible to just use
> positive values and then fix the error by interpolating values to be at
> fixed intervals when graphing?

When the demo mode was introduced into netperf, a gettimeofday() call 
was still relatively expensive, and I wanted to mitigate the effect of 
demo mode on overall performance.  So, what the code does is guess how 
many units of work will complete within the desired output interval. 
Then, once that quantity of work has been completed, netperf makes the 
gettimeofday() call to see if it is time to emit an interim result, 
adjusting the guesstimate for units of work per time interval 
accordingly.  However, if things slow-down considerably, that can lead 
to a rather long interval between interim results.

So, now that at least some platforms have an "inexpensive" 
gettimeofday() call, if a negative value is used, that signals netperf 
to make the gettimeofday() call after each unit of work (each send or 
recv call, or pair in the case of an RR test).  That should result in 
hitting the desired interval more frequently, save for when a single one 
of those calls takes longer.

I used rrdtool so I could get all the tests "snapped" to the same set of 
one second intervals starting on one second boundaries.

>> I see you are running the TCP_RR test for less time than the
>> TCP_STREAM/TCP_MAERTS test. What do you then do to show the latency
>> without the bulk transfer load?
>
> I ran the TCP_RR test by itself to get a baseline result. The idea with
> the different lengths is to start the streams, wait two seconds, and
> then run the roundtrip so that it finished two seconds before the
> streams (i.e. the roundtrip test is only running while the streams are).
> This is for my test setup, which is just a few computers connected with
> ethernet; so no sudden roundtrip variances should occur. I can see how
> it would be useful to get something that can be graphed over time; I'll
> try to look into getting that working.

You might give the two bulk transfer tests a bit longer to get going - 
say 15 seconds or so.  In at least some of my runs of bloat.sh I've seen 
the throughput take a while to build-up.  That is perhaps part of the 
reason why Dave Taht is calling for long RTT tests?

>> I was thinking of trying to write a version of bloat.sh in python but
>> before I did I wanted to know if python was sufficiently available in
>> most folks bufferbloat testing environments. I figure in
>> "full-featured" *nix systems that isn't an issue, but what about in
>> the routers?
>
> Is there any reason why it wouldn't be possible to run the python script
> on a full-featured machine and run the netperf instances via an ssh
> tunnel on the router (on a different interface than the one being
> tested, of course)?

Not really, apart from script complexity and brittleness.  The 
netperf_by_flavor.py script does something along those lines for 
OpenStack Nova instances.  (One of only two python scripts I've written 
thusfar, the second being post_proc.py which is used by that to 
post-process results of its run of the runemomniaggdemo.sh script)

happy benchmarking,

rick

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Michael Richardson]

    >> But: It became obvious fast that long RTT tests were needed,
    >> which I've been trying to establish the infrastructure to do

    toke> I assume that by "infrastructure" you mean "(netperf) servers
    toke> far away"? What would be needed for a test server in terms of
    toke> resources (bandwidth and otherwise)? I could try and persuade
    toke> my university to let me setup a test server on their network
    toke> (which is in Denmark)...

I interpret the question to mean networks where is there significant
actual delay along them.  I seem to recall that there are some ways to
do this Linux machines, but most commercial test equipment can simulate
things, including dropping packets.
I think, however, that we do not want/need and packets dropped, as then
the bandwidth constraint would not be in the device under test.


-- 
Michael Richardson
-on the road-

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Stephen Hemminger]

On Wed, 10 Oct 2012 19:27:11 -0400
Michael Richardson <mcr@sandelman.ca> wrote:

> 
>     >> But: It became obvious fast that long RTT tests were needed,
>     >> which I've been trying to establish the infrastructure to do
> 
>     toke> I assume that by "infrastructure" you mean "(netperf) servers
>     toke> far away"? What would be needed for a test server in terms of
>     toke> resources (bandwidth and otherwise)? I could try and persuade
>     toke> my university to let me setup a test server on their network
>     toke> (which is in Denmark)...
> 
> I interpret the question to mean networks where is there significant
> actual delay along them.  I seem to recall that there are some ways to
> do this Linux machines, but most commercial test equipment can simulate
> things, including dropping packets.
> I think, however, that we do not want/need and packets dropped, as then
> the bandwidth constraint would not be in the device under test.

netem can do all the stuff commercial gear can.
In fact, it is used by one of the commercial products!

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Rick Jones]

On 10/09/2012 12:35 PM, Toke Høiland-Jørgensen wrote:
> I assume that by "infrastructure" you mean "(netperf) servers far away"? What
> would be needed for a test server in terms of resources (bandwidth and
> otherwise)? I could try and persuade my university to let me setup a
> test server on their network (which is in Denmark)...

Keep in mind that netperf needs two open ports - one for the control 
connection and one for the data connection.

As far as an access control mechanism goes, it is pretty lame, but the 
2.6.0 and later versions of netperf/netserver include the ability to 
define a passphrase the netserver must see from netperf before it will 
allow a test to be run.  It is the global -Z option.

http://www.netperf.org/svn/netperf2/tags/netperf-2.6.0/Release_Notes

happy benchmarking,

rick jones

Re: [Codel] [Bloat] better testing, linux 3.6.1, cerowrt credits, other stuff

[Luigi Rizzo]

On Wed, Oct 10, 2012 at 04:29:11PM -0700, Stephen Hemminger wrote:
> On Wed, 10 Oct 2012 19:27:11 -0400
> Michael Richardson <mcr@sandelman.ca> wrote:
> 
> > 
> >     >> But: It became obvious fast that long RTT tests were needed,
> >     >> which I've been trying to establish the infrastructure to do
> > 
> >     toke> I assume that by "infrastructure" you mean "(netperf) servers
> >     toke> far away"? What would be needed for a test server in terms of
> >     toke> resources (bandwidth and otherwise)? I could try and persuade
> >     toke> my university to let me setup a test server on their network
> >     toke> (which is in Denmark)...
> > 
> > I interpret the question to mean networks where is there significant
> > actual delay along them.  I seem to recall that there are some ways to
> > do this Linux machines, but most commercial test equipment can simulate
> > things, including dropping packets.
> > I think, however, that we do not want/need and packets dropped, as then
> > the bandwidth constraint would not be in the device under test.
> 
> netem can do all the stuff commercial gear can.
> In fact, it is used by one of the commercial products!

similarly you can use ipfw+dummynet, it has been running on
Linux and a variety of other OS for a long time, see

http://info.iet.unipi.it/~luigi/dummynet/

	cheers
	luigi