[Bloat] Network test tools for many parallel/concurrent connections?

[Bloat] Network test tools for many parallel/concurrent connections?

[Jesper Dangaard Brouer]

(I'm testing fq_codel and codel)

I need a test tool that can start many TCP streams (>1024).
During/after the testrun I want to know if the connections got a fair
share of the bandwidth.

Can anyone recomment tools for this?

After the test I would also like to, "deep-dive" analyse one of the TCP
streams to see how the congestion window, outstanding-win/data is
behaving.  Back in 2005 I used-to-use a tool  called
"tcptrace" (http://www.tcptrace.org).
Have any better tools surfaced?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

I still use tcptrace and xplot.org for deep dives.

I just fired off 2048 netperfs to localhost on my laptop.  It started
bogging down at 1000 but made it to the end, all connections chugging away.

Probably a better tool would be the apache benchmark `ab` or something else
that is built to stress out web sites.
On May 14, 2013 10:15 AM, "Jesper Dangaard Brouer" <jbrouer@redhat.com>
wrote:

>
> (I'm testing fq_codel and codel)
>
> I need a test tool that can start many TCP streams (>1024).
> During/after the testrun I want to know if the connections got a fair
> share of the bandwidth.
>
> Can anyone recomment tools for this?
>
> After the test I would also like to, "deep-dive" analyse one of the TCP
> streams to see how the congestion window, outstanding-win/data is
> behaving.  Back in 2005 I used-to-use a tool  called
> "tcptrace" (http://www.tcptrace.org).
> Have any better tools surfaced?
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Sr. Network Kernel Developer at Red Hat
>   Author of http://www.iptv-analyzer.org
>   LinkedIn: http://www.linkedin.com/in/brouer
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 1966 bytes --]

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Stephen Hemminger]

On Tue, 14 May 2013 15:48:38 +0200
Jesper Dangaard Brouer <jbrouer@redhat.com> wrote:

> 
> (I'm testing fq_codel and codel)
> 
> I need a test tool that can start many TCP streams (>1024).
> During/after the testrun I want to know if the connections got a fair
> share of the bandwidth.
> 
> Can anyone recomment tools for this?
> 
> After the test I would also like to, "deep-dive" analyse one of the TCP
> streams to see how the congestion window, outstanding-win/data is
> behaving.  Back in 2005 I used-to-use a tool  called
> "tcptrace" (http://www.tcptrace.org).
> Have any better tools surfaced?
> 


You may want to look at some of the "realistic" load tools since
in real life not all flows are 100% of bandwidth and long lived.

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Isaac Konikoff]

Candela makes a Linux based network traffic test tool called 
LANforge-FIRE. It can generate in excess of 50,000 TCP connections among 
other features. It is a proprietary tool, but we'll send out free 
licenses to students and those working on non-commercial open source 
projects.

Thanks,
Isaac



On 05/14/2013 07:46 AM, Dave Taht wrote:
> I still use tcptrace and xplot.org <http://xplot.org> for deep dives.
>
> I just fired off 2048 netperfs to localhost on my laptop.  It started
> bogging down at 1000 but made it to the end, all connections chugging away.
>
> Probably a better tool would be the apache benchmark `ab` or something
> else that is built to stress out web sites.
>
> On May 14, 2013 10:15 AM, "Jesper Dangaard Brouer" <jbrouer@redhat.com
> <mailto:jbrouer@redhat.com>> wrote:
>
>
>     (I'm testing fq_codel and codel)
>
>     I need a test tool that can start many TCP streams (>1024).
>     During/after the testrun I want to know if the connections got a fair
>     share of the bandwidth.
>
>     Can anyone recomment tools for this?
>
>     After the test I would also like to, "deep-dive" analyse one of the TCP
>     streams to see how the congestion window, outstanding-win/data is
>     behaving.  Back in 2005 I used-to-use a tool  called
>     "tcptrace" (http://www.tcptrace.org).
>     Have any better tools surfaced?
>
>     --
>     Best regards,
>        Jesper Dangaard Brouer
>        MSc.CS, Sr. Network Kernel Developer at Red Hat
>        Author of http://www.iptv-analyzer.org
>        LinkedIn: http://www.linkedin.com/in/brouer
>     _______________________________________________
>     Bloat mailing list
>     Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>     https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>


-- 
Isaac Konikoff
Candela Technologies
konikofi@candelatech.com
Office: +1 360 380 1618
Cell: +1 360 389 2453
Fax: +1 360 380 1431

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

On May 14, 2013 12:21 PM, "Stephen Hemminger" <stephen@networkplumber.org>
wrote:
>
> On Tue, 14 May 2013 15:48:38 +0200
> Jesper Dangaard Brouer <jbrouer@redhat.com> wrote:
>
> >
> > (I'm testing fq_codel and codel)
> >
> > I need a test tool that can start many TCP streams (>1024).
> > During/after the testrun I want to know if the connections got a fair
> > share of the bandwidth.
> >
> > Can anyone recomment tools for this?
> >
> > After the test I would also like to, "deep-dive" analyse one of the TCP
> > streams to see how the congestion window, outstanding-win/data is
> > behaving.  Back in 2005 I used-to-use a tool  called
> > "tcptrace" (http://www.tcptrace.org).
> > Have any better tools surfaced?
> >
>
>
> You may want to look at some of the "realistic" load tools since
> in real life not all flows are 100% of bandwidth and long lived.

You may want to look at some realistic load tools since in real life
99.9Xx% of all flows are 100% of bandwidth AND long lived.

At various small timescales a flow or flows can be 100% of bandwidth.

But it still takes one full rate flow to mess up your whole day.

This is why I suggested ab.

Here bandwidth is an average usually taken over a second and often much
more. If you sample at a higher resolution, like a ms, you are either at
capacity or empty.

Another way of thinking about it is for example, mrtg takes samples every
30 seconds and the most detailed presentation of that data it gives you is
on a 5 minute interval. The biggest fq codel site I have almost never shows
a 5 minute average over 60% of capacity, but I know full well that Netflix
users are clobbering things on a 10 sec interval and that there are
frequent peaks where it is running at capacity for a few seconds at a time
from looking at the data on a much finer interval and the fq codel drop
statistics.

> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: Type: text/html, Size: 2698 bytes --]

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 4477 bytes --]

There are really three kinds of killer traffic here, and it's important to
understand the differences so as to best design testing:

   1) long lived flows that clobber you and ruin your whole day.

   2) "streaming" video traffic (e.g. netflix, youtube, hulu), that are
actually "chunking" data over TCP, and putting periodic latency into your
connection as they temporarily build some queue.

fq_codel can deal really, really well with both 1 and 2.  But the number of
flows is usually not very large.

   3) the DOS attacks of visiting a new sharded web page on your
broadband/wireless connection, where you get the multiplication of N
connections * TCP Initial Window size, sometime resulting in pulses of
order hundred packets in a ton of new flows.  I've measured transient
latency of order 100's of milliseconds on a 50Mbps cable system! These web
sites generate a bunch of flows effectively simultaneously, each with often
only a few packets so never even do slow start to speak of.

Exactly what damage is done given 3, using fq_codel's algorithm isn't
entirely clear to me.  Many/most images on such sharded web sites are quite
small, even less than one packet at times.

fq_codel is clearly radically better than nothing at handling 3, but I
suspect we still have work to do...  Spdy will help if/when fully deployed,
but the ability to game buffers remains, and will continue to provide
incentive to anti-social applications to mis-behave.  We're really far from
done, but as Matt Mathis notes, what we have now in fq_codel is soooo,
sooooo much better than the current disaster, we shouldn't wait to deploy
something 'better' while working out problems like that.

I've thought for a while that exactly how we want to define a "flow" may
depend on where we are in the network: what's appropriate for an ISP is
different than what we do in the home, for example.

How best to test for the problems these generate, at various points in the
network, is still a somewhat open question.  And ensuring everything works
well at scale is extremely important.

I'm glad Jesper is doing scaling tests!
                                        - Jim



On Tue, May 14, 2013 at 1:01 PM, Dave Taht <dave.taht@gmail.com> wrote:

>
> On May 14, 2013 12:21 PM, "Stephen Hemminger" <stephen@networkplumber.org>
> wrote:
> >
> > On Tue, 14 May 2013 15:48:38 +0200
> > Jesper Dangaard Brouer <jbrouer@redhat.com> wrote:
> >
> > >
> > > (I'm testing fq_codel and codel)
> > >
> > > I need a test tool that can start many TCP streams (>1024).
> > > During/after the testrun I want to know if the connections got a fair
> > > share of the bandwidth.
> > >
> > > Can anyone recomment tools for this?
> > >
> > > After the test I would also like to, "deep-dive" analyse one of the TCP
> > > streams to see how the congestion window, outstanding-win/data is
> > > behaving.  Back in 2005 I used-to-use a tool  called
> > > "tcptrace" (http://www.tcptrace.org).
> > > Have any better tools surfaced?
> > >
> >
> >
> > You may want to look at some of the "realistic" load tools since
> > in real life not all flows are 100% of bandwidth and long lived.
>
> You may want to look at some realistic load tools since in real life
> 99.9Xx% of all flows are 100% of bandwidth AND long lived.
>
> At various small timescales a flow or flows can be 100% of bandwidth.
>
> But it still takes one full rate flow to mess up your whole day.
>
> This is why I suggested ab.
>
> Here bandwidth is an average usually taken over a second and often much
> more. If you sample at a higher resolution, like a ms, you are either at
> capacity or empty.
>
> Another way of thinking about it is for example, mrtg takes samples every
> 30 seconds and the most detailed presentation of that data it gives you is
> on a 5 minute interval. The biggest fq codel site I have almost never shows
> a 5 minute average over 60% of capacity, but I know full well that Netflix
> users are clobbering things on a 10 sec interval and that there are
> frequent peaks where it is running at capacity for a few seconds at a time
> from looking at the data on a much finer interval and the fq codel drop
> statistics.
>
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
> 

[-- Attachment #2: Type: text/html, Size: 7042 bytes --]

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Jesper Dangaard Brouer]

On Tue, 14 May 2013 08:47:26 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> You may want to look at some of the "realistic" load tools since
> in real life not all flows are 100% of bandwidth and long lived.

Well perhaps it would be easier to use some apache load test tool, but
in this test scenario I just want see how well fq_codel scales with
many concurrent bulk transfers.

Can people on the list recommend any commercial test-equip solutions?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] [Codel] Network test tools for many parallel/concurrent connections?

[Jesper Dangaard Brouer]

On Tue, 14 May 2013 07:46:02 -0700
Dave Taht <dave.taht@gmail.com> wrote:

> I still use tcptrace and xplot.org for deep dives.

Okay, I just hoped something new popped up.

Didn't we add some tcp trace system to the kernel?


> I just fired off 2048 netperfs to localhost on my laptop.  It started
> bogging down at 1000 but made it to the end, all connections chugging
> away.

So you just did a for loop with netperf's I assume.
If so, remember to adjust the "ulimit" of the shell.
Run 'ulimit -a' and you most likely need to adjust "ulimit -u".

I also played with "iperf --parallel" but iperf is not consistently
starting all the TCP connections.


> Probably a better tool would be the apache benchmark `ab` or
> something else that is built to stress out web sites.

Yes, I guess that might be the easiest "quick" solution.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Jesper Dangaard Brouer]

On Tue, 14 May 2013 09:28:54 -0700
Isaac Konikoff <konikofi@candelatech.com> wrote:

> Candela makes a Linux based network traffic test tool called 
> LANforge-FIRE. It can generate in excess of 50,000 TCP connections
> among other features. It is a proprietary tool, but we'll send out
> free licenses to students and those working on non-commercial open
> source projects.

I guess Red Hat does not qualify as a non-commercial open source
project ;-)

This actually sounds interesting.  I'll try an talk to my manager to
find some budget for this purpose (lets pick this up offlist).

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] [Codel] Network test tools for many parallel/concurrent connections?

[Rick Jones]

It will not match what one can get from tcptrace, or commercial 
solutions, but netperf can be asked to emit a number of potentially 
"intersting" things.  Using the "omni output selectors" one can request 
statistics for some interesting latencies:

raj@tardy:~$ netperf -- -O ? | grep LAT
RT_LATENCY
MIN_LATENCY
MAX_LATENCY
P50_LATENCY
P90_LATENCY
P99_LATENCY
MEAN_LATENCY
STDDEV_LATENCY

For a STREAM test those will be based on time in the send call.  For a 
MAERTS test those will be time in the receive call.  For an RR test 
those will be the round-trip times at the application layer.

You can also ./configure --enable-histogram and if the verbosity is set 
to 2 or more, a histogram of the distribution will be emitted which will 
resemble:

Histogram of time spent in send() call.
UNIT_USEC     :    0:    0:  434: 404912: 715323: 800663: 263305: 9336: 
2439: 1522
TEN_USEC      :    0: 2276:   41:   48:   97:   67:   79:   17:    5:    7
HUNDRED_USEC  :    0:   28:    2:    2:    0:    2:    0:    0:    1:    1
UNIT_MSEC     :    0:    3:    2:    0:    1:    0:    1:    0:    0:    0
TEN_MSEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
HUNDRED_MSEC  :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
UNIT_SEC      :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
TEN_SEC       :    0:    0:    0:    0:    0:    0:    0:    0:    0:    0
 >100_SECS: 0
HIST_TOTAL:      2200614

when running under Linux, netperf also knows how to report the number of 
TCP retransmissions encountered over the life of the data connection:

raj@tardy:~$ netperf -- -O ? | grep -i retran
LOCAL_TRANSPORT_RETRANS
REMOTE_TRANSPORT_RETRANS

And if you want to have an idea of what each individual netperf was 
doing in terms of mbit/s or trans/s over discrete points in its 
lifetime, you can ./configure --enable-demo and it will emit interim 
results at roughly the requested interval which can then be 
post-processed.  An example of that being done can be found in 
doc/examples/runemomniaggdemo.sh script and doc/examples/post_proc.py

happy benchmarking,

rick jones

Re: [Bloat] [Codel] Network test tools for many parallel/concurrent connections?

[Jesper Dangaard Brouer]

Hi Rick,

Thanks for your input :-)

I will definitely look into all these advanced options that netperf
provide (which is didn't know of).  Netperf is definitely my favorite
benchmarking tool, but I don't think it supports concurrent connections?

(Perhaps a stupid question:) I'm curr using netperf 2.x, any reason I
should switch to netperf 3.x ? 

Thanks you for developing netperf, 
--Jesper


On Tue, 14 May 2013 15:26:22 -0700
Rick Jones <rick.jones2@hp.com> wrote:

> It will not match what one can get from tcptrace, or commercial 
> solutions, but netperf can be asked to emit a number of potentially 
> "intersting" things.  Using the "omni output selectors" one can
> request statistics for some interesting latencies:
> 
> raj@tardy:~$ netperf -- -O ? | grep LAT
> RT_LATENCY
> MIN_LATENCY
> MAX_LATENCY
> P50_LATENCY
> P90_LATENCY
> P99_LATENCY
> MEAN_LATENCY
> STDDEV_LATENCY
> 
> For a STREAM test those will be based on time in the send call.  For
> a MAERTS test those will be time in the receive call.  For an RR test 
> those will be the round-trip times at the application layer.
> 
> You can also ./configure --enable-histogram and if the verbosity is
> set to 2 or more, a histogram of the distribution will be emitted
> which will resemble:
> 
> Histogram of time spent in send() call.
> UNIT_USEC     :    0:    0:  434: 404912: 715323: 800663: 263305:
> 9336: 2439: 1522
> TEN_USEC      :    0: 2276:   41:   48:   97:   67:   79:   17:
> 5:    7 HUNDRED_USEC  :    0:   28:    2:    2:    0:    2:    0:
> 0:    1:    1 UNIT_MSEC     :    0:    3:    2:    0:    1:    0:
> 1:    0:    0:    0 TEN_MSEC      :    0:    0:    0:    0:    0:
> 0:    0:    0:    0:    0 HUNDRED_MSEC  :    0:    0:    0:    0:
> 0:    0:    0:    0:    0:    0 UNIT_SEC      :    0:    0:    0:
> 0:    0:    0:    0:    0:    0:    0 TEN_SEC       :    0:    0:
> 0:    0:    0:    0:    0:    0:    0:    0
>  >100_SECS: 0
> HIST_TOTAL:      2200614
> 
> when running under Linux, netperf also knows how to report the number
> of TCP retransmissions encountered over the life of the data
> connection:
> 
> raj@tardy:~$ netperf -- -O ? | grep -i retran
> LOCAL_TRANSPORT_RETRANS
> REMOTE_TRANSPORT_RETRANS
> 
> And if you want to have an idea of what each individual netperf was 
> doing in terms of mbit/s or trans/s over discrete points in its 
> lifetime, you can ./configure --enable-demo and it will emit interim 
> results at roughly the requested interval which can then be 
> post-processed.  An example of that being done can be found in 
> doc/examples/runemomniaggdemo.sh script and doc/examples/post_proc.py
> 
> happy benchmarking,
> 
> rick jones



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] [Codel] Network test tools for many parallel/concurrent connections?

[Rick Jones]

On 05/15/2013 03:27 AM, Jesper Dangaard Brouer wrote:
> Hi Rick,
>
> Thanks for your input :-)
>
> I will definitely look into all these advanced options that netperf
> provide (which is didn't know of).  Netperf is definitely my favorite
> benchmarking tool, but I don't think it supports concurrent connections?

Not explicitly, no.  It is left as a scripting exercise to the 
benchmarker :)  The manual describes a few mechanisms for 
"synchronization" of tests to mitigate issues of skew error.  My 
"favorite" at this point is to post-process demo-mode output (which 
presumes reasonably well-synchronized clocks).

> (Perhaps a stupid question:) I'm curr using netperf 2.x, any reason I
> should switch to netperf 3.x ?

Netperf 3.x was something of an experimental dead-end.  It did have the 
idea of launching multiple threads in the netperf side, but retained the 
process-per-connection model on the netserver side.  I would suggest 
sticking with net[perf 2.x.

happy benchmarking,

rick

>
> Thanks you for developing netperf,

And to the *many* people who have contributed to it over the years.


> --Jesper
>
>
> On Tue, 14 May 2013 15:26:22 -0700
> Rick Jones <rick.jones2@hp.com> wrote:
>
>> It will not match what one can get from tcptrace, or commercial
>> solutions, but netperf can be asked to emit a number of potentially
>> "intersting" things.  Using the "omni output selectors" one can
>> request statistics for some interesting latencies:
>>
>> raj@tardy:~$ netperf -- -O ? | grep LAT
>> RT_LATENCY
>> MIN_LATENCY
>> MAX_LATENCY
>> P50_LATENCY
>> P90_LATENCY
>> P99_LATENCY
>> MEAN_LATENCY
>> STDDEV_LATENCY
>>
>> For a STREAM test those will be based on time in the send call.  For
>> a MAERTS test those will be time in the receive call.  For an RR test
>> those will be the round-trip times at the application layer.
>>
>> You can also ./configure --enable-histogram and if the verbosity is
>> set to 2 or more, a histogram of the distribution will be emitted
>> which will resemble:
>>
>> Histogram of time spent in send() call.
>> UNIT_USEC     :    0:    0:  434: 404912: 715323: 800663: 263305:
>> 9336: 2439: 1522
>> TEN_USEC      :    0: 2276:   41:   48:   97:   67:   79:   17:
>> 5:    7 HUNDRED_USEC  :    0:   28:    2:    2:    0:    2:    0:
>> 0:    1:    1 UNIT_MSEC     :    0:    3:    2:    0:    1:    0:
>> 1:    0:    0:    0 TEN_MSEC      :    0:    0:    0:    0:    0:
>> 0:    0:    0:    0:    0 HUNDRED_MSEC  :    0:    0:    0:    0:
>> 0:    0:    0:    0:    0:    0 UNIT_SEC      :    0:    0:    0:
>> 0:    0:    0:    0:    0:    0:    0 TEN_SEC       :    0:    0:
>> 0:    0:    0:    0:    0:    0:    0:    0
>>   >100_SECS: 0
>> HIST_TOTAL:      2200614
>>
>> when running under Linux, netperf also knows how to report the number
>> of TCP retransmissions encountered over the life of the data
>> connection:
>>
>> raj@tardy:~$ netperf -- -O ? | grep -i retran
>> LOCAL_TRANSPORT_RETRANS
>> REMOTE_TRANSPORT_RETRANS
>>
>> And if you want to have an idea of what each individual netperf was
>> doing in terms of mbit/s or trans/s over discrete points in its
>> lifetime, you can ./configure --enable-demo and it will emit interim
>> results at roughly the requested interval which can then be
>> post-processed.  An example of that being done can be found in
>> doc/examples/runemomniaggdemo.sh script and doc/examples/post_proc.py
>>
>> happy benchmarking,
>>
>> rick jones
>
>
>