[Codel] iperf3 udp flood behavior at higher rates

[Codel] iperf3 udp flood behavior at higher rates

[Dave Taht]

to fork the fq_codel_drop discussion a bit...

I have up and running two new boxes[1] that are my hope to be able to
test ath10k/ath9k hardware with, for this test, using one in the
middle as a router and a nuc i3 box as the server, all ports pure
ethernet... there's a switch in the way, too.

On tcp via netperf I get expected ~940 mbits.

On udp via iperf3 (again, all pure ethernet) -  in neither case below
am I seeing any drops in the qdisc itself anywhere on the path, yet am
only achieving 500mbit.

?

1) Using the

iperf3 -c 172.26.16.130 -u -b900M -R -l1472 -t600

udp flood version, I get some loss on the initial burst, but none
*reported* after that, and peak at about ~500Mbits.

[ ID] Interval           Transfer     Bandwidth       Jitter
Lost/Total Datagrams
[  4]   0.00-1.00   sec  52.1 MBytes   437 Mbits/sec  0.037 ms
1276/38379 (3.3%)
[  4]   1.00-2.00   sec  54.3 MBytes   456 Mbits/sec  0.042 ms  0/38699 (0%)
[  4]   2.00-3.00   sec  56.1 MBytes   470 Mbits/sec  0.030 ms  0/39933 (0%)

2) Flipping the sense of the test by getting rid of -R (from the nuc)

iperf3 -c 172.26.16.130 -u -b900M -l1472 -t600

I get on the other side a steady state throughput of a little over
520mbits (with 41% loss reported consistently)

[  5]  37.00-38.00  sec  64.2 MBytes   539 Mbits/sec  0.026 ms
31613/77355 (41%)
[  5]  38.00-39.00  sec  62.8 MBytes   527 Mbits/sec  0.023 ms
31517/76255 (41%)
[  5]  39.00-40.00  sec  62.0 MBytes   520 Mbits/sec  0.033 ms
31052/75201 (41%)

On the other:

[  4]  77.00-78.00  sec   111 MBytes   929 Mbits/sec  78915
[  4]  78.00-79.00  sec   103 MBytes   864 Mbits/sec  73371
[  4]  79.00-80.00  sec   108 MBytes   907 Mbits/sec  77034
[  4]  80.00-81.00  sec   107 MBytes   900 Mbits/sec  76423
[  4]  81.00-82.00  sec   104 MBytes   875 Mbits/sec  74277
[  4]  82.00-83.00  sec   113 MBytes   950 Mbits/sec  80666


Thinking that perhaps I was seeing loss in the rx ring, I used ethtool
to increase that from the default 256 to 4096...

only to hang things thoroughly... :( and I'm watching things reboot now.

Netperf does not have a multi-hop capable udp flood test (rick jones
can explain why... )

As I recall on this thread iperf3 was being run on a mac box as a
client, and I'll dig one up - but was it also osx on the other side of
the test?

And what other params would I tweak on linux to see a udp flood go faster?

Topology looks like this:

apu1 <-> apu2 <-> switch <-> nuc.

I could put another switch in the way, I am always nervous about
invoking hw flow control...

[1] http://www.pcengines.ch/apu2c4.htm

Re: [Codel] iperf3 udp flood behavior at higher rates

[Rick Jones]

On 05/02/2016 04:18 PM, Dave Taht wrote:
> Netperf does not have a multi-hop capable udp flood test (rick jones
> can explain why... )

Well, with an intro like that, how could I refuse?-)

In a nutshell, after repeated uses of the netperf UDP_STREAM test in 
non-air-gapped test setups with link up/down testing and a default route 
which pointed at the local "corporate/desktop/whathaveyou" network by QA 
engineers who should have known better, taking-out things like security 
camera video feeds, by default the data socket for a UDP_STREAM test is 
set SO_DONTROUTE.  This can be overridden with a test-specific -R 1 option.

This is not necessary if the test is UDP_RR or TCP_*.

happy benchmarking,

rick jones

Re: [Codel] iperf3 udp flood behavior at higher rates

[Dave Taht]

On Mon, May 2, 2016 at 4:27 PM, Rick Jones <rick.jones2@hpe.com> wrote:
> On 05/02/2016 04:18 PM, Dave Taht wrote:
>>
>> Netperf does not have a multi-hop capable udp flood test (rick jones
>> can explain why... )
>
>
> Well, with an intro like that, how could I refuse?-)
>
> In a nutshell, after repeated uses of the netperf UDP_STREAM test in
> non-air-gapped test setups with link up/down testing and a default route
> which pointed at the local "corporate/desktop/whathaveyou" network by QA
> engineers who should have known better, taking-out things like security
> camera video feeds, by default the data socket for a UDP_STREAM test is set
> SO_DONTROUTE.  This can be overridden with a test-specific -R 1 option.

That is so much more PC than what's actually in the release notes! :)
Well, a quick recompile later and applying -R 1 still gets rejected...
reading nettest_omni.c was quite entertaining, tho.

I actually like a world where it takes more effort to mess up the
network - where ping -f is not available to anyone but root, for
example. It's bothersome, in flent, to need root to get a ping
--step-size resolution of < 20ms, but I can live with it, and I'd like
a world where script kiddies with "ion cannons" had their hardware
explode on them when mis-used...

There are some things ordinary man does not need to know, and
knowledge of the netperf -t UDP_STREAM -R 1 option is one of them.

Except, maybe today, when I'm trying to figure out why iperf3 is being weird.

> This is not necessary if the test is UDP_RR or TCP_*.
>
> happy benchmarking,
>
> rick jones



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Codel] iperf3 udp flood behavior at higher rates

[Roman Yeryomin]

On 3 May 2016 at 02:18, Dave Taht <dave.taht@gmail.com> wrote:
> to fork the fq_codel_drop discussion a bit...
>
> I have up and running two new boxes[1] that are my hope to be able to
> test ath10k/ath9k hardware with, for this test, using one in the
> middle as a router and a nuc i3 box as the server, all ports pure
> ethernet... there's a switch in the way, too.
>
> On tcp via netperf I get expected ~940 mbits.
>
> On udp via iperf3 (again, all pure ethernet) -  in neither case below
> am I seeing any drops in the qdisc itself anywhere on the path, yet am
> only achieving 500mbit.

That's interesting, I have no problems with UDP over ethernet.
What about TCP with iperf3?

> ?
>
> 1) Using the
>
> iperf3 -c 172.26.16.130 -u -b900M -R -l1472 -t600
>
> udp flood version, I get some loss on the initial burst, but none
> *reported* after that, and peak at about ~500Mbits.
>
> [ ID] Interval           Transfer     Bandwidth       Jitter
> Lost/Total Datagrams
> [  4]   0.00-1.00   sec  52.1 MBytes   437 Mbits/sec  0.037 ms
> 1276/38379 (3.3%)
> [  4]   1.00-2.00   sec  54.3 MBytes   456 Mbits/sec  0.042 ms  0/38699 (0%)
> [  4]   2.00-3.00   sec  56.1 MBytes   470 Mbits/sec  0.030 ms  0/39933 (0%)
>
> 2) Flipping the sense of the test by getting rid of -R (from the nuc)
>
> iperf3 -c 172.26.16.130 -u -b900M -l1472 -t600
>
> I get on the other side a steady state throughput of a little over
> 520mbits (with 41% loss reported consistently)
>
> [  5]  37.00-38.00  sec  64.2 MBytes   539 Mbits/sec  0.026 ms
> 31613/77355 (41%)
> [  5]  38.00-39.00  sec  62.8 MBytes   527 Mbits/sec  0.023 ms
> 31517/76255 (41%)
> [  5]  39.00-40.00  sec  62.0 MBytes   520 Mbits/sec  0.033 ms
> 31052/75201 (41%)
>
> On the other:
>
> [  4]  77.00-78.00  sec   111 MBytes   929 Mbits/sec  78915
> [  4]  78.00-79.00  sec   103 MBytes   864 Mbits/sec  73371
> [  4]  79.00-80.00  sec   108 MBytes   907 Mbits/sec  77034
> [  4]  80.00-81.00  sec   107 MBytes   900 Mbits/sec  76423
> [  4]  81.00-82.00  sec   104 MBytes   875 Mbits/sec  74277
> [  4]  82.00-83.00  sec   113 MBytes   950 Mbits/sec  80666
>
>
> Thinking that perhaps I was seeing loss in the rx ring, I used ethtool
> to increase that from the default 256 to 4096...
>
> only to hang things thoroughly... :( and I'm watching things reboot now.
>
> Netperf does not have a multi-hop capable udp flood test (rick jones
> can explain why... )
>
> As I recall on this thread iperf3 was being run on a mac box as a
> client, and I'll dig one up - but was it also osx on the other side of
> the test?
>
> And what other params would I tweak on linux to see a udp flood go faster?

I would try making packets smaller (-l), maybe they are fragmented somewhere.

> Topology looks like this:
>
> apu1 <-> apu2 <-> switch <-> nuc.
>
> I could put another switch in the way, I am always nervous about
> invoking hw flow control...
>
> [1] http://www.pcengines.ch/apu2c4.htm

Re: [Codel] iperf3 udp flood behavior at higher rates

[Roman Yeryomin]

On 4 May 2016 at 11:02, Roman Yeryomin <leroi.lists@gmail.com> wrote:
> On 3 May 2016 at 02:18, Dave Taht <dave.taht@gmail.com> wrote:
>> to fork the fq_codel_drop discussion a bit...
>>
>> I have up and running two new boxes[1] that are my hope to be able to
>> test ath10k/ath9k hardware with, for this test, using one in the
>> middle as a router and a nuc i3 box as the server, all ports pure
>> ethernet... there's a switch in the way, too.
>>
>> On tcp via netperf I get expected ~940 mbits.
>>
>> On udp via iperf3 (again, all pure ethernet) -  in neither case below
>> am I seeing any drops in the qdisc itself anywhere on the path, yet am
>> only achieving 500mbit.
>
> That's interesting, I have no problems with UDP over ethernet.
> What about TCP with iperf3?
>

Also what version of iperf3 are you using?

Regards,
Roman