[Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Bob McMahon]

[-- Attachment #1: Type: text/plain, Size: 8061 bytes --]

Just an FYI in case anybody has interest in traffic tooling.

Bob

---------- Forwarded message ---------
From: Bob McMahon <bob.mcmahon@broadcom.com>
Date: Wed, Jul 17, 2019 at 1:39 PM
Subject: iperf 2 "short term" road map
To: iPerf User Group <iperf-users@lists.sourceforge.net>
Cc: Panos Vouzis <panos@netbeez.net>


Hi All,

I've been doing a lot of work on iperf 2 to handle new testing features
with a focus on TCP latency testing.  Please do let me know your thoughts
or suggest comments per the list below.  No promises on time frame as this
is not my primary day job ;)

The new features currently in development to include:

o) Reverse testing (invert the client/server roles after the connect) -
requested by many for testing through NAT gateways
o) Latency tail analysis support (latency mostly needs to evaluated from a
tail's perspective, provide those metrics w/o losing full distribution or,
at least, tail information)
o) Support GPS (atomic clock) timestamps with all histogram metrics
supporting post analysis of problem areas per distributed tooling (this
assumes all tooling are synce'd to the GPS atomic clocks.)
o) TCP bidir testing support (bidirectional or full duplex traffic on the
same socket)
o) TCP write-acks - the server will ack it's reads back (w/timestamp and
byte no.) to the client per every client write(), useful for TCP streaming
class of tests and doesn't require syncs to GPS clocks
o) TCP trip times, the write() to read() latency of every clients' writes -
requires GPS clock sync - this is a one way metric
o) TCP connect only tests (run many parallel TCP connects and measure the
connect times of each) - an issue for many mobile users is that TCP connect
times are impacting user experience
o) -P support for more than 127 and now is only limited by the kernel/os
(this already fixed)
o) CPU bound warning on exit - present a warning message when an iperf test
seemed to be CPU bound vs network i/o bound
o) Mutex warn message on exit: present a warning on exit when the test
spent time on mutex locks where that may be degrading network performance
o) Teporter thread consumption rate detector to add thread delay/suspends
only when needed (currently, the code always add reporter thread delay.)
This delay can and should reduce the cores cache invalidates per cache
coherency and the use of shared memory (and its pointers) not under os
mutex protection
o) Much better debugging support per ./configure --enable-thread-debug
o) Code clean up including using function pointers to replace per pacet,
inline runtime tests (prepping to complete the C++ implementation vs
today's hybrid of C and C++)
o) Support for raw sockets to test "L2 level" performance
o) Better code encapsulation (which includes freeing of memory)

Note:  Here's an example debug that's now supported for those who want to
delve into iperf internals.

Client:
[root@localhost iperf2-code]# src/iperf -c 192.168.1.1 -i 1 -t 4
THREAD(18576):[16:28:11.702942] Thread settings copy (malloc)
from/to=0x67ac20/0x67b640
THREAD(18576):[16:28:11.703128] Thread_run_wrapper(0x67ac20 mode=2) thread
counts tot/trfc=1/1
THREAD(18576):[16:28:11.703293] Thread_run_wrapper(0x67b640 mode=4) thread
counts tot/trfc=2/1
THREAD(18577):[16:28:11.703305] Client thread started in constructor
THREAD(18578):[16:28:11.704364] Reporter thread started
THREAD(18577):[16:28:11.708777] Init settings report 0x7f2a38020fb0
THREAD(18577):[16:28:11.708839] Update connection report 0x7f2a38020fb0
winreq=0 actual=87040
THREAD(18577):[16:28:11.708853] Post report 0x7f2a38020fb0
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
THREAD(18577):[16:28:11.708895] Init 5000 element packet ring 0x7f2a38023ee0
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
THREAD(18577):[16:28:11.708961] Init data report 0x7f2a380239a0 size 1336
using packetring 0x7f2a38023ee0
THREAD(18578):[16:28:11.709000] Remove 0x7f2a38020fb0 from reporter job
queue in rs
THREAD(18577):[16:28:11.709013] Init connection report 0x7f2a380239a0
THREAD(18578):[16:28:11.709037] Free 0x7f2a38020fb0 in rs
THREAD(18577):[16:28:11.709053] Update connection report 0x7f2a380239a0
winreq=0 actual=87040
THREAD(18577):[16:28:11.709080] Post report 0x7f2a380239a0
[  3] local 192.168.1.4 port 36276 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  7.75 MBytes  65.0 Mbits/sec
[  3]  1.0- 2.0 sec  7.75 MBytes  65.0 Mbits/sec
[  3]  2.0- 3.0 sec  6.50 MBytes  54.5 Mbits/sec
[  3]  3.0- 4.0 sec  7.12 MBytes  59.8 Mbits/sec
[  3]  0.0- 4.0 sec  29.1 MBytes  60.5 Mbits/sec
THREAD(18578):[16:28:15.747044] Remove 0x7f2a380239a0 from reporter job
queue in rs
THREAD(18577):[16:28:15.747045] Traffic thread thinks reporter is done with
0x7f2a380239a0
THREAD(18577):[16:28:15.747098] Client destructor close sock=3
THREAD(18577):[16:28:15.747126] Free packet ring 0x7f2a38023ee0 & condition
variable await consumer 0x7f2a38023ef8
THREAD(18577):[16:28:15.747156] Free report hdr 0x7f2a380239a0 delay
counter=994
THREAD(18577):[16:28:15.747168] Thread settings free=0x67ac20
THREAD(18578):[16:28:16.000164] Thread settings free=0x67b640

Server:

[root@localhost iperf2-code]# src/iperf -s -i 1
THREAD(19735):[16:27:43.620063] Thread settings copy (malloc)
from/to=0x12dac20/0x12daf10
THREAD(19735):[16:27:43.620250] Thread_run_wrapper(0x12dac20 mode=8) thread
counts tot/trfc=1/0
THREAD(19735):[16:27:43.620338] Thread_run_wrapper(0x12daf10 mode=3) thread
counts tot/trfc=2/0
THREAD(19736):[16:27:43.620505] Init settings report 0x7f6c84020930
THREAD(19736):[16:27:43.620552] Update connection report 0x7f6c84020930
winreq=0 actual=87380
THREAD(19736):[16:27:43.620578] Post report 0x7f6c84020930
THREAD(19736):[16:27:43.620619] Thread settings copy (malloc)
from/to=0x12dac20/0x7f6c84023410
THREAD(19736):[16:27:43.620636] Listener thread accepting for TCP (sock=3)
THREAD(19737):[16:27:43.620677] Reporter thread started
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
THREAD(19737):[16:27:43.620780] Remove 0x7f6c84020930 from reporter job
queue in rs
THREAD(19737):[16:27:43.620799] Free 0x7f6c84020930 in rs
THREAD(19736):[16:28:11.711590] Thread_run_wrapper(0x7f6c84023410 mode=1)
thread counts tot/trfc=3/1
THREAD(19736):[16:28:11.711658] Thread settings copy (malloc)
from/to=0x12dac20/0x7f6c84020b90
THREAD(19739):[16:28:11.711715] Server thread started (sock=4)
THREAD(19736):[16:28:11.711844] Listener thread accepting for TCP (sock=3)
THREAD(19739):[16:28:11.712438] Init 5000 element packet ring 0x7f6c80020f80
THREAD(19739):[16:28:11.712470] Init data report 0x7f6c80020a40 size 1336
using packetring 0x7f6c80020f80
THREAD(19739):[16:28:11.712496] Init connection report 0x7f6c80020a40
THREAD(19739):[16:28:11.712519] Update connection report 0x7f6c80020a40
winreq=0 actual=374400
THREAD(19739):[16:28:11.712535] Post report 0x7f6c80020a40
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.4 port 36276
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  6.70 MBytes  56.2 Mbits/sec
[  4]  1.0- 2.0 sec  7.43 MBytes  62.3 Mbits/sec
[  4]  2.0- 3.0 sec  6.91 MBytes  58.0 Mbits/sec
[  4]  3.0- 4.0 sec  6.89 MBytes  57.8 Mbits/sec
[  4]  0.0- 4.2 sec  29.1 MBytes  58.7 Mbits/sec
THREAD(19737):[16:28:15.878976] Remove 0x7f6c80020a40 from reporter job
queue in rs
THREAD(19739):[16:28:16.000140] Traffic thread thinks reporter is done with
0x7f6c80020a40
THREAD(19739):[16:28:16.000208] Server destructor close sock=4 drop-sock=-1
THREAD(19739):[16:28:16.000387] Free packet ring 0x7f6c80020f80 & condition
variable await consumer 0x7f6c80020f98
THREAD(19739):[16:28:16.000487] Free report hdr 0x7f6c80020a40 delay
counter=1030
THREAD(19739):[16:28:16.000529] Thread settings free=0x7f6c84023410
THREAD(19737):[16:29:07.000175] Thread settings free=0x12daf10

Thanks,
Bob

[-- Attachment #2: Type: text/html, Size: 8951 bytes --]

Re: [Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Dave Taht]

I don't feel like registering for yet another mailing list.

Could you encourage them to take a look at what we do with
flent, and irtt?

Bob McMahon <bob.mcmahon@broadcom.com> writes:

> Just an FYI in case anybody has interest in traffic tooling.
>
> Bob
>
> ---------- Forwarded message ---------
> From: Bob McMahon <bob.mcmahon@broadcom.com>
> Date: Wed, Jul 17, 2019 at 1:39 PM
> Subject: iperf 2 "short term" road map
> To: iPerf User Group <iperf-users@lists.sourceforge.net>
> Cc: Panos Vouzis <panos@netbeez.net>
>
> Hi All,
>
> I've been doing a lot of work on iperf 2 to handle new testing
> features with a focus on TCP latency testing. Please do let me know
> your thoughts or suggest comments per the list below. No promises on
> time frame as this is not my primary day job ;)
>
> The new features currently in development to include:
>
> o) Reverse testing (invert the client/server roles after the connect) -
> requested by many for testing through NAT gateways
> o) Latency tail analysis support (latency mostly needs to evaluated
> from a tail's perspective, provide those metrics w/o losing full
> distribution or, at least, tail information)
> o) Support GPS (atomic clock) timestamps with all histogram metrics
> supporting post analysis of problem areas per distributed tooling
> (this assumes all tooling are synce'd to the GPS atomic clocks.)
> o) TCP bidir testing support (bidirectional or full duplex traffic on
> the same socket)
> o) TCP write-acks - the server will ack it's reads back (w/timestamp
> and byte no.) to the client per every client write(), useful for TCP
> streaming class of tests and doesn't require syncs to GPS clocks
> o) TCP trip times, the write() to read() latency of every clients'
> writes - requires GPS clock sync - this is a one way metric
> o) TCP connect only tests (run many parallel TCP connects and measure
> the connect times of each) - an issue for many mobile users is that
> TCP connect times are impacting user experience
> o) -P support for more than 127 and now is only limited by the
> kernel/os (this already fixed)
> o) CPU bound warning on exit - present a warning message when an iperf
> test seemed to be CPU bound vs network i/o bound
> o) Mutex warn message on exit: present a warning on exit when the test
> spent time on mutex locks where that may be degrading network
> performance
> o) Teporter thread consumption rate detector to add thread
> delay/suspends only when needed (currently, the code always add
> reporter thread delay.) This delay can and should reduce the cores
> cache invalidates per cache coherency and the use of shared memory
> (and its pointers) not under os mutex protection
> o) Much better debugging support per ./configure --enable-thread-debug
> o) Code clean up including using function pointers to replace per
> pacet, inline runtime tests (prepping to complete the C++
> implementation vs today's hybrid of C and C++)
> o) Support for raw sockets to test "L2 level" performance
> o) Better code encapsulation (which includes freeing of memory)
>
> Note: Here's an example debug that's now supported for those who want
> to delve into iperf internals. 
>
> Client:
> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1 -i 1 -t 4
> THREAD(18576):[16:28:11.702942] Thread settings copy (malloc)
> from/to=0x67ac20/0x67b640
> THREAD(18576):[16:28:11.703128] Thread_run_wrapper(0x67ac20 mode=2)
> thread counts tot/trfc=1/1
> THREAD(18576):[16:28:11.703293] Thread_run_wrapper(0x67b640 mode=4)
> thread counts tot/trfc=2/1
> THREAD(18577):[16:28:11.703305] Client thread started in constructor
> THREAD(18578):[16:28:11.704364] Reporter thread started
> THREAD(18577):[16:28:11.708777] Init settings report 0x7f2a38020fb0
> THREAD(18577):[16:28:11.708839] Update connection report
> 0x7f2a38020fb0 winreq=0 actual=87040
> THREAD(18577):[16:28:11.708853] Post report 0x7f2a38020fb0
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> THREAD(18577):[16:28:11.708895] Init 5000 element packet ring
> 0x7f2a38023ee0
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> THREAD(18577):[16:28:11.708961] Init data report 0x7f2a380239a0 size
> 1336 using packetring 0x7f2a38023ee0
> THREAD(18578):[16:28:11.709000] Remove 0x7f2a38020fb0 from reporter
> job queue in rs
> THREAD(18577):[16:28:11.709013] Init connection report 0x7f2a380239a0
> THREAD(18578):[16:28:11.709037] Free 0x7f2a38020fb0 in rs
> THREAD(18577):[16:28:11.709053] Update connection report
> 0x7f2a380239a0 winreq=0 actual=87040
> THREAD(18577):[16:28:11.709080] Post report 0x7f2a380239a0
> [ 3] local 192.168.1.4 port 36276 connected with 192.168.1.1 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 7.75 MBytes 65.0 Mbits/sec
> [ 3] 1.0- 2.0 sec 7.75 MBytes 65.0 Mbits/sec
> [ 3] 2.0- 3.0 sec 6.50 MBytes 54.5 Mbits/sec
> [ 3] 3.0- 4.0 sec 7.12 MBytes 59.8 Mbits/sec
> [ 3] 0.0- 4.0 sec 29.1 MBytes 60.5 Mbits/sec
> THREAD(18578):[16:28:15.747044] Remove 0x7f2a380239a0 from reporter
> job queue in rs
> THREAD(18577):[16:28:15.747045] Traffic thread thinks reporter is done
> with 0x7f2a380239a0
> THREAD(18577):[16:28:15.747098] Client destructor close sock=3
> THREAD(18577):[16:28:15.747126] Free packet ring 0x7f2a38023ee0 &
> condition variable await consumer 0x7f2a38023ef8
> THREAD(18577):[16:28:15.747156] Free report hdr 0x7f2a380239a0 delay
> counter=994
> THREAD(18577):[16:28:15.747168] Thread settings free=0x67ac20
> THREAD(18578):[16:28:16.000164] Thread settings free=0x67b640
>
> Server:
>
> [root@localhost iperf2-code]# src/iperf -s -i 1 
> THREAD(19735):[16:27:43.620063] Thread settings copy (malloc)
> from/to=0x12dac20/0x12daf10
> THREAD(19735):[16:27:43.620250] Thread_run_wrapper(0x12dac20 mode=8)
> thread counts tot/trfc=1/0
> THREAD(19735):[16:27:43.620338] Thread_run_wrapper(0x12daf10 mode=3)
> thread counts tot/trfc=2/0
> THREAD(19736):[16:27:43.620505] Init settings report 0x7f6c84020930
> THREAD(19736):[16:27:43.620552] Update connection report
> 0x7f6c84020930 winreq=0 actual=87380
> THREAD(19736):[16:27:43.620578] Post report 0x7f6c84020930
> THREAD(19736):[16:27:43.620619] Thread settings copy (malloc)
> from/to=0x12dac20/0x7f6c84023410
> THREAD(19736):[16:27:43.620636] Listener thread accepting for TCP
> (sock=3)
> THREAD(19737):[16:27:43.620677] Reporter thread started
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> THREAD(19737):[16:27:43.620780] Remove 0x7f6c84020930 from reporter
> job queue in rs
> THREAD(19737):[16:27:43.620799] Free 0x7f6c84020930 in rs
> THREAD(19736):[16:28:11.711590] Thread_run_wrapper(0x7f6c84023410
> mode=1) thread counts tot/trfc=3/1
> THREAD(19736):[16:28:11.711658] Thread settings copy (malloc)
> from/to=0x12dac20/0x7f6c84020b90
> THREAD(19739):[16:28:11.711715] Server thread started (sock=4)
> THREAD(19736):[16:28:11.711844] Listener thread accepting for TCP
> (sock=3)
> THREAD(19739):[16:28:11.712438] Init 5000 element packet ring
> 0x7f6c80020f80
> THREAD(19739):[16:28:11.712470] Init data report 0x7f6c80020a40 size
> 1336 using packetring 0x7f6c80020f80
> THREAD(19739):[16:28:11.712496] Init connection report 0x7f6c80020a40
> THREAD(19739):[16:28:11.712519] Update connection report
> 0x7f6c80020a40 winreq=0 actual=374400
> THREAD(19739):[16:28:11.712535] Post report 0x7f6c80020a40
> [ 4] local 192.168.1.1 port 5001 connected with 192.168.1.4 port 36276
> [ ID] Interval Transfer Bandwidth
> [ 4] 0.0- 1.0 sec 6.70 MBytes 56.2 Mbits/sec
> [ 4] 1.0- 2.0 sec 7.43 MBytes 62.3 Mbits/sec
> [ 4] 2.0- 3.0 sec 6.91 MBytes 58.0 Mbits/sec
> [ 4] 3.0- 4.0 sec 6.89 MBytes 57.8 Mbits/sec
> [ 4] 0.0- 4.2 sec 29.1 MBytes 58.7 Mbits/sec
> THREAD(19737):[16:28:15.878976] Remove 0x7f6c80020a40 from reporter
> job queue in rs
> THREAD(19739):[16:28:16.000140] Traffic thread thinks reporter is done
> with 0x7f6c80020a40
> THREAD(19739):[16:28:16.000208] Server destructor close sock=4
> drop-sock=-1
> THREAD(19739):[16:28:16.000387] Free packet ring 0x7f6c80020f80 &
> condition variable await consumer 0x7f6c80020f98
> THREAD(19739):[16:28:16.000487] Free report hdr 0x7f6c80020a40 delay
> counter=1030
> THREAD(19739):[16:28:16.000529] Thread settings free=0x7f6c84023410
> THREAD(19737):[16:29:07.000175] Thread settings free=0x12daf10
>
> Thanks,
> Bob
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

Re: [Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Toke Høiland-Jørgensen]

Bob McMahon <bob.mcmahon@broadcom.com> writes:

> Just an FYI in case anybody has interest in traffic tooling.

I do! Specifically, I am planning to teach Flent to automatically switch
between iperf and netperf as the underlying test tool[0]. I believe that
there are a few netperf features missing from iperf that Flent currently
uses, so I'll get back to you with actual feature requests for those
once I've had a chance to take a look at this in more detail :)

-Toke

[0] The main driver for this is netperf's weird license which makes it
impossible to package for FOSS-only distributions; leading to things
like this: https://bugzilla.redhat.com/show_bug.cgi?id=1729939

Re: [Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Bob McMahon]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

Hi Toke,

Do let me know.  We're focused on the network i/o testing aspect (per being
a WiFi chip vendor) and are intentionally not trying to provide CPU load
metrics. (I think netperf provides both.)  A feature we are adding is to
warn when we think something other than the socket reads() and writes()
have become bottlenecks, e.g. in a CPU constrained system it becomes an
"entangled metric" between i/o and CPU though still presents in network i/o
units which can be misleading to network device vendors.

Also, many might want to consider monitoring "network power" which is
average throughput / latency or delay, i.e. "something good" / "something
bad"

Bob

On Wed, Jul 17, 2019 at 3:26 PM Toke Høiland-Jørgensen <toke@redhat.com>
wrote:

> Bob McMahon <bob.mcmahon@broadcom.com> writes:
>
> > Just an FYI in case anybody has interest in traffic tooling.
>
> I do! Specifically, I am planning to teach Flent to automatically switch
> between iperf and netperf as the underlying test tool[0]. I believe that
> there are a few netperf features missing from iperf that Flent currently
> uses, so I'll get back to you with actual feature requests for those
> once I've had a chance to take a look at this in more detail :)
>
> -Toke
>
> [0] The main driver for this is netperf's weird license which makes it
> impossible to package for FOSS-only distributions; leading to things
> like this: https://bugzilla.redhat.com/show_bug.cgi?id=1729939
>

[-- Attachment #2: Type: text/html, Size: 2026 bytes --]

Re: [Make-wifi-fast] Fwd: iperf 2 "short term" road map

[Toke Høiland-Jørgensen]

Bob McMahon <bob.mcmahon@broadcom.com> writes:

> Hi Toke,
>
> Do let me know.  We're focused on the network i/o testing aspect (per being
> a WiFi chip vendor) and are intentionally not trying to provide CPU load
> metrics. (I think netperf provides both.)  A feature we are adding is to
> warn when we think something other than the socket reads() and writes()
> have become bottlenecks, e.g. in a CPU constrained system it becomes an
> "entangled metric" between i/o and CPU though still presents in network i/o
> units which can be misleading to network device vendors.
>
> Also, many might want to consider monitoring "network power" which is
> average throughput / latency or delay, i.e. "something good" / "something
> bad"

It's more socket-level statistics I'm after for Flent use; we generally
run multiple instances (flows) concurrently and graph the results. This
is the variables we currently extract from Netperf:

    output_vars = 'THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,' \
                  'TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,' \
                  'REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,' \
                  'REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,' \
                  'LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,' \
                  'REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,' \
                  'LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,' \
                  'REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD'

Some of these can be set as well (such as TOS and congestion control
algorithm).


Also, how safe is it to run an iperf server instance on the public
internet? :)

-Toke