<div dir="ltr">Hi All,<div><br>I've added an experimental feature to <a href="https://sourceforge.net/projects/iperf2/" target="_blank">iperf 2.0.14</a> per option --near-congestion=<value>. This is useful when measuring TCP write to read latencies at high throughput but not at bufferbloat. It requires a very much controlled network to be repeatable and, even then, still requires some knob tuning.<br><br><b>The idea is to rate limit the tcp writes by sampling the RTT after each write buffer completes (per -l.)</b> The delay is weighted per a multiplier of the RTT. This allows the queues to drain. The RTT multiplier can be any value zero or greater. It's loosely based <a href="https://www.lk.cs.ucla.edu/data/files/Kleinrock/Internet%20Congestion%20Control%20Using%20the%20Power%20Metric-Keep%20the%20Pipe%20Just%20Full%2C%20But%20No%20Fuller%20July%202018.pdf" target="_blank">off this paper</a>.<br><br>This feature is committed in version starting Dec 22<br><br>[rjmcmahon@localhost iperf2-code]$ iperf -v<br>iperf version 2.0.14a (22 Dec 2020) pthreads<br><br>Here's an example testing against a raspberry pi 4 wired gigE.</div><div><br></div><div><b>First with the delay being 0.4 of the RTT sampled after the -l write size.</b> Throughput is fairly high and the write to read latency averages 3.5 ms.<br><br>[rjmcmahon@localhost Code]$ iperf -c 192.168.1.108 -i 1 --trip-times<span style="background-color:rgb(255,255,0)"> --near-congestion=0.4</span> -e<br>------------------------------------------------------------<br>Client connecting to 192.168.1.108, TCP port 5001 with pid 11219 (1 flows)<br>Write buffer size: 131072 Byte<br>TCP near-congestion delay weight set to 0.4000<br>TCP window size: 85.0 KByte (default)<br>------------------------------------------------------------<br>[ 1] local 192.168.1.62%enp2s0 port 48100 connected with 192.168.1.108 port 5001 (MSS=1448) (trip-times) (sock=3) (ct=0.28 ms) on 2020-12-22 22:32:39 (PST)<br>[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr<br>[ 1] 0.00-1.00 sec 112 MBytes 943 Mbits/sec 900/0 0 567K/2566 us 45921<br>[ 1] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 897/0 0 567K/1653 us 71126<br>[ 1] 2.00-3.00 sec 111 MBytes 931 Mbits/sec 888/0 0 695K/2418 us 48136<br>[ 1] 3.00-4.00 sec 111 MBytes 932 Mbits/sec 889/0 0 695K/1407 us 82817<br>[ 1] 4.00-5.00 sec 109 MBytes 913 Mbits/sec 871/0 0 695K/2062 us 55366<br>[ 1] 5.00-6.00 sec 112 MBytes 935 Mbits/sec 892/0 0 695K/2484 us 47068<br>[ 1] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 895/0 0 695K/2511 us 46718<br>[ 1] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 895/0 0 695K/1895 us 61905<br>[ 1] 8.00-9.00 sec 111 MBytes 929 Mbits/sec 886/0 0 695K/2495 us 46545<br>[ 1] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 896/0 0 695K/3403 us 34511<br>[ 1] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec 8912/0 0 695K/3403 us <span style="background-color:rgb(255,255,0)">34322</span><br><br>root@rpi4-rjm-1:/usr/local/src/iperf2-code# src/iperf -s -i 1 -e<br>------------------------------------------------------------<br>Server listening on TCP port 5001 with pid 3805<br>Read buffer size: 128 KByte (Dist bin width=16.0 KByte)<br>TCP window size: 128 KByte (default)<br>------------------------------------------------------------<br>[ 1] local 192.168.1.108%eth0 port 5001 connected with 192.168.1.62 port 48100 (MSS=453) (trip-times) (sock=4) (peer 2.0.14-alpha) on 2020-12-22 22:32:39 (PST)<br>[ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist<br>[ 1] 0.00-1.00 sec 112 MBytes 940 Mbits/sec 3.668/2.166/12.241/1.761 ms (896/131130) 423 KByte 32032 46203=46139:39:7:7:4:2:0:5<br>[ 1] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 3.172/2.295/5.212/0.158 ms (896/131013) 364 KByte 37009 38631=38511:93:3:0:1:0:0:23<br>[ 1] 2.00-3.00 sec 111 MBytes 932 Mbits/sec 3.447/1.205/6.307/0.997 ms (889/131103) 391 KByte 33815 31142=30806:18:0:0:0:0:0:318<br>[ 1] 3.00-4.00 sec 111 MBytes 932 Mbits/sec 4.485/1.247/10.803/2.049 ms (889/131040) 512 KByte 25972 24613=24166:4:2:0:0:1:0:440<br>[ 1] 4.00-5.00 sec 109 MBytes 913 Mbits/sec 3.866/1.248/6.951/1.442 ms (871/131072) 431 KByte 29529 871=0:0:0:0:0:0:0:871<br>[ 1] 5.00-6.00 sec 112 MBytes 936 Mbits/sec 3.295/1.249/5.933/0.750 ms (892/131192) 376 KByte 35514 37172=36953:6:4:1:0:0:1:207<br>[ 1] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 3.537/1.252/9.147/1.041 ms (895/131019) 405 KByte 33153 39414=39229:14:3:1:0:0:0:167<br>[ 1] 7.00-8.00 sec 112 MBytes 936 Mbits/sec 3.467/1.224/7.949/0.904 ms (893/131004) 396 KByte 33742 38755=38566:10:2:0:0:0:0:177<br>[ 1] 8.00-9.00 sec 111 MBytes 932 Mbits/sec 3.510/1.212/6.808/1.064 ms (888/131132) 399 KByte 33173 27034=26636:2:2:1:0:0:0:393<br>[ 1] 9.00-10.00 sec 112 MBytes 937 Mbits/sec 3.308/2.732/7.023/0.600 ms (894/131012) 379 KByte 35406 43303=43197:15:4:0:0:0:0:87<br>[ 1] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec <span style="background-color:rgb(255,255,0)">3.576/</span>1.205/12.241/1.249 ms (8909/131072) 408 KByte 32644 327144=324203:201:27:10:5:3:1:2694<br><br></div><div><b>Next, the delay is 0 so <span style="background-color:rgb(255,255,0)">TCP drives to bufferbloat.</span> Network power is 8512 vs 34322 and the average latency is 30 milliseconds vs 3.5.</b><br><br><b style="background-color:rgb(255,255,0)">Trading off a little bit of top line throughput for a major improvement in latency.</b><br><br>[rjmcmahon@localhost Code]$ iperf -c 192.168.1.108 -i 1 --trip-times <span style="background-color:rgb(255,255,0)">--near-congestion=0</span> -e<br>------------------------------------------------------------<br>Client connecting to 192.168.1.108, TCP port 5001 with pid 12140 (1 flows)<br>Write buffer size: 131072 Byte<br>TCP near-congestion delay weight set to 0.0000<br>TCP window size: 85.0 KByte (default)<br>------------------------------------------------------------<br>[ 1] local 192.168.1.62%enp2s0 port 48662 connected with 192.168.1.108 port 5001 (MSS=1448) (trip-times) (sock=3) (ct=0.19 ms) on 2020-12-22 22:36:18 (PST)<br>[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr<br>[ 1] 0.00-1.00 sec 116 MBytes 970 Mbits/sec 926/0 0 1528K/13054 us 9288<br>[ 1] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1671K/14330 us 8214<br>[ 1] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 900/0 0 1784K/15225 us 7748<br>[ 1] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1870K/16005 us 7354<br>[ 1] 4.00-5.00 sec 112 MBytes 935 Mbits/sec 892/0 0 1934K/16513 us 7080<br>[ 1] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1442K/12335 us 9542<br>[ 1] 6.00-7.00 sec 112 MBytes 940 Mbits/sec 896/0 0 1518K/12872 us 9124<br>[ 1] 7.00-8.00 sec 113 MBytes 945 Mbits/sec 901/0 0 1571K/13423 us 8798<br>[ 1] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 897/0 0 1607K/13718 us 8571<br>[ 1] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 896/0 0 1631K/13863 us 8472<br>[ 1] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 9005/0 70 1631K/13863 us <span style="background-color:rgb(255,255,0)">8512</span><br><br>root@rpi4-rjm-1:/usr/local/src/iperf2-code# src/iperf -s -i 1 -e<br>------------------------------------------------------------<br>Server listening on TCP port 5001 with pid 3820<br>Read buffer size: 128 KByte (Dist bin width=16.0 KByte)<br>TCP window size: 128 KByte (default)<br>------------------------------------------------------------<br>[ 1] local 192.168.1.108%eth0 port 5001 connected with 192.168.1.62 port 48662 (MSS=453) (trip-times) (sock=4) (peer 2.0.14-alpha) on 2020-12-22 22:36:18 (PST)<br>[ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist<br>[ 1] 0.00-1.00 sec 112 MBytes 940 Mbits/sec 28.251/2.674/50.071/7.127 ms (896/131160) 3.27 MByte 4160 50060=49977:31:13:8:3:3:5:20<br>[ 1] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 30.198/24.293/36.232/3.449 ms (898/131058) 3.39 MByte 3897 54903=54902:1:0:0:0:0:0:0<br>[ 1] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 30.184/24.276/36.088/3.467 ms (898/131053) 3.38 MByte 3899 54905=54903:2:0:0:0:0:0:0<br>[ 1] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 30.172/24.259/36.306/3.444 ms (898/131060) 3.39 MByte 3901 55248=55245:3:0:0:0:0:0:0<br>[ 1] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 30.170/24.267/36.208/3.433 ms (898/131053) 3.40 MByte 3901 55389=55383:6:0:0:0:0:0:0<br>[ 1] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 30.302/15.238/48.700/3.973 ms (898/131056) 3.40 MByte 3884 53258=53224:7:0:0:0:1:1:25<br>[ 1] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 30.140/24.269/36.058/3.454 ms (897/131204) 3.36 MByte 3905 55104=55102:2:0:0:0:0:0:0<br>[ 1] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 30.267/24.299/36.244/3.430 ms (898/131053) 3.42 MByte 3888 55107=55106:1:0:0:0:0:0:0<br>[ 1] 8.00-9.00 sec 112 MBytes 942 Mbits/sec 30.179/24.241/36.158/3.446 ms (898/131056) 3.40 MByte 3900 55194=55192:1:1:0:0:0:0:0<br>[ 1] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 30.170/24.273/36.415/3.452 ms (898/131056) 3.39 MByte 3901 55100=55095:5:0:0:0:0:0:0<br>[ 1] 0.00-10.03 sec 1.10 GBytes 941 Mbits/sec <span style="background-color:rgb(255,255,0)">30.005</span>/2.674/50.071/4.059 ms (9002/131072) 3.37 MByte 3922 545735=545595:60:14:8:3:4:6:45<font color="#888888"><font color="#888888"><br><br>Bob</font></font></div></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span>