[Rpm] [LibreQoS] the grinch meets cloudflare's christmas present
rjmcmahon
rjmcmahon at rjmcmahon.com
Fri Jan 6 15:47:40 EST 2023
For responsiveness, the bounceback seems reasonable even with upstream
competition. Bunch more TCP retries though.
[rjmcmahon at ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e
--trip-times -i 1 --bounceback -t 3
------------------------------------------------------------
Client connecting to (**hidden**), TCP port 5001 with pid 111022 (1
flows)
Write buffer size: 100 Byte
Bursting: 100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[ 1] local *.*.*.86%enp7s0 port 36976 connected with *.*.*.123 port
5001 (prefetch=16384) (bb w/quickack len/hold=100/0) (trip-times)
(sock=3) (icwnd/mss/irtt=14/1448/9862) (ct=9.90 ms) on 2023-01-06
12:42:18 (PST)
[ ID] Interval Transfer Bandwidth BB
cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
[ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
10=12.195/9.298/16.457/2.679 ms 0 14K/11327 us 82 rps
[ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
10=12.613/9.271/15.489/2.788 ms 0 14K/12165 us 79 rps
[ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
10=13.390/9.376/15.986/2.520 ms 0 14K/13164 us 75 rps
[ 1] 0.00-3.03 sec 5.86 KBytes 15.8 Kbits/sec
30=12.733/9.271/16.457/2.620 ms 0 14K/15138 us 79 rps
[ 1] 0.00-3.03 sec OWD Delays (ms) Cnt=30 To=7.937/4.634/11.327/2.457
From=4.778/4.401/5.350/0.258 Asymmetry=3.166/0.097/6.311/2.318 79 rps
[ 1] 0.00-3.03 sec BB8(f)-PDF:
bin(w=100us):cnt(30)=93:2,94:3,95:2,97:1,100:1,102:1,105:1,114:2,142:1,143:1,144:2,145:3,146:1,147:1,148:1,151:1,152:1,154:1,155:1,156:1,160:1,165:1
(5.00/95.00/99.7%=93/160/165,Outliers=0,obl/obu=0/0)
[rjmcmahon at ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e
--trip-times -i 1 --bounceback -t 3 --bounceback-congest=up,4
------------------------------------------------------------
Client connecting to (**hidden**), TCP port 5001 with pid 111069 (1
flows)
Write buffer size: 100 Byte
Bursting: 100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs &
tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[ 2] local *.*.*.85%enp4s0 port 38342 connected with *.*.*.123 port
5001 (prefetch=16384) (trip-times) (sock=3) (qack)
(icwnd/mss/irtt=14/1448/10613) (ct=10.66 ms) on 2023-01-06 12:42:36
(PST)
[ 1] local *.*.*.85%enp4s0 port 38360 connected with *.*.*.123 port
5001 (prefetch=16384) (bb w/quickack len/hold=100/0) (trip-times)
(sock=4) (icwnd/mss/irtt=14/1448/14901) (ct=14.96 ms) on 2023-01-06
12:42:36 (PST)
[ 3] local *.*.*.85%enp4s0 port 38386 connected with *.*.*.123 port
5001 (prefetch=16384) (trip-times) (sock=7) (qack)
(icwnd/mss/irtt=14/1448/15295) (ct=15.31 ms) on 2023-01-06 12:42:36
(PST)
[ 4] local *.*.*.85%enp4s0 port 38348 connected with *.*.*.123 port
5001 (prefetch=16384) (trip-times) (sock=5) (qack)
(icwnd/mss/irtt=14/1448/14901) (ct=14.95 ms) on 2023-01-06 12:42:36
(PST)
[ 5] local *.*.*.85%enp4s0 port 38372 connected with *.*.*.123 port
5001 (prefetch=16384) (trip-times) (sock=6) (qack)
(icwnd/mss/irtt=14/1448/15371) (ct=15.42 ms) on 2023-01-06 12:42:36
(PST)
[ ID] Interval Transfer Bandwidth Write/Err Rtry
Cwnd/RTT(var) NetPwr
[ 3] 0.00-1.00 sec 1.29 MBytes 10.8 Mbits/sec 13502/0 115
28K/22594(904) us 59.76
[ 4] 0.00-1.00 sec 1.63 MBytes 13.6 Mbits/sec 17048/0 140
42K/22728(568) us 75.01
[ ID] Interval Transfer Bandwidth BB
cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS
[ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec
10=76.140/17.224/123.195/43.168 ms 0 14K/68136 us 13 rps
[ 5] 0.00-1.00 sec 1.04 MBytes 8.72 Mbits/sec 10893/0 82
25K/23400(644) us 46.55
[SUM] 0.00-1.00 sec 3.95 MBytes 33.2 Mbits/sec 41443/0 337
[ 2] 0.00-1.00 sec 1.10 MBytes 9.25 Mbits/sec 11566/0 77
22K/23557(432) us 49.10
[ 3] 1.00-2.00 sec 1.24 MBytes 10.4 Mbits/sec 13037/0 20
28K/14427(503) us 90.37
[ 4] 1.00-2.00 sec 1.43 MBytes 12.0 Mbits/sec 14954/0 31
12K/13348(407) us 112
[ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec
10=14.581/10.801/20.356/3.599 ms 0 14K/27791 us 69 rps
[ 5] 1.00-2.00 sec 1.26 MBytes 10.6 Mbits/sec 13191/0 16
12K/14749(675) us 89.44
[SUM] 1.00-2.00 sec 3.93 MBytes 32.9 Mbits/sec 41182/0 67
[ 2] 1.00-2.00 sec 1000 KBytes 8.19 Mbits/sec 10237/0 13
19K/14467(1068) us 70.76
[ 3] 2.00-3.00 sec 1.33 MBytes 11.2 Mbits/sec 13994/0 4
24K/20749(495) us 67.44
[ 4] 2.00-3.00 sec 1.20 MBytes 10.1 Mbits/sec 12615/0 3
31K/20877(718) us 60.43
[ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec
10=11.298/9.407/14.245/1.330 ms 0 14K/15474 us 89 rps
[ 5] 2.00-3.00 sec 1.08 MBytes 9.03 Mbits/sec 11284/0 3
28K/21031(430) us 53.65
[SUM] 2.00-3.00 sec 3.61 MBytes 30.3 Mbits/sec 37893/0 10
[ 2] 2.00-3.00 sec 1.29 MBytes 10.8 Mbits/sec 13492/0 3
29K/20409(688) us 66.11
[ 3] 0.00-3.03 sec 3.87 MBytes 10.7 Mbits/sec 40534/0 139
25K/20645(557) us 64.85
[ 5] 0.00-3.03 sec 3.37 MBytes 9.35 Mbits/sec 35369/0 101
29K/20489(668) us 57.02
[ 4] 0.00-3.03 sec 4.26 MBytes 11.8 Mbits/sec 44618/0 174
32K/21240(961) us 69.40
[ 2] 0.00-3.03 sec 3.37 MBytes 9.31 Mbits/sec 35296/0 94
19K/21504(948) us 54.13
[ 1] 0.00-3.14 sec 7.81 KBytes 20.4 Kbits/sec
40=28.332/5.611/123.195/34.940 ms 0 14K/14000 us 35 rps
[ 1] 0.00-3.14 sec OWD Delays (ms) Cnt=40
To=23.730/1.110/118.744/34.957 From=4.567/4.356/5.171/0.141
Asymmetry=19.332/0.189/114.294/34.869 35 rps
[ 1] 0.00-3.14 sec BB8(f)-PDF:
bin(w=100us):cnt(40)=57:1,94:2,95:2,96:2,98:1,101:1,106:1,109:2,111:1,112:2,113:1,115:1,118:1,119:2,143:2,145:1,146:1,152:1,158:1,173:1,176:1,194:1,195:1,204:1,205:1,274:1,554:1,790:1,925:1,1125:1,1126:1,1225:1,1232:1
(5.00/95.00/99.7%=94/1225/1232,Outliers=0,obl/obu=0/0)
[SUM] 0.00-3.11 sec 11.5 MBytes 31.0 Mbits/sec 120521/0 414
[ CT] final connect times (min/avg/max/stdev) =
10.661/14.261/15.423/2023.369 ms (tot/err) = 5/0
> Some thoughts are not to use UDP for testing here. Also, these speed
> tests have little to no information for network engineers about what's
> going on. Iperf 2 may better assist network engineers but then I'm
> biased ;)
>
> Running iperf 2 https://sourceforge.net/projects/iperf2/ with
> --trip-times. Though the sampling and central limit theorem averaging
> is hiding the real distributions (use --histograms to get those)
>
> Below are 4 parallel TCP streams from my home to one of my servers in
> the cloud. First where TCP is limited per CCA. Second is source side
> write rate limiting. Things to note:
>
> o) connect times for both at 10-15 ms
> o) multiple TCP retries on a few rites - one case is 4 with 5 writes.
> Source side pacing eliminates retries
> o) Fairness with CCA isn't great but quite good with source side write
> pacing
> o) Queue depth with CCA is about 150 Kbytes about 100K byte with
> source side pacing
> o) min write to read is about 80 ms for both
> o) max is 220 ms vs 97 ms
> o) stdev for CCA write/read is 30 ms vs 3 ms
> o) TCP RTT is 20ms w/CCA and 90 ms with ssp - seems odd here as
> TCP_QUICACK and TCP_NODELAY are both enabled.
>
> [ CT] final connect times (min/avg/max/stdev) =
> 10.326/13.522/14.986/2150.329 ms (tot/err) = 4/0
> [rjmcmahon at ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e
> --trip-times -i 1 -P 4 -t 10 -w 4m --tcp-quickack -N
> ------------------------------------------------------------
> Client connecting to (**hidden**), TCP port 5001 with pid 107678 (4
> flows)
> Write buffer size: 131072 Byte
> TOS set to 0x0 and nodelay (Nagle off)
> TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
> Event based writes (pending queue watermark at 16384 bytes)
> ------------------------------------------------------------
> [ 1] local *.*.*.85%enp4s0 port 42480 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=3) (qack)
> (icwnd/mss/irtt=14/1448/10534) (ct=10.63 ms) on 2023-01-06 12:17:56
> (PST)
> [ 4] local *.*.*.85%enp4s0 port 42488 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=5) (qack)
> (icwnd/mss/irtt=14/1448/14023) (ct=14.08 ms) on 2023-01-06 12:17:56
> (PST)
> [ 3] local *.*.*.85%enp4s0 port 42502 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=6) (qack)
> (icwnd/mss/irtt=14/1448/14642) (ct=14.70 ms) on 2023-01-06 12:17:56
> (PST)
> [ 2] local *.*.*.85%enp4s0 port 42484 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=4) (qack)
> (icwnd/mss/irtt=14/1448/14728) (ct=14.79 ms) on 2023-01-06 12:17:56
> (PST)
> [ ID] Interval Transfer Bandwidth Write/Err Rtry
> Cwnd/RTT(var) NetPwr
> ...
> [ 4] 4.00-5.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 3
> 29K/21088(1142) us 68.37
> [ 2] 4.00-5.00 sec 1.62 MBytes 13.6 Mbits/sec 13/0 2
> 31K/19284(612) us 88.36
> [ 1] 4.00-5.00 sec 896 KBytes 7.34 Mbits/sec 7/0 5
> 16K/18996(658) us 48.30
> [ 3] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 5
> 18K/18133(208) us 57.83
> [SUM] 4.00-5.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 15
> [ 4] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 4
> 29K/14717(489) us 89.06
> [ 1] 5.00-6.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 4
> 16K/15874(408) us 66.06
> [ 3] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 4
> 16K/15826(382) us 74.54
> [ 2] 5.00-6.00 sec 1.50 MBytes 12.6 Mbits/sec 12/0 6
> 9K/14878(557) us 106
> [SUM] 5.00-6.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 18
> [ 4] 6.00-7.00 sec 1.75 MBytes 14.7 Mbits/sec 14/0 4
> 25K/15472(496) us 119
> [ 2] 6.00-7.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 2
> 26K/16417(427) us 63.87
> [ 1] 6.00-7.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 5
> 16K/16268(679) us 80.57
> [ 3] 6.00-7.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 6
> 15K/16629(799) us 63.06
> [SUM] 6.00-7.00 sec 5.00 MBytes 41.9 Mbits/sec 40/0 17
> [ 4] 7.00-8.00 sec 1.75 MBytes 14.7 Mbits/sec 14/0 4
> 22K/13986(519) us 131
> [ 1] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 4
> 16K/12679(377) us 93.04
> [ 3] 7.00-8.00 sec 896 KBytes 7.34 Mbits/sec 7/0 5
> 14K/12971(367) us 70.74
> [ 2] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 6
> 15K/14740(779) us 80.03
> [SUM] 7.00-8.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 19
>
> [root at bobcat iperf2-code]# iperf -s -i 1 -e --hide-ips -w 4m
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 233615
> Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
> TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
> ------------------------------------------------------------
> [ 1] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 42480 (trip-times) (sock=4) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11636) on 2023-01-06 12:17:56 (PST)
> [ 2] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 42502 (trip-times) (sock=5) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11898) on 2023-01-06 12:17:56 (PST)
> [ 3] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 42484 (trip-times) (sock=6) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11938) on 2023-01-06 12:17:56 (PST)
> [ 4] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 42488 (trip-times) (sock=7) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11919) on 2023-01-06 12:17:56 (PST)
> [ ID] Interval Transfer Bandwidth Burst Latency
> avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
> ...
> [ 2] 4.00-5.00 sec 1.06 MBytes 8.86 Mbits/sec
> 129.819/90.391/186.075/31.346 ms (9/123080) 154 KByte 8.532803
> 467=461:6:0:0:0:0:0:0
> [ 3] 4.00-5.00 sec 1.52 MBytes 12.8 Mbits/sec
> 103.552/82.653/169.274/28.382 ms (12/132854) 149 KByte 15.40
> 646=643:1:2:0:0:0:0:0
> [ 4] 4.00-5.00 sec 1.39 MBytes 11.6 Mbits/sec
> 107.503/66.843/143.038/24.269 ms (11/132294) 149 KByte 13.54
> 619=617:1:1:0:0:0:0:0
> [ 1] 4.00-5.00 sec 988 KBytes 8.10 Mbits/sec
> 141.389/119.961/178.785/18.812 ms (7/144593) 170 KByte 7.158641
> 409=404:5:0:0:0:0:0:0
> [SUM] 4.00-5.00 sec 4.93 MBytes 41.4 Mbits/sec
> 2141=2125:13:3:0:0:0:0:0
> [ 4] 5.00-6.00 sec 1.29 MBytes 10.8 Mbits/sec
> 118.943/86.253/176.128/31.248 ms (10/135098) 164 KByte 11.36
> 511=506:2:3:0:0:0:0:0
> [ 2] 5.00-6.00 sec 1.09 MBytes 9.17 Mbits/sec
> 139.821/102.418/218.875/40.422 ms (9/127424) 148 KByte 8.202049
> 487=484:2:1:0:0:0:0:0
> [ 3] 5.00-6.00 sec 1.51 MBytes 12.6 Mbits/sec
> 102.146/77.085/140.893/18.441 ms (13/121520) 151 KByte 15.47
> 640=636:1:3:0:0:0:0:0
> [ 1] 5.00-6.00 sec 981 KBytes 8.04 Mbits/sec
> 161.901/105.582/219.931/36.260 ms (8/125614) 134 KByte 6.206944
> 415=413:2:0:0:0:0:0:0
> [SUM] 5.00-6.00 sec 4.85 MBytes 40.7 Mbits/sec
> 2053=2039:7:7:0:0:0:0:0
> [ 4] 6.00-7.00 sec 1.74 MBytes 14.6 Mbits/sec
> 88.846/74.297/101.859/7.118 ms (14/130526) 156 KByte 20.57
> 711=707:3:1:0:0:0:0:0
> [ 1] 6.00-7.00 sec 1.22 MBytes 10.2 Mbits/sec
> 120.639/100.257/157.567/21.770 ms (10/127568) 157 KByte 10.57
> 494=488:5:1:0:0:0:0:0
> [ 2] 6.00-7.00 sec 1015 KBytes 8.32 Mbits/sec
> 144.632/124.368/171.349/16.597 ms (8/129958) 143 KByte 7.188321
> 408=403:5:0:0:0:0:0:0
> [ 3] 6.00-7.00 sec 1.02 MBytes 8.60 Mbits/sec
> 143.516/102.322/173.001/24.089 ms (8/134302) 146 KByte 7.486359
> 484=480:4:0:0:0:0:0:0
> [SUM] 6.00-7.00 sec 4.98 MBytes 41.7 Mbits/sec
> 2097=2078:17:2:0:0:0:0:0
> [ 4] 7.00-8.00 sec 1.77 MBytes 14.9 Mbits/sec
> 85.406/65.797/103.418/12.609 ms (14/132595) 153 KByte 21.74
> 692=687:2:3:0:0:0:0:0
> [ 2] 7.00-8.00 sec 957 KBytes 7.84 Mbits/sec
> 153.936/131.452/191.464/19.361 ms (7/140042) 160 KByte 6.368199
> 429=425:4:0:0:0:0:0:0
> [ 1] 7.00-8.00 sec 1.13 MBytes 9.44 Mbits/sec
> 131.146/109.737/166.774/22.035 ms (9/131124) 146 KByte 8.998528
> 520=516:4:0:0:0:0:0:0
> [ 3] 7.00-8.00 sec 1.13 MBytes 9.51 Mbits/sec
> 126.512/88.404/220.175/42.237 ms (9/132089) 172 KByte 9.396784
> 527=524:1:2:0:0:0:0:0
> [SUM] 7.00-8.00 sec 4.96 MBytes 41.6 Mbits/sec
> 2168=2152:11:5:0:0:0:0:0
>
> With source side rate limiting to 9 mb/s per stream.
>
> [rjmcmahon at ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e
> --trip-times -i 1 -P 4 -t 10 -w 4m --tcp-quickack -N -b 9m
> ------------------------------------------------------------
> Client connecting to (**hidden**), TCP port 5001 with pid 108884 (4
> flows)
> Write buffer size: 131072 Byte
> TOS set to 0x0 and nodelay (Nagle off)
> TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
> Event based writes (pending queue watermark at 16384 bytes)
> ------------------------------------------------------------
> [ 1] local *.*.*.85%enp4s0 port 46448 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=3) (qack)
> (icwnd/mss/irtt=14/1448/10666) (ct=10.70 ms) on 2023-01-06 12:27:45
> (PST)
> [ 3] local *.*.*.85%enp4s0 port 46460 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=6) (qack)
> (icwnd/mss/irtt=14/1448/16499) (ct=16.54 ms) on 2023-01-06 12:27:45
> (PST)
> [ 2] local *.*.*.85%enp4s0 port 46454 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=4) (qack)
> (icwnd/mss/irtt=14/1448/16580) (ct=16.86 ms) on 2023-01-06 12:27:45
> (PST)
> [ 4] local *.*.*.85%enp4s0 port 46458 connected with *.*.*.123 port
> 5001 (prefetch=16384) (trip-times) (sock=5) (qack)
> (icwnd/mss/irtt=14/1448/16802) (ct=16.83 ms) on 2023-01-06 12:27:45
> (PST)
> [ ID] Interval Transfer Bandwidth Write/Err Rtry
> Cwnd/RTT(var) NetPwr
> ...
> [ 2] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 134K/88055(12329) us 11.91
> [ 4] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 132K/74867(11755) us 14.01
> [ 1] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 134K/89101(13134) us 11.77
> [ 3] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 131K/91451(11938) us 11.47
> [SUM] 4.00-5.00 sec 4.00 MBytes 33.6 Mbits/sec 32/0 0
> [ 2] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 134K/85135(14580) us 13.86
> [ 4] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 132K/85124(15654) us 13.86
> [ 1] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 134K/91336(11335) us 12.92
> [ 3] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 131K/89185(13499) us 13.23
> [SUM] 5.00-6.00 sec 4.50 MBytes 37.7 Mbits/sec 36/0 0
> [ 2] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 134K/85687(13489) us 13.77
> [ 4] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 132K/82803(13001) us 14.25
> [ 1] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 134K/86869(15186) us 13.58
> [ 3] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
> 131K/91447(12515) us 12.90
> [SUM] 6.00-7.00 sec 4.50 MBytes 37.7 Mbits/sec 36/0 0
> [ 2] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 134K/81814(13168) us 12.82
> [ 4] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 132K/89008(13283) us 11.78
> [ 1] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 134K/89494(12151) us 11.72
> [ 3] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0
> 131K/91083(12797) us 11.51
> [SUM] 7.00-8.00 sec 4.00 MBytes 33.6 Mbits/sec 32/0 0
>
> [root at bobcat iperf2-code]# iperf -s -i 1 -e --hide-ips -w 4m
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 233981
> Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
> TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
> ------------------------------------------------------------
> [ 1] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 46448 (trip-times) (sock=4) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11987) on 2023-01-06 12:27:45 (PST)
> [ 2] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 46454 (trip-times) (sock=5) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11132) on 2023-01-06 12:27:45 (PST)
> [ 3] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 46460 (trip-times) (sock=6) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/11097) on 2023-01-06 12:27:45 (PST)
> [ 4] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port
> 46458 (trip-times) (sock=7) (peer 2.1.9-master) (qack)
> (icwnd/mss/irtt=14/1448/17823) on 2023-01-06 12:27:45 (PST)
> [ ID] Interval Transfer Bandwidth Burst Latency
> avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
> [ 4] 0.00-1.00 sec 1.09 MBytes 9.15 Mbits/sec
> 93.383/90.103/95.661/2.232 ms (8/143028) 128 KByte 12.25
> 451=451:0:0:0:0:0:0:0
> [ 3] 0.00-1.00 sec 1.08 MBytes 9.06 Mbits/sec
> 96.834/95.229/102.645/2.442 ms (8/141580) 131 KByte 11.70
> 472=472:0:0:0:0:0:0:0
> [ 1] 0.00-1.00 sec 1.10 MBytes 9.19 Mbits/sec
> 95.183/92.623/97.579/1.431 ms (8/143571) 131 KByte 12.07
> 495=495:0:0:0:0:0:0:0
> [ 2] 0.00-1.00 sec 1.09 MBytes 9.15 Mbits/sec
> 89.317/84.865/94.906/3.674 ms (8/143028) 122 KByte 12.81
> 489=489:0:0:0:0:0:0:0
> [ ID] Interval Transfer Bandwidth Reads=Dist
> [SUM] 0.00-1.00 sec 4.36 MBytes 36.6 Mbits/sec
> 1907=1907:0:0:0:0:0:0:0
> [ 4] 1.00-2.00 sec 1.07 MBytes 8.95 Mbits/sec
> 92.649/89.987/95.036/1.828 ms (9/124314) 96.5 KByte 12.08
> 492=492:0:0:0:0:0:0:0
> [ 3] 1.00-2.00 sec 1.06 MBytes 8.93 Mbits/sec
> 96.305/95.647/96.794/0.432 ms (9/123992) 100 KByte 11.59
> 480=480:0:0:0:0:0:0:0
> [ 1] 1.00-2.00 sec 1.06 MBytes 8.89 Mbits/sec
> 92.578/90.866/94.145/1.371 ms (9/123510) 95.8 KByte 12.01
> 513=513:0:0:0:0:0:0:0
> [ 2] 1.00-2.00 sec 1.07 MBytes 8.96 Mbits/sec
> 90.767/87.984/94.352/1.944 ms (9/124475) 94.7 KByte 12.34
> 489=489:0:0:0:0:0:0:0
> [SUM] 1.00-2.00 sec 4.26 MBytes 35.7 Mbits/sec
> 1974=1974:0:0:0:0:0:0:0
> [ 4] 2.00-3.00 sec 1.09 MBytes 9.13 Mbits/sec
> 93.977/91.795/96.561/1.693 ms (8/142656) 112 KByte 12.14
> 497=497:0:0:0:0:0:0:0
> [ 3] 2.00-3.00 sec 1.08 MBytes 9.04 Mbits/sec
> 96.544/95.815/97.798/0.693 ms (8/141208) 114 KByte 11.70
> 503=503:0:0:0:0:0:0:0
> [ 1] 2.00-3.00 sec 1.07 MBytes 9.01 Mbits/sec
> 93.970/91.193/96.325/1.796 ms (8/140846) 111 KByte 11.99
> 509=509:0:0:0:0:0:0:0
> [ 2] 2.00-3.00 sec 1.08 MBytes 9.10 Mbits/sec
> 92.843/90.216/96.355/2.040 ms (8/142113) 111 KByte 12.25
> 509=509:0:0:0:0:0:0:0
> [SUM] 2.00-3.00 sec 4.32 MBytes 36.3 Mbits/sec
> 2018=2018:0:0:0:0:0:0:0
> [ 4] 3.00-4.00 sec 1.06 MBytes 8.86 Mbits/sec
> 93.222/89.063/96.104/2.346 ms (9/123027) 96.1 KByte 11.88
> 487=487:0:0:0:0:0:0:0
> [ 3] 3.00-4.00 sec 1.07 MBytes 8.97 Mbits/sec
> 96.277/95.051/97.230/0.767 ms (9/124636) 101 KByte 11.65
> 489=489:0:0:0:0:0:0:0
> [ 1] 3.00-4.00 sec 1.08 MBytes 9.02 Mbits/sec
> 93.899/88.732/96.972/2.737 ms (9/125280) 98.6 KByte 12.01
> 493=493:0:0:0:0:0:0:0
> [ 2] 3.00-4.00 sec 1.07 MBytes 8.97 Mbits/sec
> 92.490/89.862/95.265/1.796 ms (9/124636) 96.6 KByte 12.13
> 493=493:0:0:0:0:0:0:0
> [SUM] 3.00-4.00 sec 4.27 MBytes 35.8 Mbits/sec
> 1962=1962:0:0:0:0:0:0:0
> [ 4] 4.00-5.00 sec 1.07 MBytes 9.00 Mbits/sec
> 92.431/81.888/96.221/4.524 ms (9/124958) 96.8 KByte 12.17
> 498=498:0:0:0:0:0:0:0
> [ 1] 4.00-5.00 sec 1.07 MBytes 8.97 Mbits/sec
> 95.018/93.445/96.200/0.957 ms (9/124636) 99.3 KByte 11.81
> 490=490:0:0:0:0:0:0:0
> [ 2] 4.00-5.00 sec 1.06 MBytes 8.90 Mbits/sec
> 93.874/86.485/95.672/2.810 ms (9/123671) 97.3 KByte 11.86
> 481=481:0:0:0:0:0:0:0
> [ 3] 4.00-5.00 sec 1.08 MBytes 9.09 Mbits/sec
> 95.737/93.881/97.197/0.972 ms (9/126245) 101 KByte 11.87
> 484=484:0:0:0:0:0:0:0
> [SUM] 4.00-5.00 sec 4.29 MBytes 36.0 Mbits/sec
> 1953=1953:0:0:0:0:0:0:0
> [ 4] 5.00-6.00 sec 1.09 MBytes 9.13 Mbits/sec
> 92.908/86.844/95.994/3.012 ms (8/142656) 111 KByte 12.28
> 467=467:0:0:0:0:0:0:0
> [ 3] 5.00-6.00 sec 1.07 MBytes 8.94 Mbits/sec
> 96.593/95.343/97.660/0.876 ms (8/139760) 113 KByte 11.58
> 478=478:0:0:0:0:0:0:0
> [ 1] 5.00-6.00 sec 1.08 MBytes 9.03 Mbits/sec
> 95.021/91.421/97.167/1.893 ms (8/141027) 112 KByte 11.87
> 491=491:0:0:0:0:0:0:0
> [ 2] 5.00-6.00 sec 1.08 MBytes 9.06 Mbits/sec
> 92.162/82.720/97.692/5.060 ms (8/141570) 109 KByte 12.29
> 488=488:0:0:0:0:0:0:0
> [SUM] 5.00-6.00 sec 4.31 MBytes 36.2 Mbits/sec
> 1924=1924:0:0:0:0:0:0:0
> [ 4] 6.00-7.00 sec 1.04 MBytes 8.70 Mbits/sec
> 92.793/85.343/96.967/3.552 ms (9/120775) 93.9 KByte 11.71
> 485=485:0:0:0:0:0:0:0
> [ 2] 6.00-7.00 sec 1.05 MBytes 8.79 Mbits/sec
> 91.679/84.479/96.760/3.975 ms (9/122062) 93.8 KByte 11.98
> 472=472:0:0:0:0:0:0:0
> [ 3] 6.00-7.00 sec 1.06 MBytes 8.88 Mbits/sec
> 96.982/95.933/98.371/0.680 ms (9/123349) 100 KByte 11.45
> 477=477:0:0:0:0:0:0:0
> [ 1] 6.00-7.00 sec 1.05 MBytes 8.80 Mbits/sec
> 94.342/91.660/96.025/1.660 ms (9/122223) 96.7 KByte 11.66
> 494=494:0:0:0:0:0:0:0
> [SUM] 6.00-7.00 sec 4.19 MBytes 35.2 Mbits/sec
> 1928=1928:0:0:0:0:0:0:0
> [ 4] 7.00-8.00 sec 1.10 MBytes 9.25 Mbits/sec
> 92.515/88.182/96.351/2.538 ms (8/144466) 112 KByte 12.49
> 510=510:0:0:0:0:0:0:0
> [ 3] 7.00-8.00 sec 1.09 MBytes 9.13 Mbits/sec
> 96.580/95.737/98.977/1.098 ms (8/142656) 115 KByte 11.82
> 480=480:0:0:0:0:0:0:0
> [ 1] 7.00-8.00 sec 1.10 MBytes 9.21 Mbits/sec
> 95.269/91.719/97.514/2.126 ms (8/143923) 115 KByte 12.09
> 515=515:0:0:0:0:0:0:0
> [ 2] 7.00-8.00 sec 1.11 MBytes 9.29 Mbits/sec
> 90.073/84.700/96.176/4.324 ms (8/145190) 110 KByte 12.90
> 508=508:0:0:0:0:0:0:0
> [SUM] 7.00-8.00 sec 4.40 MBytes 36.9 Mbits/sec
> 2013=2013:0:0:0:0:0:0:0
>
> Bob
>
>>> -----Original Message-----
>>
>>> From: LibreQoS <libreqos-bounces at lists.bufferbloat.net> On Behalf Of
>> Dave Taht
>>
>>> via LibreQoS
>>
>>> Sent: Wednesday, January 4, 2023 12:26 PM
>>
>>> Subject: [LibreQoS] the grinch meets cloudflare's christmas present
>>
>>>
>>
>>> Please try the new, the shiny, the really wonderful test here:
>>
>>>
>> https://urldefense.com/v3/__https://speed.cloudflare.com/__;!!BhdT!iZcFJ8WVU9S
>> [1]
>>
>>>
>> 9zz5t456oxkfObrC5Xb9j5AG8UO3DqD5x4GAJkawZr0iGwEUtF0_09U8mCDnAkrJ9QEMHGbCMKVw$
>> [1]
>>
>>>
>>
>>> I would really appreciate some independent verification of
>>
>>> measurements using this tool. In my brief experiments it appears -
>> as
>>
>>> all the commercial tools to date - to dramatically understate the
>>
>>> bufferbloat, on my LTE, (and my starlink terminal is out being
>>
>>> hacked^H^H^H^H^H^Hworked on, so I can't measure that)
>>
>> [acm]
>>
>> Hi Dave, I made some time to test "cloudflare's christmas present"
>> yesterday.
>>
>> I'm on DOCSIS 3.1 service with 1Gbps Down. The Upstream has a "turbo"
>> mode with 40-50Mbps for the first ~3 sec, then steady-state about
>> 23Mbps.
>>
>> When I saw the ~620Mbps Downstream measurement, I was ready to
>> complain that even the IP-Layer Capacity was grossly underestimated.
>> In addition, the Latency measurements seem very low (as you asserted),
>> although the cloud server was “nearby”.
>>
>> However, I found that Ookla and the ISP-provided measurement were also
>> reporting ~600Mbps! So the cloudflare Downstream capacity (or
>> throughput?) measurement was consistent with others. Our UDPST server
>> was unreachable, otherwise I would have added that measurement, too.
>>
>> The Upstream measurement graph seems to illustrate the “turbo”
>> mode, with the dip after attaining 44.5Mbps.
>>
>> UDPST saturates the uplink and we measure the full 250ms of the
>> Upstream buffer. Cloudflare’s latency measurements don’t even come
>> close.
>>
>> Al
>>
>>
>>
>> Links:
>> ------
>> [1]
>> https://urldefense.com/v3/__https:/speed.cloudflare.com/__;!!BhdT!iZcFJ8WVU9S9zz5t456oxkfObrC5Xb9j5AG8UO3DqD5x4GAJkawZr0iGwEUtF0_09U8mCDnAkrJ9QEMHGbCMKVw$
>> _______________________________________________
>> Rpm mailing list
>> Rpm at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm
More information about the Rpm
mailing list