From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DAD233B2A4; Fri, 6 Jan 2023 15:47:40 -0500 (EST) Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id 25E1D1B25F; Fri, 6 Jan 2023 12:47:40 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com 25E1D1B25F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1673038060; bh=DvvOyY/+oRibEYkdtKwDg4iyhct8J1LBFSI6axH3bqM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=Zes3DE4lxOQL4qkHCnHUOGACilKi6WQlAVkRQcuVcu938GVukQfR//ruxCAgdMIqO 9iTVMCOdHa2nt1Qa1Ygti8qxO7imclXfe261TAw4SLtKPCKxsXrdd9Hrv3zzHdFPeG nd9LG+SE8pbanh0e0FM0zQwRwKuOtiAE7UvjzVvU= MIME-Version: 1.0 Date: Fri, 06 Jan 2023 12:47:40 -0800 From: rjmcmahon To: "MORTON JR., AL" Cc: Dave Taht , bloat , libreqos , Cake List , Dave Taht via Starlink , Rpm , IETF IPPM WG In-Reply-To: References: Message-ID: X-Sender: rjmcmahon@rjmcmahon.com Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sat, 07 Jan 2023 10:10:13 -0500 Subject: Re: [LibreQoS] [Rpm] the grinch meets cloudflare's christmas present X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Jan 2023 20:47:41 -0000 For responsiveness, the bounceback seems reasonable even with upstream competition. Bunch more TCP retries though. [rjmcmahon@ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e --trip-times -i 1 --bounceback -t 3 ------------------------------------------------------------ Client connecting to (**hidden**), TCP port 5001 with pid 111022 (1 flows) Write buffer size: 100 Byte Bursting: 100 Byte writes 10 times every 1.00 second(s) Bounce-back test (size= 100 Byte) (server hold req=0 usecs & tcp_quickack) TOS set to 0x0 and nodelay (Nagle off) TCP window size: 16.0 KByte (default) Event based writes (pending queue watermark at 16384 bytes) ------------------------------------------------------------ [ 1] local *.*.*.86%enp7s0 port 36976 connected with *.*.*.123 port 5001 (prefetch=16384) (bb w/quickack len/hold=100/0) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/9862) (ct=9.90 ms) on 2023-01-06 12:42:18 (PST) [ ID] Interval Transfer Bandwidth BB cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec 10=12.195/9.298/16.457/2.679 ms 0 14K/11327 us 82 rps [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec 10=12.613/9.271/15.489/2.788 ms 0 14K/12165 us 79 rps [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec 10=13.390/9.376/15.986/2.520 ms 0 14K/13164 us 75 rps [ 1] 0.00-3.03 sec 5.86 KBytes 15.8 Kbits/sec 30=12.733/9.271/16.457/2.620 ms 0 14K/15138 us 79 rps [ 1] 0.00-3.03 sec OWD Delays (ms) Cnt=30 To=7.937/4.634/11.327/2.457 From=4.778/4.401/5.350/0.258 Asymmetry=3.166/0.097/6.311/2.318 79 rps [ 1] 0.00-3.03 sec BB8(f)-PDF: bin(w=100us):cnt(30)=93:2,94:3,95:2,97:1,100:1,102:1,105:1,114:2,142:1,143:1,144:2,145:3,146:1,147:1,148:1,151:1,152:1,154:1,155:1,156:1,160:1,165:1 (5.00/95.00/99.7%=93/160/165,Outliers=0,obl/obu=0/0) [rjmcmahon@ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e --trip-times -i 1 --bounceback -t 3 --bounceback-congest=up,4 ------------------------------------------------------------ Client connecting to (**hidden**), TCP port 5001 with pid 111069 (1 flows) Write buffer size: 100 Byte Bursting: 100 Byte writes 10 times every 1.00 second(s) Bounce-back test (size= 100 Byte) (server hold req=0 usecs & tcp_quickack) TOS set to 0x0 and nodelay (Nagle off) TCP window size: 16.0 KByte (default) Event based writes (pending queue watermark at 16384 bytes) ------------------------------------------------------------ [ 2] local *.*.*.85%enp4s0 port 38342 connected with *.*.*.123 port 5001 (prefetch=16384) (trip-times) (sock=3) (qack) (icwnd/mss/irtt=14/1448/10613) (ct=10.66 ms) on 2023-01-06 12:42:36 (PST) [ 1] local *.*.*.85%enp4s0 port 38360 connected with *.*.*.123 port 5001 (prefetch=16384) (bb w/quickack len/hold=100/0) (trip-times) (sock=4) (icwnd/mss/irtt=14/1448/14901) (ct=14.96 ms) on 2023-01-06 12:42:36 (PST) [ 3] local *.*.*.85%enp4s0 port 38386 connected with *.*.*.123 port 5001 (prefetch=16384) (trip-times) (sock=7) (qack) (icwnd/mss/irtt=14/1448/15295) (ct=15.31 ms) on 2023-01-06 12:42:36 (PST) [ 4] local *.*.*.85%enp4s0 port 38348 connected with *.*.*.123 port 5001 (prefetch=16384) (trip-times) (sock=5) (qack) (icwnd/mss/irtt=14/1448/14901) (ct=14.95 ms) on 2023-01-06 12:42:36 (PST) [ 5] local *.*.*.85%enp4s0 port 38372 connected with *.*.*.123 port 5001 (prefetch=16384) (trip-times) (sock=6) (qack) (icwnd/mss/irtt=14/1448/15371) (ct=15.42 ms) on 2023-01-06 12:42:36 (PST) [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr [ 3] 0.00-1.00 sec 1.29 MBytes 10.8 Mbits/sec 13502/0 115 28K/22594(904) us 59.76 [ 4] 0.00-1.00 sec 1.63 MBytes 13.6 Mbits/sec 17048/0 140 42K/22728(568) us 75.01 [ ID] Interval Transfer Bandwidth BB cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS [ 1] 0.00-1.00 sec 1.95 KBytes 16.0 Kbits/sec 10=76.140/17.224/123.195/43.168 ms 0 14K/68136 us 13 rps [ 5] 0.00-1.00 sec 1.04 MBytes 8.72 Mbits/sec 10893/0 82 25K/23400(644) us 46.55 [SUM] 0.00-1.00 sec 3.95 MBytes 33.2 Mbits/sec 41443/0 337 [ 2] 0.00-1.00 sec 1.10 MBytes 9.25 Mbits/sec 11566/0 77 22K/23557(432) us 49.10 [ 3] 1.00-2.00 sec 1.24 MBytes 10.4 Mbits/sec 13037/0 20 28K/14427(503) us 90.37 [ 4] 1.00-2.00 sec 1.43 MBytes 12.0 Mbits/sec 14954/0 31 12K/13348(407) us 112 [ 1] 1.00-2.00 sec 1.95 KBytes 16.0 Kbits/sec 10=14.581/10.801/20.356/3.599 ms 0 14K/27791 us 69 rps [ 5] 1.00-2.00 sec 1.26 MBytes 10.6 Mbits/sec 13191/0 16 12K/14749(675) us 89.44 [SUM] 1.00-2.00 sec 3.93 MBytes 32.9 Mbits/sec 41182/0 67 [ 2] 1.00-2.00 sec 1000 KBytes 8.19 Mbits/sec 10237/0 13 19K/14467(1068) us 70.76 [ 3] 2.00-3.00 sec 1.33 MBytes 11.2 Mbits/sec 13994/0 4 24K/20749(495) us 67.44 [ 4] 2.00-3.00 sec 1.20 MBytes 10.1 Mbits/sec 12615/0 3 31K/20877(718) us 60.43 [ 1] 2.00-3.00 sec 1.95 KBytes 16.0 Kbits/sec 10=11.298/9.407/14.245/1.330 ms 0 14K/15474 us 89 rps [ 5] 2.00-3.00 sec 1.08 MBytes 9.03 Mbits/sec 11284/0 3 28K/21031(430) us 53.65 [SUM] 2.00-3.00 sec 3.61 MBytes 30.3 Mbits/sec 37893/0 10 [ 2] 2.00-3.00 sec 1.29 MBytes 10.8 Mbits/sec 13492/0 3 29K/20409(688) us 66.11 [ 3] 0.00-3.03 sec 3.87 MBytes 10.7 Mbits/sec 40534/0 139 25K/20645(557) us 64.85 [ 5] 0.00-3.03 sec 3.37 MBytes 9.35 Mbits/sec 35369/0 101 29K/20489(668) us 57.02 [ 4] 0.00-3.03 sec 4.26 MBytes 11.8 Mbits/sec 44618/0 174 32K/21240(961) us 69.40 [ 2] 0.00-3.03 sec 3.37 MBytes 9.31 Mbits/sec 35296/0 94 19K/21504(948) us 54.13 [ 1] 0.00-3.14 sec 7.81 KBytes 20.4 Kbits/sec 40=28.332/5.611/123.195/34.940 ms 0 14K/14000 us 35 rps [ 1] 0.00-3.14 sec OWD Delays (ms) Cnt=40 To=23.730/1.110/118.744/34.957 From=4.567/4.356/5.171/0.141 Asymmetry=19.332/0.189/114.294/34.869 35 rps [ 1] 0.00-3.14 sec BB8(f)-PDF: bin(w=100us):cnt(40)=57:1,94:2,95:2,96:2,98:1,101:1,106:1,109:2,111:1,112:2,113:1,115:1,118:1,119:2,143:2,145:1,146:1,152:1,158:1,173:1,176:1,194:1,195:1,204:1,205:1,274:1,554:1,790:1,925:1,1125:1,1126:1,1225:1,1232:1 (5.00/95.00/99.7%=94/1225/1232,Outliers=0,obl/obu=0/0) [SUM] 0.00-3.11 sec 11.5 MBytes 31.0 Mbits/sec 120521/0 414 [ CT] final connect times (min/avg/max/stdev) = 10.661/14.261/15.423/2023.369 ms (tot/err) = 5/0 > Some thoughts are not to use UDP for testing here. Also, these speed > tests have little to no information for network engineers about what's > going on. Iperf 2 may better assist network engineers but then I'm > biased ;) > > Running iperf 2 https://sourceforge.net/projects/iperf2/ with > --trip-times. Though the sampling and central limit theorem averaging > is hiding the real distributions (use --histograms to get those) > > Below are 4 parallel TCP streams from my home to one of my servers in > the cloud. First where TCP is limited per CCA. Second is source side > write rate limiting. Things to note: > > o) connect times for both at 10-15 ms > o) multiple TCP retries on a few rites - one case is 4 with 5 writes. > Source side pacing eliminates retries > o) Fairness with CCA isn't great but quite good with source side write > pacing > o) Queue depth with CCA is about 150 Kbytes about 100K byte with > source side pacing > o) min write to read is about 80 ms for both > o) max is 220 ms vs 97 ms > o) stdev for CCA write/read is 30 ms vs 3 ms > o) TCP RTT is 20ms w/CCA and 90 ms with ssp - seems odd here as > TCP_QUICACK and TCP_NODELAY are both enabled. > > [ CT] final connect times (min/avg/max/stdev) = > 10.326/13.522/14.986/2150.329 ms (tot/err) = 4/0 > [rjmcmahon@ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e > --trip-times -i 1 -P 4 -t 10 -w 4m --tcp-quickack -N > ------------------------------------------------------------ > Client connecting to (**hidden**), TCP port 5001 with pid 107678 (4 > flows) > Write buffer size: 131072 Byte > TOS set to 0x0 and nodelay (Nagle off) > TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte) > Event based writes (pending queue watermark at 16384 bytes) > ------------------------------------------------------------ > [ 1] local *.*.*.85%enp4s0 port 42480 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=3) (qack) > (icwnd/mss/irtt=14/1448/10534) (ct=10.63 ms) on 2023-01-06 12:17:56 > (PST) > [ 4] local *.*.*.85%enp4s0 port 42488 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=5) (qack) > (icwnd/mss/irtt=14/1448/14023) (ct=14.08 ms) on 2023-01-06 12:17:56 > (PST) > [ 3] local *.*.*.85%enp4s0 port 42502 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=6) (qack) > (icwnd/mss/irtt=14/1448/14642) (ct=14.70 ms) on 2023-01-06 12:17:56 > (PST) > [ 2] local *.*.*.85%enp4s0 port 42484 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=4) (qack) > (icwnd/mss/irtt=14/1448/14728) (ct=14.79 ms) on 2023-01-06 12:17:56 > (PST) > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT(var) NetPwr > ... > [ 4] 4.00-5.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 3 > 29K/21088(1142) us 68.37 > [ 2] 4.00-5.00 sec 1.62 MBytes 13.6 Mbits/sec 13/0 2 > 31K/19284(612) us 88.36 > [ 1] 4.00-5.00 sec 896 KBytes 7.34 Mbits/sec 7/0 5 > 16K/18996(658) us 48.30 > [ 3] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 5 > 18K/18133(208) us 57.83 > [SUM] 4.00-5.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 15 > [ 4] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 4 > 29K/14717(489) us 89.06 > [ 1] 5.00-6.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 4 > 16K/15874(408) us 66.06 > [ 3] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 4 > 16K/15826(382) us 74.54 > [ 2] 5.00-6.00 sec 1.50 MBytes 12.6 Mbits/sec 12/0 6 > 9K/14878(557) us 106 > [SUM] 5.00-6.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 18 > [ 4] 6.00-7.00 sec 1.75 MBytes 14.7 Mbits/sec 14/0 4 > 25K/15472(496) us 119 > [ 2] 6.00-7.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 2 > 26K/16417(427) us 63.87 > [ 1] 6.00-7.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 5 > 16K/16268(679) us 80.57 > [ 3] 6.00-7.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 6 > 15K/16629(799) us 63.06 > [SUM] 6.00-7.00 sec 5.00 MBytes 41.9 Mbits/sec 40/0 17 > [ 4] 7.00-8.00 sec 1.75 MBytes 14.7 Mbits/sec 14/0 4 > 22K/13986(519) us 131 > [ 1] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 4 > 16K/12679(377) us 93.04 > [ 3] 7.00-8.00 sec 896 KBytes 7.34 Mbits/sec 7/0 5 > 14K/12971(367) us 70.74 > [ 2] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 6 > 15K/14740(779) us 80.03 > [SUM] 7.00-8.00 sec 4.88 MBytes 40.9 Mbits/sec 39/0 19 > > [root@bobcat iperf2-code]# iperf -s -i 1 -e --hide-ips -w 4m > ------------------------------------------------------------ > Server listening on TCP port 5001 with pid 233615 > Read buffer size: 128 KByte (Dist bin width=16.0 KByte) > TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte) > ------------------------------------------------------------ > [ 1] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 42480 (trip-times) (sock=4) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11636) on 2023-01-06 12:17:56 (PST) > [ 2] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 42502 (trip-times) (sock=5) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11898) on 2023-01-06 12:17:56 (PST) > [ 3] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 42484 (trip-times) (sock=6) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11938) on 2023-01-06 12:17:56 (PST) > [ 4] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 42488 (trip-times) (sock=7) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11919) on 2023-01-06 12:17:56 (PST) > [ ID] Interval Transfer Bandwidth Burst Latency > avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist > ... > [ 2] 4.00-5.00 sec 1.06 MBytes 8.86 Mbits/sec > 129.819/90.391/186.075/31.346 ms (9/123080) 154 KByte 8.532803 > 467=461:6:0:0:0:0:0:0 > [ 3] 4.00-5.00 sec 1.52 MBytes 12.8 Mbits/sec > 103.552/82.653/169.274/28.382 ms (12/132854) 149 KByte 15.40 > 646=643:1:2:0:0:0:0:0 > [ 4] 4.00-5.00 sec 1.39 MBytes 11.6 Mbits/sec > 107.503/66.843/143.038/24.269 ms (11/132294) 149 KByte 13.54 > 619=617:1:1:0:0:0:0:0 > [ 1] 4.00-5.00 sec 988 KBytes 8.10 Mbits/sec > 141.389/119.961/178.785/18.812 ms (7/144593) 170 KByte 7.158641 > 409=404:5:0:0:0:0:0:0 > [SUM] 4.00-5.00 sec 4.93 MBytes 41.4 Mbits/sec > 2141=2125:13:3:0:0:0:0:0 > [ 4] 5.00-6.00 sec 1.29 MBytes 10.8 Mbits/sec > 118.943/86.253/176.128/31.248 ms (10/135098) 164 KByte 11.36 > 511=506:2:3:0:0:0:0:0 > [ 2] 5.00-6.00 sec 1.09 MBytes 9.17 Mbits/sec > 139.821/102.418/218.875/40.422 ms (9/127424) 148 KByte 8.202049 > 487=484:2:1:0:0:0:0:0 > [ 3] 5.00-6.00 sec 1.51 MBytes 12.6 Mbits/sec > 102.146/77.085/140.893/18.441 ms (13/121520) 151 KByte 15.47 > 640=636:1:3:0:0:0:0:0 > [ 1] 5.00-6.00 sec 981 KBytes 8.04 Mbits/sec > 161.901/105.582/219.931/36.260 ms (8/125614) 134 KByte 6.206944 > 415=413:2:0:0:0:0:0:0 > [SUM] 5.00-6.00 sec 4.85 MBytes 40.7 Mbits/sec > 2053=2039:7:7:0:0:0:0:0 > [ 4] 6.00-7.00 sec 1.74 MBytes 14.6 Mbits/sec > 88.846/74.297/101.859/7.118 ms (14/130526) 156 KByte 20.57 > 711=707:3:1:0:0:0:0:0 > [ 1] 6.00-7.00 sec 1.22 MBytes 10.2 Mbits/sec > 120.639/100.257/157.567/21.770 ms (10/127568) 157 KByte 10.57 > 494=488:5:1:0:0:0:0:0 > [ 2] 6.00-7.00 sec 1015 KBytes 8.32 Mbits/sec > 144.632/124.368/171.349/16.597 ms (8/129958) 143 KByte 7.188321 > 408=403:5:0:0:0:0:0:0 > [ 3] 6.00-7.00 sec 1.02 MBytes 8.60 Mbits/sec > 143.516/102.322/173.001/24.089 ms (8/134302) 146 KByte 7.486359 > 484=480:4:0:0:0:0:0:0 > [SUM] 6.00-7.00 sec 4.98 MBytes 41.7 Mbits/sec > 2097=2078:17:2:0:0:0:0:0 > [ 4] 7.00-8.00 sec 1.77 MBytes 14.9 Mbits/sec > 85.406/65.797/103.418/12.609 ms (14/132595) 153 KByte 21.74 > 692=687:2:3:0:0:0:0:0 > [ 2] 7.00-8.00 sec 957 KBytes 7.84 Mbits/sec > 153.936/131.452/191.464/19.361 ms (7/140042) 160 KByte 6.368199 > 429=425:4:0:0:0:0:0:0 > [ 1] 7.00-8.00 sec 1.13 MBytes 9.44 Mbits/sec > 131.146/109.737/166.774/22.035 ms (9/131124) 146 KByte 8.998528 > 520=516:4:0:0:0:0:0:0 > [ 3] 7.00-8.00 sec 1.13 MBytes 9.51 Mbits/sec > 126.512/88.404/220.175/42.237 ms (9/132089) 172 KByte 9.396784 > 527=524:1:2:0:0:0:0:0 > [SUM] 7.00-8.00 sec 4.96 MBytes 41.6 Mbits/sec > 2168=2152:11:5:0:0:0:0:0 > > With source side rate limiting to 9 mb/s per stream. > > [rjmcmahon@ryzen3950 iperf2-code]$ iperf -c *** --hide-ips -e > --trip-times -i 1 -P 4 -t 10 -w 4m --tcp-quickack -N -b 9m > ------------------------------------------------------------ > Client connecting to (**hidden**), TCP port 5001 with pid 108884 (4 > flows) > Write buffer size: 131072 Byte > TOS set to 0x0 and nodelay (Nagle off) > TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte) > Event based writes (pending queue watermark at 16384 bytes) > ------------------------------------------------------------ > [ 1] local *.*.*.85%enp4s0 port 46448 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=3) (qack) > (icwnd/mss/irtt=14/1448/10666) (ct=10.70 ms) on 2023-01-06 12:27:45 > (PST) > [ 3] local *.*.*.85%enp4s0 port 46460 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=6) (qack) > (icwnd/mss/irtt=14/1448/16499) (ct=16.54 ms) on 2023-01-06 12:27:45 > (PST) > [ 2] local *.*.*.85%enp4s0 port 46454 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=4) (qack) > (icwnd/mss/irtt=14/1448/16580) (ct=16.86 ms) on 2023-01-06 12:27:45 > (PST) > [ 4] local *.*.*.85%enp4s0 port 46458 connected with *.*.*.123 port > 5001 (prefetch=16384) (trip-times) (sock=5) (qack) > (icwnd/mss/irtt=14/1448/16802) (ct=16.83 ms) on 2023-01-06 12:27:45 > (PST) > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT(var) NetPwr > ... > [ 2] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 134K/88055(12329) us 11.91 > [ 4] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 132K/74867(11755) us 14.01 > [ 1] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 134K/89101(13134) us 11.77 > [ 3] 4.00-5.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 131K/91451(11938) us 11.47 > [SUM] 4.00-5.00 sec 4.00 MBytes 33.6 Mbits/sec 32/0 0 > [ 2] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 134K/85135(14580) us 13.86 > [ 4] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 132K/85124(15654) us 13.86 > [ 1] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 134K/91336(11335) us 12.92 > [ 3] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 131K/89185(13499) us 13.23 > [SUM] 5.00-6.00 sec 4.50 MBytes 37.7 Mbits/sec 36/0 0 > [ 2] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 134K/85687(13489) us 13.77 > [ 4] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 132K/82803(13001) us 14.25 > [ 1] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 134K/86869(15186) us 13.58 > [ 3] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 > 131K/91447(12515) us 12.90 > [SUM] 6.00-7.00 sec 4.50 MBytes 37.7 Mbits/sec 36/0 0 > [ 2] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 134K/81814(13168) us 12.82 > [ 4] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 132K/89008(13283) us 11.78 > [ 1] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 134K/89494(12151) us 11.72 > [ 3] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec 8/0 0 > 131K/91083(12797) us 11.51 > [SUM] 7.00-8.00 sec 4.00 MBytes 33.6 Mbits/sec 32/0 0 > > [root@bobcat iperf2-code]# iperf -s -i 1 -e --hide-ips -w 4m > ------------------------------------------------------------ > Server listening on TCP port 5001 with pid 233981 > Read buffer size: 128 KByte (Dist bin width=16.0 KByte) > TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte) > ------------------------------------------------------------ > [ 1] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 46448 (trip-times) (sock=4) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11987) on 2023-01-06 12:27:45 (PST) > [ 2] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 46454 (trip-times) (sock=5) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11132) on 2023-01-06 12:27:45 (PST) > [ 3] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 46460 (trip-times) (sock=6) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/11097) on 2023-01-06 12:27:45 (PST) > [ 4] local *.*.*.123%eth0 port 5001 connected with *.*.*.171 port > 46458 (trip-times) (sock=7) (peer 2.1.9-master) (qack) > (icwnd/mss/irtt=14/1448/17823) on 2023-01-06 12:27:45 (PST) > [ ID] Interval Transfer Bandwidth Burst Latency > avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist > [ 4] 0.00-1.00 sec 1.09 MBytes 9.15 Mbits/sec > 93.383/90.103/95.661/2.232 ms (8/143028) 128 KByte 12.25 > 451=451:0:0:0:0:0:0:0 > [ 3] 0.00-1.00 sec 1.08 MBytes 9.06 Mbits/sec > 96.834/95.229/102.645/2.442 ms (8/141580) 131 KByte 11.70 > 472=472:0:0:0:0:0:0:0 > [ 1] 0.00-1.00 sec 1.10 MBytes 9.19 Mbits/sec > 95.183/92.623/97.579/1.431 ms (8/143571) 131 KByte 12.07 > 495=495:0:0:0:0:0:0:0 > [ 2] 0.00-1.00 sec 1.09 MBytes 9.15 Mbits/sec > 89.317/84.865/94.906/3.674 ms (8/143028) 122 KByte 12.81 > 489=489:0:0:0:0:0:0:0 > [ ID] Interval Transfer Bandwidth Reads=Dist > [SUM] 0.00-1.00 sec 4.36 MBytes 36.6 Mbits/sec > 1907=1907:0:0:0:0:0:0:0 > [ 4] 1.00-2.00 sec 1.07 MBytes 8.95 Mbits/sec > 92.649/89.987/95.036/1.828 ms (9/124314) 96.5 KByte 12.08 > 492=492:0:0:0:0:0:0:0 > [ 3] 1.00-2.00 sec 1.06 MBytes 8.93 Mbits/sec > 96.305/95.647/96.794/0.432 ms (9/123992) 100 KByte 11.59 > 480=480:0:0:0:0:0:0:0 > [ 1] 1.00-2.00 sec 1.06 MBytes 8.89 Mbits/sec > 92.578/90.866/94.145/1.371 ms (9/123510) 95.8 KByte 12.01 > 513=513:0:0:0:0:0:0:0 > [ 2] 1.00-2.00 sec 1.07 MBytes 8.96 Mbits/sec > 90.767/87.984/94.352/1.944 ms (9/124475) 94.7 KByte 12.34 > 489=489:0:0:0:0:0:0:0 > [SUM] 1.00-2.00 sec 4.26 MBytes 35.7 Mbits/sec > 1974=1974:0:0:0:0:0:0:0 > [ 4] 2.00-3.00 sec 1.09 MBytes 9.13 Mbits/sec > 93.977/91.795/96.561/1.693 ms (8/142656) 112 KByte 12.14 > 497=497:0:0:0:0:0:0:0 > [ 3] 2.00-3.00 sec 1.08 MBytes 9.04 Mbits/sec > 96.544/95.815/97.798/0.693 ms (8/141208) 114 KByte 11.70 > 503=503:0:0:0:0:0:0:0 > [ 1] 2.00-3.00 sec 1.07 MBytes 9.01 Mbits/sec > 93.970/91.193/96.325/1.796 ms (8/140846) 111 KByte 11.99 > 509=509:0:0:0:0:0:0:0 > [ 2] 2.00-3.00 sec 1.08 MBytes 9.10 Mbits/sec > 92.843/90.216/96.355/2.040 ms (8/142113) 111 KByte 12.25 > 509=509:0:0:0:0:0:0:0 > [SUM] 2.00-3.00 sec 4.32 MBytes 36.3 Mbits/sec > 2018=2018:0:0:0:0:0:0:0 > [ 4] 3.00-4.00 sec 1.06 MBytes 8.86 Mbits/sec > 93.222/89.063/96.104/2.346 ms (9/123027) 96.1 KByte 11.88 > 487=487:0:0:0:0:0:0:0 > [ 3] 3.00-4.00 sec 1.07 MBytes 8.97 Mbits/sec > 96.277/95.051/97.230/0.767 ms (9/124636) 101 KByte 11.65 > 489=489:0:0:0:0:0:0:0 > [ 1] 3.00-4.00 sec 1.08 MBytes 9.02 Mbits/sec > 93.899/88.732/96.972/2.737 ms (9/125280) 98.6 KByte 12.01 > 493=493:0:0:0:0:0:0:0 > [ 2] 3.00-4.00 sec 1.07 MBytes 8.97 Mbits/sec > 92.490/89.862/95.265/1.796 ms (9/124636) 96.6 KByte 12.13 > 493=493:0:0:0:0:0:0:0 > [SUM] 3.00-4.00 sec 4.27 MBytes 35.8 Mbits/sec > 1962=1962:0:0:0:0:0:0:0 > [ 4] 4.00-5.00 sec 1.07 MBytes 9.00 Mbits/sec > 92.431/81.888/96.221/4.524 ms (9/124958) 96.8 KByte 12.17 > 498=498:0:0:0:0:0:0:0 > [ 1] 4.00-5.00 sec 1.07 MBytes 8.97 Mbits/sec > 95.018/93.445/96.200/0.957 ms (9/124636) 99.3 KByte 11.81 > 490=490:0:0:0:0:0:0:0 > [ 2] 4.00-5.00 sec 1.06 MBytes 8.90 Mbits/sec > 93.874/86.485/95.672/2.810 ms (9/123671) 97.3 KByte 11.86 > 481=481:0:0:0:0:0:0:0 > [ 3] 4.00-5.00 sec 1.08 MBytes 9.09 Mbits/sec > 95.737/93.881/97.197/0.972 ms (9/126245) 101 KByte 11.87 > 484=484:0:0:0:0:0:0:0 > [SUM] 4.00-5.00 sec 4.29 MBytes 36.0 Mbits/sec > 1953=1953:0:0:0:0:0:0:0 > [ 4] 5.00-6.00 sec 1.09 MBytes 9.13 Mbits/sec > 92.908/86.844/95.994/3.012 ms (8/142656) 111 KByte 12.28 > 467=467:0:0:0:0:0:0:0 > [ 3] 5.00-6.00 sec 1.07 MBytes 8.94 Mbits/sec > 96.593/95.343/97.660/0.876 ms (8/139760) 113 KByte 11.58 > 478=478:0:0:0:0:0:0:0 > [ 1] 5.00-6.00 sec 1.08 MBytes 9.03 Mbits/sec > 95.021/91.421/97.167/1.893 ms (8/141027) 112 KByte 11.87 > 491=491:0:0:0:0:0:0:0 > [ 2] 5.00-6.00 sec 1.08 MBytes 9.06 Mbits/sec > 92.162/82.720/97.692/5.060 ms (8/141570) 109 KByte 12.29 > 488=488:0:0:0:0:0:0:0 > [SUM] 5.00-6.00 sec 4.31 MBytes 36.2 Mbits/sec > 1924=1924:0:0:0:0:0:0:0 > [ 4] 6.00-7.00 sec 1.04 MBytes 8.70 Mbits/sec > 92.793/85.343/96.967/3.552 ms (9/120775) 93.9 KByte 11.71 > 485=485:0:0:0:0:0:0:0 > [ 2] 6.00-7.00 sec 1.05 MBytes 8.79 Mbits/sec > 91.679/84.479/96.760/3.975 ms (9/122062) 93.8 KByte 11.98 > 472=472:0:0:0:0:0:0:0 > [ 3] 6.00-7.00 sec 1.06 MBytes 8.88 Mbits/sec > 96.982/95.933/98.371/0.680 ms (9/123349) 100 KByte 11.45 > 477=477:0:0:0:0:0:0:0 > [ 1] 6.00-7.00 sec 1.05 MBytes 8.80 Mbits/sec > 94.342/91.660/96.025/1.660 ms (9/122223) 96.7 KByte 11.66 > 494=494:0:0:0:0:0:0:0 > [SUM] 6.00-7.00 sec 4.19 MBytes 35.2 Mbits/sec > 1928=1928:0:0:0:0:0:0:0 > [ 4] 7.00-8.00 sec 1.10 MBytes 9.25 Mbits/sec > 92.515/88.182/96.351/2.538 ms (8/144466) 112 KByte 12.49 > 510=510:0:0:0:0:0:0:0 > [ 3] 7.00-8.00 sec 1.09 MBytes 9.13 Mbits/sec > 96.580/95.737/98.977/1.098 ms (8/142656) 115 KByte 11.82 > 480=480:0:0:0:0:0:0:0 > [ 1] 7.00-8.00 sec 1.10 MBytes 9.21 Mbits/sec > 95.269/91.719/97.514/2.126 ms (8/143923) 115 KByte 12.09 > 515=515:0:0:0:0:0:0:0 > [ 2] 7.00-8.00 sec 1.11 MBytes 9.29 Mbits/sec > 90.073/84.700/96.176/4.324 ms (8/145190) 110 KByte 12.90 > 508=508:0:0:0:0:0:0:0 > [SUM] 7.00-8.00 sec 4.40 MBytes 36.9 Mbits/sec > 2013=2013:0:0:0:0:0:0:0 > > Bob > >>> -----Original Message----- >> >>> From: LibreQoS On Behalf Of >> Dave Taht >> >>> via LibreQoS >> >>> Sent: Wednesday, January 4, 2023 12:26 PM >> >>> Subject: [LibreQoS] the grinch meets cloudflare's christmas present >> >>> >> >>> Please try the new, the shiny, the really wonderful test here: >> >>> >> https://urldefense.com/v3/__https://speed.cloudflare.com/__;!!BhdT!iZcFJ8WVU9S >> [1] >> >>> >> 9zz5t456oxkfObrC5Xb9j5AG8UO3DqD5x4GAJkawZr0iGwEUtF0_09U8mCDnAkrJ9QEMHGbCMKVw$ >> [1] >> >>> >> >>> I would really appreciate some independent verification of >> >>> measurements using this tool. In my brief experiments it appears - >> as >> >>> all the commercial tools to date - to dramatically understate the >> >>> bufferbloat, on my LTE, (and my starlink terminal is out being >> >>> hacked^H^H^H^H^H^Hworked on, so I can't measure that) >> >> [acm] >> >> Hi Dave, I made some time to test "cloudflare's christmas present" >> yesterday. >> >> I'm on DOCSIS 3.1 service with 1Gbps Down. The Upstream has a "turbo" >> mode with 40-50Mbps for the first ~3 sec, then steady-state about >> 23Mbps. >> >> When I saw the ~620Mbps Downstream measurement, I was ready to >> complain that even the IP-Layer Capacity was grossly underestimated. >> In addition, the Latency measurements seem very low (as you asserted), >> although the cloud server was “nearby”. >> >> However, I found that Ookla and the ISP-provided measurement were also >> reporting ~600Mbps! So the cloudflare Downstream capacity (or >> throughput?) measurement was consistent with others. Our UDPST server >> was unreachable, otherwise I would have added that measurement, too. >> >> The Upstream measurement graph seems to illustrate the “turbo” >> mode, with the dip after attaining 44.5Mbps. >> >> UDPST saturates the uplink and we measure the full 250ms of the >> Upstream buffer. Cloudflare’s latency measurements don’t even come >> close. >> >> Al >> >> >> >> Links: >> ------ >> [1] >> https://urldefense.com/v3/__https:/speed.cloudflare.com/__;!!BhdT!iZcFJ8WVU9S9zz5t456oxkfObrC5Xb9j5AG8UO3DqD5x4GAJkawZr0iGwEUtF0_09U8mCDnAkrJ9QEMHGbCMKVw$ >> _______________________________________________ >> Rpm mailing list >> Rpm@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm