From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 85F953B29E for ; Tue, 6 Oct 2020 20:10:18 -0400 (EDT) Received: by mail-ed1-x543.google.com with SMTP id l17so267643edq.12 for ; Tue, 06 Oct 2020 17:10:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6ePTMSHlatt/J5IrMricbkuCEBUrdE4POGFeJMjPO6w=; b=dUxLbKwjL/mPQh0CK9Zn+61btaDiAp5rl+3lSQQEYzByjUTUtleP+/65rt8CwyhfdC /GS4NBRlvm7T9fbkYKuyiQxtVHcejwP/UTGypBLrWF5LCuFjjXzdtPIbbaT0kVbSiYFd 86GHhuDaRwHxBnYEP2yHbieIzK6xjNzvWxqdM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6ePTMSHlatt/J5IrMricbkuCEBUrdE4POGFeJMjPO6w=; b=c+JLXErz5L/WdHMKrBuPICn2EA4wWqaT/iRRC6JXccaQOF3B9BpQUc1+AFqbH1Pc4M UexqP1NjGJ6jkoX7fsRBcfKVCahl0M8wBMBwL4awz18m7w/HMYk719DA8niDPr0BRCgA h2SlqTevPi/L8T4tvoKg3FvHNnEtLX0eQfTpfYyJ49VPOur+rOVT40I4hoV8eCbZL4OE NMOxtzng/4rI4dNqgH6zM9fhhyV1EDOsmdfkTZW6k2mz2+IdjNNQImXUmU4fYu8Ild9R hztO6w9trxQpEbw1BzNFYskvBw0KIk1rDJnX9Anqx1L5UOyIGAjRIpo6mcfYWpk8JyCi B/dA== X-Gm-Message-State: AOAM532TxrVUPyip7H0h7EIePu/KMwzZxuCAata6wLDnkquGKbjacav/ jD9NdO2ka4gdk6RDrT6QTrb3ibG7J6GydhFZGrkHNFN6JVO8gA== X-Google-Smtp-Source: ABdhPJxIYv4/UTkaRuLcxkJkEQiegvB3xD/gzrXBXx7fk2AQcKE0oBRgATHGB2vu/EwF+1K6DPgxQ25ZzcqvIeK1RUs= X-Received: by 2002:a05:6402:1779:: with SMTP id da25mr713258edb.60.1602029417155; Tue, 06 Oct 2020 17:10:17 -0700 (PDT) MIME-Version: 1.0 References: <87d01vfue4.fsf@toke.dk> <87a6wzfrro.fsf@toke.dk> In-Reply-To: From: Bob McMahon Date: Tue, 6 Oct 2020 17:10:06 -0700 Message-ID: To: Michael Welzl Cc: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= , Make-Wifi-fast Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="0000000000005d42a405b1098a4c" Subject: Re: [Make-wifi-fast] Where is the bloat in WiFi? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Oct 2020 00:10:18 -0000 --0000000000005d42a405b1098a4c Content-Type: multipart/alternative; boundary="00000000000054b17305b1098a91" --00000000000054b17305b1098a91 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable You might want to try iperf 2.0.14 --trip-times option to measure your links. Below a run with a PC w/a 1G NIC connected to the 1G cable MODEM no WiFi, i.e. wired, full duplex socket. The uplink tcp write latencies are larger than one second. Downlinks just under 100 ms. Slow down the uplink traffic to less than the congestion point and write to read latency drops to ~65 ms. I think this suggests my CABLE MODEM is full of bloat. UPLINK ONLY -------------------- Server: [rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr ------------------------------------------------------------ Server listening on TCP port 5001 with pid 25408 Read buffer size: 128 KByte (Dist bin width=3D16.0 KByte) TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58158 (trip-times) (MSS=3D1448) (peer 2.0.14-alpha) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=3DDist [ 4] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 59.680/51.222/96.282/34.950 ms (10/131072) 80.9 KByte 21.96 305=3D305:0:0:0:0:0:0:0 [ 4] 1.00-2.00 sec 1.12 MBytes 9.44 Mbits/sec 65.218/51.951/87.895/10.877 ms (9/131072) 79.6 KByte 18.09 262=3D262:0:0:0:0:0:0:0 [ 4] 2.00-3.00 sec 1.25 MBytes 10.5 Mbits/sec 62.479/51.170/67.962/ 5.189 ms (10/131072) 76.3 KByte 20.98 289=3D289:0:0:0:0:0:0:0 [ 4] 3.00-4.00 sec 1.12 MBytes 9.44 Mbits/sec 62.861/52.478/68.339/ 5.212 ms (9/131072) 76.7 KByte 18.77 262=3D262:0:0:0:0:0:0:0 [ 4] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec 64.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23 292=3D292:0:0:0:0:0:0:0 [ 4] 5.00-6.00 sec 1.14 MBytes 9.60 Mbits/sec 64.618/50.914/68.277/ 5.696 ms (9/133334) 80.2 KByte 18.57 259=3D259:0:0:0:0:0:0:0 [ 4] 6.00-7.00 sec 1.23 MBytes 10.3 Mbits/sec 66.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29 279=3D279:0:0:0:0:0:0:0 [ 4] 7.00-8.00 sec 1.16 MBytes 9.70 Mbits/sec 68.954/51.902/100.783/15.733 ms (9/134782) 86.6 KByte 17.59 275=3D274:1:0:0:0:0:0:0 [ 4] 8.00-9.00 sec 1.22 MBytes 10.2 Mbits/sec 63.855/51.386/69.919/ 5.284 ms (9/141851) 84.4 KByte 19.99 283=3D283:0:0:0:0:0:0:0 [ 4] 9.00-10.00 sec 1.18 MBytes 9.86 Mbits/sec 67.975/50.996/88.151/11.318 ms (10/123252) 78.0 KByte 18.13 275=3D275:0:0:0:0:0:0:0 [ 4] 10.00-10.14 sec 205 KBytes 11.7 Mbits/sec 99.865/55.342/77.370/ 7.319 ms (2/104962) 97.6 KByte 14.61 43=3D43:0:0:0:0:0:0:0 [ 4] 0.00-10.14 sec 12.1 MBytes 10.0 Mbits/sec 59.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04 2824=3D2823:1:0:0:0:0:0:0 Client: [root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times -Z bbr -b 10m -e ------------------------------------------------------------ Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 48006 (1 flows) Write buffer size: 128 KByte TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.35%enp2s0 port 58160 connected with 45.33.58.123 port 5001 (trip-times) (MSS=3D1448) (ct=3D13.05 ms) [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 3] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 11/0 0 135K/49251 us 26.61 [ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 141K/24791 us 52.87 [ 3] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 135K/24973 us 47.24 [ 3] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 158K/25021 us 52.38 [ 3] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 141K/21247 us 55.52 [ 3] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 144K/23535 us 55.69 [ 3] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 135K/27130 us 43.48 [ 3] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 135K/24813 us 52.82 [ 3] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 155K/22697 us 51.97 [ 3] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 124K/21954 us 59.70 [ 3] 10.00-10.07 sec 256 KBytes 31.6 Mbits/sec 2/0 0 124K/21954 us 179.64 [ 3] 0.00-10.07 sec 12.3 MBytes 10.2 Mbits/sec 99/0 0 124K/21954 us 58.12 FULL DUPLEX: --------------------- Server: [rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr ------------------------------------------------------------ Server listening on TCP port 5001 with pid 25379 Read buffer size: 128 KByte (Dist bin width=3D16.0 KByte) TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58152 (full-duplex) (trip-times) (MSS=3D1448) (peer 2.0.14-alpha) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=3DDist [ 4] 0.00-1.00 sec 1.05 MBytes 8.79 Mbits/sec 547.494/68.304/904.670/301.250 ms (8/137379) 9.75 MByte 2.01 345=3D344:1:0:0:0:0:0:0 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 4] 0.00-1.00 sec 40.3 MBytes 338 Mbits/sec 323/0 0 1267K/20319 us 2077.13 [ ID] Interval Transfer Bandwidth [FD4] 0.00-1.00 sec 41.3 MBytes 346 Mbits/sec [ 4] 1.00-2.00 sec 1.34 MBytes 11.3 Mbits/sec 1307.675/904.670/1454.073/145.211 ms (11/127950) 2.30 MByte 1.08 277=3D262:5:4:6:0:0:0:0 [ 4] 1.00-2.00 sec 38.8 MBytes 325 Mbits/sec 310/0 0 1001K/24428 us 1663.35 [FD4] 1.00-2.00 sec 40.1 MBytes 336 Mbits/sec [ 4] 2.00-3.00 sec 38.0 MBytes 319 Mbits/sec 304/0 0 1156K/26355 us 1511.89 [ 4] 2.00-3.00 sec 1.33 MBytes 11.1 Mbits/sec 1298.825/1139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07 372=3D366:2:1:3:0:0:0:0 [FD4] 2.00-3.00 sec 39.3 MBytes 330 Mbits/sec [ 4] 3.00-4.00 sec 1.32 MBytes 11.1 Mbits/sec 1302.239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06 217=3D199:4:8:4:2:0:0:0 [ 4] 3.00-4.00 sec 39.6 MBytes 332 Mbits/sec 317/0 0 970K/19064 us 2179.49 [FD4] 3.00-4.00 sec 40.9 MBytes 343 Mbits/sec [ 4] 4.00-5.00 sec 38.5 MBytes 323 Mbits/sec 308/0 0 998K/20599 us 1959.81 [ 4] 4.00-5.00 sec 1.39 MBytes 11.7 Mbits/sec 1310.588/1172.769/1233.522/21.090 ms (11/132426) 1.79 MByte 1.11 223=3D196:18:5:4:0:0:0:0 [FD4] 4.00-5.00 sec 39.9 MBytes 335 Mbits/sec [ 4] 5.00-6.00 sec 38.6 MBytes 324 Mbits/sec 309/0 0 1340K/17717 us 2286.01 [ 4] 5.00-6.00 sec 1.33 MBytes 11.2 Mbits/sec 1305.731/1173.179/1220.196/17.125 ms (11/127160) 1.72 MByte 1.07 158=3D136:10:5:6:1:0:0:0 [FD4] 5.00-6.00 sec 40.0 MBytes 335 Mbits/sec [ 4] 6.00-7.00 sec 1.36 MBytes 11.4 Mbits/sec 1301.735/1162.410/1216.840/20.348 ms (10/142193) 1.87 MByte 1.09 140=3D116:7:8:6:3:0:0:0 [ 4] 6.00-7.00 sec 39.4 MBytes 330 Mbits/sec 315/0 0 1074K/20040 us 2060.26 [FD4] 6.00-7.00 sec 40.7 MBytes 342 Mbits/sec [ 4] 7.00-8.00 sec 1.35 MBytes 11.3 Mbits/sec 1291.605/1160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10 216=3D198:5:6:7:0:0:0:0 [ 4] 7.00-8.00 sec 38.9 MBytes 326 Mbits/sec 311/0 0 1074K/30895 us 1319.42 [FD4] 7.00-8.00 sec 40.2 MBytes 337 Mbits/sec [ 4] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec 299/0 0 1264K/29185 us 1342.83 [ 4] 8.00-9.00 sec 1.27 MBytes 10.7 Mbits/sec 1326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.01 110=3D86:9:7:5:1:0:2:0 [FD4] 8.00-9.00 sec 38.6 MBytes 324 Mbits/sec [ 4] 9.00-10.00 sec 1.34 MBytes 11.2 Mbits/sec 1312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByte 1.07 124=3D101:12:6:2:1:0:2:0 [ 4] 9.00-10.00 sec 39.2 MBytes 329 Mbits/sec 314/0 0 967K/21976 us 1872.80 [FD4] 9.00-10.00 sec 40.6 MBytes 341 Mbits/sec [ 4] 10.00-10.00 sec 256 KBytes 477 Mbits/sec 2/0 0 967K/21976 us 2711.06 [ 4] 0.00-10.00 sec 389 MBytes 326 Mbits/sec 3112/0 0 967K/21976 us 1854.69 [ 4] 10.00-11.00 sec 963 KBytes 7.89 Mbits/sec 1624.535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61 215=3D210:3:0:0:1:1:0:0 [ 4] 11.00-11.41 sec 622 KBytes 12.6 Mbits/sec 1780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0.88 29=3D11:11:4:3:0:0:0:0 [ 4] 0.00-11.41 sec 14.6 MBytes 10.8 Mbits/sec 1184.903/68.304/1548.718/215.041 ms (117/131072) 1.52 MByte 1.13 2426=3D2225:87:54:46:9:1:4:0 [FD4] 10.00-1602028944.71 sec 1.80 MBytes 0.009 bits/sec [FD4] 0.00-1602028944.71 sec 404 MBytes 2.11 bits/sec Client [root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times --full-duplex -Z bbr WARNING: tcp congestion control will only be applied on transmit traffic, use -Z on the server ------------------------------------------------------------ Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 47994 (1 flows) Write buffer size: 128 KByte TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.35%enp2s0 port 58154 connected with 45.33.58.123 port 5001 (full-duplex) (trip-times) (MSS=3D1448) (ct=3D12.42 ms) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=3DDist [ 3] 0.00-1.00 sec 35.8 MBytes 301 Mbits/sec 89.806/29.604/103.341/16.204 ms (286/131398) 3.58 MByte 418.46 17764=3D17726:35:3:0:0:0:0:0 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 3] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec 21/0 91 62K/44390 us 59.06 [ ID] Interval Transfer Bandwidth [FD3] 0.00-1.00 sec 38.3 MBytes 322 Mbits/sec [ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 8 56K/36570 us 35.84 [ 3] 1.00-2.00 sec 38.7 MBytes 325 Mbits/sec 94.258/61.108/104.117/ 9.454 ms (310/130882) 3.63 MByte 430.45 20775=3D20768:5:2:0:0:0:0:0 [FD3] 1.00-2.00 sec 39.9 MBytes 335 Mbits/sec [ 3] 2.00-3.00 sec 38.6 MBytes 324 Mbits/sec 94.554/62.312/105.451/ 9.324 ms (308/131379) 3.69 MByte 427.96 18929=3D18913:15:1:0:0:0:0:0 [ 3] 2.00-3.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 12 65K/54911 us 26.26 [FD3] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec [ 3] 3.00-4.00 sec 37.3 MBytes 313 Mbits/sec 95.362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71 17567=3D17537:27:3:0:0:0:0:0 [ 3] 3.00-4.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 20 67K/46946 us 30.71 [FD3] 3.00-4.00 sec 38.6 MBytes 324 Mbits/sec [ 3] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 51 59K/67662 us 19.37 [ 3] 4.00-5.00 sec 38.4 MBytes 322 Mbits/sec 90.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.99 19095=3D19077:16:2:0:0:0:0:0 [FD3] 4.00-5.00 sec 39.6 MBytes 333 Mbits/sec [ 3] 5.00-6.00 sec 38.5 MBytes 323 Mbits/sec 88.832/64.621/102.534/ 9.742 ms (308/130979) 3.40 MByte 454.13 17674=3D17657:17:0:0:0:0:0:0 [ 3] 5.00-6.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 48 65K/64456 us 22.37 [FD3] 5.00-6.00 sec 39.8 MBytes 334 Mbits/sec [ 3] 6.00-7.00 sec 38.4 MBytes 322 Mbits/sec 87.121/62.694/102.124/10.260 ms (307/131199) 3.24 MByte 462.33 19853=3D19852:1:0:0:0:0:0:0 [ 3] 6.00-7.00 sec 3.75 MBytes 31.5 Mbits/sec 30/0 59 67K/151435 us 25.97 [FD3] 6.00-7.00 sec 42.2 MBytes 354 Mbits/sec [ 3] 7.00-8.00 sec 38.5 MBytes 323 Mbits/sec 87.936/62.694/102.226/10.016 ms (308/131222) 3.41 MByte 459.61 19067=3D19052:15:0:0:0:0:0:0 [ 3] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 60 73K/56735 us 20.79 [FD3] 7.00-8.00 sec 39.7 MBytes 333 Mbits/sec [ 3] 8.00-9.00 sec 35.9 MBytes 301 Mbits/sec 91.441/56.797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54 18900=3D18881:10:2:0:0:0:0:7 [ 3] 8.00-9.00 sec 1.50 MBytes 12.6 Mbits/sec 13/0 26 82K/49077 us 32.05 [FD3] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec [ 3] 9.00-10.00 sec 36.3 MBytes 305 Mbits/sec 90.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88 17888=3D17871:17:0:0:0:0:0:0 [ 3] 10.00-10.07 sec 2.72 MBytes 310 Mbits/sec 91.852/69.979/102.633/ 8.280 ms (22/129636) 5.96 MByte 421.94 1418=3D1414:4:0:0:0:0:0:0 [ 3] 0.00-10.07 sec 379 MBytes 316 Mbits/sec 90.712/29.604/148.892/12.195 ms (3033/131072) 3.41 MByte 435.04 188930=3D188748:162:13:0:0:0:0:7 [ 3] 9.00-10.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 0 5K/55760 us 25.86 [FD3] 9.00-10.00 sec 40.4 MBytes 339 Mbits/sec [ 3] 10.00-10.49 sec 223 KBytes 3.69 Mbits/sec 2/0 0 5K/55760 us 8.28 [ 3] 0.00-10.49 sec 17.1 MBytes 13.7 Mbits/sec 139/0 424 5K/55760 us 30.63 [FD3] 10.00-10.49 sec 223 KBytes 3.77 Mbits/sec [FD3] 0.00-10.49 sec 396 MBytes 317 Mbits/sec Bob On Tue, Oct 6, 2020 at 6:24 AM Michael Welzl wrote: > Thanks a lot - just the info I was looking for (also from others who hav= e > responded in the meantime, thanks!) > > > > On 6 Oct 2020, at 14:44, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > > > > Michael Welzl writes: > > > >> Hi, and thanks for a quick answer! > >> > >> But, it's not quite what I was looking for.... see below: > >> > >>> On 6 Oct 2020, at 13:47, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >>> > >>> Michael Welzl writes: > >>> > >>>> Hi all, > >>>> > >>>> A simple question to y'all who spent so much time on Cake and things > >>>> ... in a household using WiFi, which buffer is usually bloated? Wher= e > >>>> does the latency really come from? > >>>> > >>>> Is it: > >>>> 1. the access point's downlink queue, feeding into the WiFi network, > >>> > >>> This we mostly fixed, but only if you're on a recent OpenWrt with the > >>> right WiFi drivers. > >> > >> Well okay... I was curious about where the bottleneck is. I can > >> translate my question into: "if Cake is installed everywhere, where > >> does it have the most work to do?". > > > > Well, CAKE only runs on the upstream link, so that's where it does its > > work. The software shaper model doesn't really work that well on WiFi, > > so we generally encourage people to just run fixed drivers there > > instead. That being said I have heard of one or two WiFi deployments > > where that was not an option, and where CAKE was used as a shaper > > instead. This was for a fixed WiFi backhaul, though, and even so they > > had to set the shaper quite a lot lower than the max bandwidth to get > > reliable performance. > > > >>> Otherwise, this is a major source of latency *if* > >>> the WiFi link is faster than the downlink from the internet. > >> > >> Huh? Slower, you mean? > > > > No, if the WiFi link is faster, the upstream link becomes the bottlenec= k > > and CAKE has work to do :) > > > >>> This > >>> depends on both the internet connection and the current rate each WiF= i > >>> station operates at, so it can vary wildly, and on very short time > >>> scales. > >> > >> Sure... I was asking for the "if" in your statement above - since this > >> is an operationally-oriented list: what do people see? What is the > >> more common case? > > > > Right, well as you can probably tell that might not have been entirely > > clear from your initial post ;) > > > >>> The extent to which this happens depends on where you are in the > >>> world; personally I've been bottlenecked on the WiFi link ever since > >>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps, > >>> maybe that'll change again?). > >> > >> THIS is what I was after :) one data point, cool - so far, so good... > > > > Ah, you're after anecdotes - well why didn't you say so? ;) > > > > In that case I'll add that my own connection is the only one I've come > > across where the WiFi link is *never* the bottleneck. In Denmark we are > > finally (slowly) getting out of the dark ages as far as fibre > > deployments are concerned, but most operators will sell connections > > capped at 100Mbps or 250Mbps, which is still less than the throughout o= f > > a 802.11ac link with good signal conditions (my phone consistently gets > > ~250-350 Mbps on a speedtest.net run). > > > > DSL connections tend to have awful latency, and are still quite common, > > but they are pretty easy to fix with CAKE. Cable connections likewise, > > or so is my impression (those are not so common around these parts). > > > > The worst are definitely LTE/mobile broadband connections. Wildly > > varying link speeds, and awful over-buffering; so you really have to > > clamp them down to get anything useful out of CAKE. > > > > -Toke > > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast --00000000000054b17305b1098a91 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
You might want to try iperf 2.0.14 --trip-times option to = measure your links.=C2=A0 Below a run with a PC w/a 1G NIC connected to the= 1G cable MODEM no WiFi, i.e. wired, full duplex socket.=C2=A0 The uplink t= cp write latencies are larger than one second.=C2=A0 Downlinks just under 1= 00 ms.=C2=A0 Slow down the uplink traffic to less than the congestion point= =C2=A0and write to read latency drops to ~65 ms.=C2=A0 I think this suggest= s my CABLE MODEM is full of bloat.

UPLINK ONLY
------------------= --

Server:
[rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -= Z bbr
------------------------------------------------------------
Se= rver listening on TCP port 5001 with pid 25408
Read buffer size: =C2=A01= 28 KByte (Dist bin width=3D16.0 KByte)
TCP congestion control set to bbr=
TCP window size: =C2=A0128 KByte (default)
-------------------------= -----------------------------------
[ =C2=A04] local 45.33.58.123%eth0 p= ort 5001 connected with 73.92.17.76 port 58158 (trip-times) (MSS=3D1448) (p= eer 2.0.14-alpha)
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer =C2= =A0 =C2=A0Bandwidth =C2=A0 =C2=A0Burst Latency avg/min/max/stdev (cnt/size)= inP NetPwr =C2=A0Reads=3DDist
[ =C2=A04] 0.00-1.00 sec =C2=A01.25 MByte= s =C2=A010.5 Mbits/sec =C2=A059.680/51.222/96.282/34.950 ms (10/131072) 80.= 9 KByte 21.96 =C2=A0305=3D305:0:0:0:0:0:0:0
[ =C2=A04] 1.00-2.00 sec =C2= =A01.12 MBytes =C2=A09.44 Mbits/sec =C2=A065.218/51.951/87.895/10.877 ms (9= /131072) 79.6 KByte 18.09 =C2=A0262=3D262:0:0:0:0:0:0:0
[ =C2=A04] 2.00-= 3.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/sec =C2=A062.479/51.170/67.962/= 5.189 ms (10/131072) 76.3 KByte 20.98 =C2=A0289=3D289:0:0:0:0:0:0:0
[ = =C2=A04] 3.00-4.00 sec =C2=A01.12 MBytes =C2=A09.44 Mbits/sec =C2=A062.861/= 52.478/68.339/ 5.212 ms (9/131072) 76.7 KByte 18.77 =C2=A0262=3D262:0:0:0:0= :0:0:0
[ =C2=A04] 4.00-5.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/sec = =C2=A064.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23 =C2=A0292= =3D292:0:0:0:0:0:0:0
[ =C2=A04] 5.00-6.00 sec =C2=A01.14 MBytes =C2=A09.= 60 Mbits/sec =C2=A064.618/50.914/68.277/ 5.696 ms (9/133334) 80.2 KByte 18.57 =C2=A0259=3D259:0= :0:0:0:0:0:0
[ =C2=A04] 6.00-7.00 sec =C2=A01.23 MBytes =C2=A010.3 Mbits= /sec =C2=A066.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29 =C2= =A0279=3D279:0:0:0:0:0:0:0
[ =C2=A04] 7.00-8.00 sec =C2=A01.16 MBytes = =C2=A09.70 Mbits/sec =C2=A068.954/51.902/100.783/15.733 ms (9/134782) 86.6 = KByte 17.59 =C2=A0275=3D274:1:0:0:0:0:0:0
[ =C2=A04] 8.00-9.00 sec =C2= =A01.22 MBytes =C2=A010.2 Mbits/sec =C2=A063.855/51.386/69.919/ 5.284 ms (9= /141851) 84.4 KByte 19.99 =C2=A0283=3D283:0:0:0:0:0:0:0
[ =C2=A04] 9.00-= 10.00 sec =C2=A01.18 MBytes =C2=A09.86 Mbits/sec =C2=A067.975/50.996/88.151= /11.318 ms (10/123252) 78.0 KByte 18.13 =C2=A0275=3D275:0:0:0:0:0:0:0
[ = =C2=A04] 10.00-10.14 sec =C2=A0 205 KBytes =C2=A011.7 Mbits/sec =C2=A099.86= 5/55.342/77.370/ 7.319 ms (2/104962) 97.6 KByte 14.61 =C2=A043=3D43:0:0:0:0= :0:0:0
[ =C2=A04] 0.00-10.14 sec =C2=A012.1 MBytes =C2=A010.0 Mbits/sec = =C2=A059.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04 =C2=A028= 24=3D2823:1:0:0:0:0:0:0

Client:

[root@localhost iperf2-code]= # iperf -c bobcat.rjmcmahon.com= -i1 --trip-times -Z bbr -b= 10m -e
---------------------------------------------------------= ---
Client connecting to bobcat.= rjmcmahon.com, TCP port 5001 with pid 48006 (1 flows)
Write buffer s= ize: =C2=A0128 KByte
TCP congestion control set to bbr
TCP window siz= e: =C2=A0128 KByte (default)
-------------------------------------------= -----------------
[ =C2=A03] local 192.168.1.35%enp2s0 port 58160 connec= ted with 45.33.58.123 port 5001 (trip-times) (MSS=3D1448) (ct=3D13.05 ms)[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer =C2=A0 =C2=A0Bandwidth= =C2=A0 =C2=A0 =C2=A0 Write/Err =C2=A0Rtry =C2=A0 =C2=A0 Cwnd/RTT =C2=A0 = =C2=A0 =C2=A0 =C2=A0NetPwr
[ =C2=A03] 0.00-1.00 sec =C2=A01.25 MBytes = =C2=A010.5 Mbits/sec =C2=A011/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 = =C2=A0 =C2=A0135K/49251 us =C2=A026.61
[ =C2=A03] 1.00-2.00 sec =C2=A01.= 25 MBytes =C2=A010.5 Mbits/sec =C2=A010/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00 =C2=A0 =C2=A0 =C2=A0141K/24791 us =C2=A052.87
[ =C2=A03] 2.00-3.00 = sec =C2=A01.12 MBytes =C2=A09.44 Mbits/sec =C2=A09/0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0135K/24973 us =C2=A047.24
[ =C2=A03] = 3.00-4.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/sec =C2=A010/0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0158K/25021 us =C2=A052.38
= [ =C2=A03] 4.00-5.00 sec =C2=A01.12 MBytes =C2=A09.44 Mbits/sec =C2=A09/0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0141K/21247 us =C2= =A055.52
[ =C2=A03] 5.00-6.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/sec= =C2=A010/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0144K/23= 535 us =C2=A055.69
[ =C2=A03] 6.00-7.00 sec =C2=A01.12 MBytes =C2=A09.44= Mbits/sec =C2=A09/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2= =A0135K/27130 us =C2=A043.48
[ =C2=A03] 7.00-8.00 sec =C2=A01.25 MBytes = =C2=A010.5 Mbits/sec =C2=A010/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 = =C2=A0 =C2=A0135K/24813 us =C2=A052.82
[ =C2=A03] 8.00-9.00 sec =C2=A01.= 12 MBytes =C2=A09.44 Mbits/sec =C2=A09/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 0 =C2=A0 =C2=A0 =C2=A0155K/22697 us =C2=A051.97
[ =C2=A03] 9.00-10.00 se= c =C2=A01.25 MBytes =C2=A010.5 Mbits/sec =C2=A010/0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0124K/21954 us =C2=A059.70
[ =C2=A03] = 10.00-10.07 sec =C2=A0 256 KBytes =C2=A031.6 Mbits/sec =C2=A02/0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0124K/21954 us =C2=A0179.64[ =C2=A03] 0.00-10.07 sec =C2=A012.3 MBytes =C2=A010.2 Mbits/sec =C2=A099/= 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0124K/21954 us =C2= =A058.12

FULL DUPLEX:
---------------------
Server:

[r= jmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
--------------= ----------------------------------------------
Server listening on TCP p= ort 5001 with pid 25379
Read buffer size: =C2=A0128 KByte (Dist bin widt= h=3D16.0 KByte)
TCP congestion control set to bbr
TCP window size: = =C2=A0128 KByte (default)
----------------------------------------------= --------------
[ =C2=A04] local 45.33.58.123%eth0 port 5001 connected wi= th 73.92.17.76 port 58152 (full-duplex) (trip-times) (MSS=3D1448) (peer 2.0= .14-alpha)
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer =C2=A0 =C2= =A0Bandwidth =C2=A0 =C2=A0Burst Latency avg/min/max/stdev (cnt/size) inP Ne= tPwr =C2=A0Reads=3DDist
[ =C2=A04] 0.00-1.00 sec =C2=A01.05 MBytes =C2= =A08.79 Mbits/sec =C2=A0547.494/68.304/904.670/301.250 ms (8/137379) 9.75 M= Byte 2.01 =C2=A0345=3D344:1:0:0:0:0:0:0
[ ID] Interval =C2=A0 =C2=A0 =C2= =A0 =C2=A0Transfer =C2=A0 =C2=A0Bandwidth =C2=A0 =C2=A0 =C2=A0 Write/Err = =C2=A0Rtry =C2=A0 =C2=A0 Cwnd/RTT =C2=A0 =C2=A0 =C2=A0 =C2=A0NetPwr
[ = =C2=A04] 0.00-1.00 sec =C2=A040.3 MBytes =C2=A0 338 Mbits/sec =C2=A0323/0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1267K/20319 us =C2=A02077= .13
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bandwidth=
[FD4] 0.00-1.00 sec =C2=A041.3 MBytes =C2=A0 346 Mbits/sec
[ =C2=A04= ] 1.00-2.00 sec =C2=A01.34 MBytes =C2=A011.3 Mbits/sec =C2=A01307.675/904.670/1454.073/145.211 = ms (11/127950) 2.30 MByte 1.08 =C2=A0277=3D262:5:4:6:0:0:0:0
[ =C2=A04] = 1.00-2.00 sec =C2=A038.8 MBytes =C2=A0 325 Mbits/sec =C2=A0310/0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1001K/24428 us =C2=A01663.35
[FD= 4] 1.00-2.00 sec =C2=A040.1 MBytes =C2=A0 336 Mbits/sec
[ =C2=A04] 2.00-= 3.00 sec =C2=A038.0 MBytes =C2=A0 319 Mbits/sec =C2=A0304/0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1156K/26355 us =C2=A01511.89
[ =C2= =A04] 2.00-3.00 sec =C2=A01.33 MBytes =C2=A011.1 Mbits/sec =C2=A01298.825/1= 139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07 =C2=A0372=3D366:2:1:= 3:0:0:0:0
[FD4] 2.00-3.00 sec =C2=A039.3 MBytes =C2=A0 330 Mbits/sec
= [ =C2=A04] 3.00-4.00 sec =C2=A01.32 MBytes =C2=A011.1 Mbits/sec =C2=A01302.= 239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06 =C2=A0217=3D199= :4:8:4:2:0:0:0
[ =C2=A04] 3.00-4.00 sec =C2=A039.6 MBytes =C2=A0 332 Mbi= ts/sec =C2=A0317/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0= 970K/19064 us =C2=A02179.49
[FD4] 3.00-4.00 sec =C2=A040.9 MBytes =C2=A0= 343 Mbits/sec
[ =C2=A04] 4.00-5.00 sec =C2=A038.5 MBytes =C2=A0 323 Mbi= ts/sec =C2=A0308/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0= 998K/20599 us =C2=A01959.81
[ =C2=A04] 4.00-5.00 sec =C2=A01.39 MBytes = =C2=A011.7 Mbits/sec =C2=A01310.588/1172.769/1233.522/21.090 ms (11/132426)= 1.79 MByte 1.11 =C2=A0223=3D196:18:5:4:0:0:0:0
[FD4] 4.00-5.00 sec =C2= =A039.9 MBytes =C2=A0 335 Mbits/sec
[ =C2=A04] 5.00-6.00 sec =C2=A038.6 = MBytes =C2=A0 324 Mbits/sec =C2=A0309/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00= =C2=A0 =C2=A0 1340K/17717 us =C2=A02286.01
[ =C2=A04] 5.00-6.00 sec =C2= =A01.33 MBytes =C2=A011.2 Mbits/sec =C2=A01305.731/1173.179/1220.196/17.125= ms (11/127160) 1.72 MByte 1.07 =C2=A0158=3D136:10:5:6:1:0:0:0
[FD4] 5.0= 0-6.00 sec =C2=A040.0 MBytes =C2=A0 335 Mbits/sec
[ =C2=A04] 6.00-7.00 s= ec =C2=A01.36 MBytes =C2=A011.4 Mbits/sec =C2=A01301.735/1162.410/1216.840/= 20.348 ms (10/142193) 1.87 MByte 1.09 =C2=A0140=3D116:7:8:6:3:0:0:0
[ = =C2=A04] 6.00-7.00 sec =C2=A039.4 MBytes =C2=A0 330 Mbits/sec =C2=A0315/0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1074K/20040 us =C2=A02060= .26
[FD4] 6.00-7.00 sec =C2=A040.7 MBytes =C2=A0 342 Mbits/sec
[ =C2= =A04] 7.00-8.00 sec =C2=A01.35 MBytes =C2=A011.3 Mbits/sec =C2=A01291.605/1= 160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10 =C2=A0216=3D198:5:6:= 7:0:0:0:0
[ =C2=A04] 7.00-8.00 sec =C2=A038.9 MBytes =C2=A0 326 Mbits/se= c =C2=A0311/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1074K/30895= us =C2=A01319.42
[FD4] 7.00-8.00 sec =C2=A040.2 MBytes =C2=A0 337 Mbits= /sec
[ =C2=A04] 8.00-9.00 sec =C2=A037.4 MBytes =C2=A0 314 Mbits/sec =C2= =A0299/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 1264K/29185 us = =C2=A01342.83
[ =C2=A04] 8.00-9.00 sec =C2=A01.27 MBytes =C2=A010.7 Mbit= s/sec =C2=A01326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.0= 1 =C2=A0110=3D86:9:7:5:1:0:2:0
[FD4] 8.00-9.00 sec =C2=A038.6 MBytes =C2= =A0 324 Mbits/sec
[ =C2=A04] 9.00-10.00 sec =C2=A01.34 MBytes =C2=A011.2= Mbits/sec =C2=A01312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByt= e 1.07 =C2=A0124=3D101:12:6:2:1:0:2:0
[ =C2=A04] 9.00-10.00 sec =C2=A039= .2 MBytes =C2=A0 329 Mbits/sec =C2=A0314/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00 =C2=A0 =C2=A0 =C2=A0967K/21976 us =C2=A01872.80
[FD4] 9.00-10.00 se= c =C2=A040.6 MBytes =C2=A0 341 Mbits/sec
[ =C2=A04] 10.00-10.00 sec =C2= =A0 256 KBytes =C2=A0 477 Mbits/sec =C2=A02/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A00 =C2=A0 =C2=A0 =C2=A0967K/21976 us =C2=A02711.06
[ =C2=A04] 0.00-= 10.00 sec =C2=A0 389 MBytes =C2=A0 326 Mbits/sec =C2=A03112/0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0967K/21976 us =C2=A01854.69
[= =C2=A04] 10.00-11.00 sec =C2=A0 963 KBytes =C2=A07.89 Mbits/sec =C2=A01624= .535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61 =C2=A0215=3D210= :3:0:0:1:1:0:0
[ =C2=A04] 11.00-11.41 sec =C2=A0 622 KBytes =C2=A012.6 M= bits/sec =C2=A01780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0= .88 =C2=A029=3D11:11:4:3:0:0:0:0
[ =C2=A04] 0.00-11.41 sec =C2=A014.6 MB= ytes =C2=A010.8 Mbits/sec =C2=A01184.903/68.304/1548.718/215.041 ms (117/13= 1072) 1.52 MByte 1.13 =C2=A02426=3D2225:87:54:46:9:1:4:0
[FD4] 10.00-160= 2028944.71 sec =C2=A01.80 MBytes =C2=A00.009 bits/sec
[FD4] 0.00-1602028= 944.71 sec =C2=A0 404 MBytes =C2=A02.11 bits/sec

Client

[root= @localhost iperf2-code]# iperf -c b= obcat.rjmcmahon.com -i1 --trip-times --full-duplex -Z bbr
WARNING: t= cp congestion control will only be applied on transmit traffic, use -Z on t= he server
------------------------------------------------------------Client connecting to bobcat.rjmcm= ahon.com, TCP port 5001 with pid 47994 (1 flows)
Write buffer size: = =C2=A0128 KByte
TCP congestion control set to bbr
TCP window size: = =C2=A0128 KByte (default)
----------------------------------------------= --------------
[ =C2=A03] local 192.168.1.35%enp2s0 port 58154 connected= with 45.33.58.123 port 5001 (full-duplex) (trip-times) (MSS=3D1448) (ct=3D= 12.42 ms)
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer =C2=A0 =C2= =A0Bandwidth =C2=A0 =C2=A0Burst Latency avg/min/max/stdev (cnt/size) inP Ne= tPwr =C2=A0Reads=3DDist
[ =C2=A03] 0.00-1.00 sec =C2=A035.8 MBytes =C2= =A0 301 Mbits/sec =C2=A089.806/29.604/103.341/16.204 ms (286/131398) 3.58 M= Byte 418.46 =C2=A017764=3D17726:35:3:0:0:0:0:0
[ ID] Interval =C2=A0 =C2= =A0 =C2=A0 =C2=A0Transfer =C2=A0 =C2=A0Bandwidth =C2=A0 =C2=A0 =C2=A0 Write= /Err =C2=A0Rtry =C2=A0 =C2=A0 Cwnd/RTT =C2=A0 =C2=A0 =C2=A0 =C2=A0NetPwr[ =C2=A03] 0.00-1.00 sec =C2=A02.50 MBytes =C2=A021.0 Mbits/sec =C2=A021/0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 91 =C2=A0 =C2=A0 =C2=A0 62K/44390 us =C2=A059.= 06
[ ID] Interval =C2=A0 =C2=A0 =C2=A0 Transfer =C2=A0 =C2=A0 Bandwidth<= br>[FD3] 0.00-1.00 sec =C2=A038.3 MBytes =C2=A0 322 Mbits/sec
[ =C2=A03]= 1.00-2.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/sec =C2=A010/0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A08 =C2=A0 =C2=A0 =C2=A0 56K/36570 us =C2=A035.84
= [ =C2=A03] 1.00-2.00 sec =C2=A038.7 MBytes =C2=A0 325 Mbits/sec =C2=A094.258/61.108/104.117/ 9.= 454 ms (310/130882) 3.63 MByte 430.45 =C2=A020775=3D20768:5:2:0:0:0:0:0
= [FD3] 1.00-2.00 sec =C2=A039.9 MBytes =C2=A0 335 Mbits/sec
[ =C2=A03] 2.= 00-3.00 sec =C2=A038.6 MBytes =C2=A0 324 Mbits/sec =C2=A094.554/62.312/105.= 451/ 9.324 ms (308/131379) 3.69 MByte 427.96 =C2=A018929=3D18913:15:1:0:0:0= :0:0
[ =C2=A03] 2.00-3.00 sec =C2=A01.38 MBytes =C2=A011.5 Mbits/sec =C2= =A011/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 12 =C2=A0 =C2=A0 =C2=A0 65K/54911 us = =C2=A026.26
[FD3] 2.00-3.00 sec =C2=A040.0 MBytes =C2=A0 335 Mbits/sec[ =C2=A03] 3.00-4.00 sec =C2=A037.3 MBytes =C2=A0 313 Mbits/sec =C2=A095.= 362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71 =C2=A017567=3D1= 7537:27:3:0:0:0:0:0
[ =C2=A03] 3.00-4.00 sec =C2=A01.38 MBytes =C2=A011.= 5 Mbits/sec =C2=A011/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 20 =C2=A0 =C2=A0 =C2=A0 = 67K/46946 us =C2=A030.71
[FD3] 3.00-4.00 sec =C2=A038.6 MBytes =C2=A0 32= 4 Mbits/sec
[ =C2=A03] 4.00-5.00 sec =C2=A01.25 MBytes =C2=A010.5 Mbits/= sec =C2=A010/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 51 =C2=A0 =C2=A0 =C2=A0 59K/6766= 2 us =C2=A019.37
[ =C2=A03] 4.00-5.00 sec =C2=A038.4 MBytes =C2=A0 322 M= bits/sec =C2=A090.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.= 99 =C2=A019095=3D19077:16:2:0:0:0:0:0
[FD3] 4.00-5.00 sec =C2=A039.6 MBy= tes =C2=A0 333 Mbits/sec
[ =C2=A03] 5.00-6.00 sec =C2=A038.5 MBytes =C2= =A0 323 Mbits/sec =C2=A088.832/64.621/102.534/ 9.742 ms (308/130979) 3.40 M= Byte 454.13 =C2=A017674=3D17657:17:0:0:0:0:0:0
[ =C2=A03] 5.00-6.00 sec = =C2=A01.38 MBytes =C2=A011.5 Mbits/sec =C2=A011/0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 48 =C2=A0 =C2=A0 =C2=A0 65K/64456 us =C2=A022.37
[FD3] 5.00-6.00 sec= =C2=A039.8 MBytes =C2=A0 334 Mbits/sec
[ =C2=A03] 6.00-7.00 sec =C2=A03= 8.4 MBytes =C2=A0 322 Mbits/sec =C2=A087.121/62.694/102.124/10.260 ms (307/= 131199) 3.24 MByte 462.33 =C2=A019853=3D19852:1:0:0:0:0:0:0
[ =C2=A03] 6= .00-7.00 sec =C2=A03.75 MBytes =C2=A031.5 Mbits/sec =C2=A030/0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 59 =C2=A0 =C2=A0 =C2=A0 67K/151435 us =C2=A025.97
[FD3= ] 6.00-7.00 sec =C2=A042.2 MBytes =C2=A0 354 Mbits/sec
[ =C2=A03] 7.00-8= .00 sec =C2=A038.5 MBytes =C2=A0 323 Mbits/sec =C2=A087.936/62.694/102.226/= 10.016 ms (308/131222) 3.41 MByte 459.61 =C2=A019067=3D19052:15:0:0:0:0:0:0=
[ =C2=A03] 7.00-8.00 sec =C2=A01.12 MBytes =C2=A09.44 Mbits/sec =C2=A09= /0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 60 =C2=A0 =C2=A0 =C2=A0 73K/56735 us =C2=A02= 0.79
[FD3] 7.00-8.00 sec =C2=A039.7 MBytes =C2=A0 333 Mbits/sec
[ =C2= =A03] 8.00-9.00 sec =C2=A035.9 MBytes =C2=A0 301 Mbits/sec =C2=A091.441/56.= 797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54 =C2=A018900=3D18881:10= :2:0:0:0:0:7
[ =C2=A03] 8.00-9.00 sec =C2=A01.50 MBytes =C2=A012.6 Mbits= /sec =C2=A013/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 82K/490= 77 us =C2=A032.05
[FD3] 8.00-9.00 sec =C2=A037.4 MBytes =C2=A0 314 Mbits= /sec
[ =C2=A03] 9.00-10.00 sec =C2=A036.3 MBytes =C2=A0 305 Mbits/sec = =C2=A090.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88 =C2=A0= 17888=3D17871:17:0:0:0:0:0:0
[ =C2=A03] 10.00-10.07 sec =C2=A02.72 MByte= s =C2=A0 310 Mbits/sec =C2=A091.852/69.979/102.633/ 8.280 ms (22/129636) 5.= 96 MByte 421.94 =C2=A01418=3D1414:4:0:0:0:0:0:0
[ =C2=A03] 0.00-10.07 se= c =C2=A0 379 MBytes =C2=A0 316 Mbits/sec =C2=A090.712/29.604/148.892/12.195= ms (3033/131072) 3.41 MByte 435.04 =C2=A0188930=3D188748:162:13:0:0:0:0:7<= br>[ =C2=A03] 9.00-10.00 sec =C2=A01.38 MBytes =C2=A011.5 Mbits/sec =C2=A01= 1/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A05K/55760= us =C2=A025.86
[FD3] 9.00-10.00 sec =C2=A040.4 MBytes =C2=A0 339 Mbits/= sec
[ =C2=A03] 10.00-10.49 sec =C2=A0 223 KBytes =C2=A03.69 Mbits/sec = =C2=A02/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A05K= /55760 us =C2=A08.28
[ =C2=A03] 0.00-10.49 sec =C2=A017.1 MBytes =C2=A01= 3.7 Mbits/sec =C2=A0139/0 =C2=A0 =C2=A0 =C2=A0 =C2=A0424 =C2=A0 =C2=A0 =C2= =A0 =C2=A05K/55760 us =C2=A030.63
[FD3] 10.00-10.49 sec =C2=A0 223 KByte= s =C2=A03.77 Mbits/sec
[FD3] 0.00-10.49 sec =C2=A0 396 MBytes =C2=A0 317= Mbits/sec

Bob

On Tue, Oct 6, 2020 at 6:24 AM Michael Wel= zl <michawe@ifi.uio.no> wro= te:
Thanks a lot= - just the info I was looking for=C2=A0 (also from others who have respond= ed in the meantime, thanks!)


> On 6 Oct 2020, at 14:44, Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk> wrote:
>
> Michael Welzl <michawe@ifi.uio.no> writes:
>
>> Hi, and thanks for a quick answer!
>>
>> But, it's not quite what I was looking for.... see below:
>>
>>> On 6 Oct 2020, at 13:47, Toke H=C3=B8iland-J=C3=B8rgensen <= toke@toke.dk> wrot= e:
>>>
>>> Michael Welzl <michawe@ifi.uio.no> writes:
>>>
>>>> Hi all,
>>>>
>>>> A simple question to y'all who spent so much time on C= ake and things
>>>> ... in a household using WiFi, which buffer is usually blo= ated? Where
>>>> does the latency really come from?
>>>>
>>>> Is it:
>>>> 1. the access point's downlink queue, feeding into the= WiFi network,
>>>
>>> This we mostly fixed, but only if you're on a recent OpenW= rt with the
>>> right WiFi drivers.
>>
>> Well okay... I was curious about where the bottleneck is. I can >> translate my question into: "if Cake is installed everywhere,= where
>> does it have the most work to do?".
>
> Well, CAKE only runs on the upstream link, so that's where it does= its
> work. The software shaper model doesn't really work that well on W= iFi,
> so we generally encourage people to just run fixed drivers there
> instead. That being said I have heard of one or two WiFi deployments > where that was not an option, and where CAKE was used as a shaper
> instead. This was for a fixed WiFi backhaul, though, and even so they<= br> > had to set the shaper quite a lot lower than the max bandwidth to get<= br> > reliable performance.
>
>>> Otherwise, this is a major source of latency *if*
>>> the WiFi link is faster than the downlink from the internet. >>
>> Huh? Slower, you mean?
>
> No, if the WiFi link is faster, the upstream link becomes the bottlene= ck
> and CAKE has work to do :)
>
>>> This
>>> depends on both the internet connection and the current rate e= ach WiFi
>>> station operates at, so it can vary wildly, and on very short = time
>>> scales.
>>
>> Sure... I was asking for the "if" in your statement abov= e - since this
>> is an operationally-oriented list: what do people see? What is the=
>> more common case?
>
> Right, well as you can probably tell that might not have been entirely=
> clear from your initial post ;)
>
>>> The extent to which this happens depends on where you are in t= he
>>> world; personally I've been bottlenecked on the WiFi link = ever since
>>> I got a fibre upstream (and with 802.11ax rates maxing at >= 1Gbps,
>>> maybe that'll change again?).
>>
>> THIS is what I was after :) one data point, cool - so far, so good= ...
>
> Ah, you're after anecdotes - well why didn't you say so? ;) >
> In that case I'll add that my own connection is the only one I'= ;ve come
> across where the WiFi link is *never* the bottleneck. In Denmark we ar= e
> finally (slowly) getting out of the dark ages as far as fibre
> deployments are concerned, but most operators will sell connections > capped at 100Mbps or 250Mbps, which is still less than the throughout = of
> a 802.11ac link with good signal conditions (my phone consistently get= s
> ~250-350 Mbps on a speedtest.net run).
>
> DSL connections tend to have awful latency, and are still quite common= ,
> but they are pretty easy to fix with CAKE. Cable connections likewise,=
> or so is my impression (those are not so common around these parts). >
> The worst are definitely LTE/mobile broadband connections. Wildly
> varying link speeds, and awful over-buffering; so you really have to > clamp them down to get anything useful out of CAKE.
>
> -Toke

_______________________________________________
Make-wifi-fast mailing list
M= ake-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wif= i-fast
--00000000000054b17305b1098a91-- --0000000000005d42a405b1098a4c Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQPwYJKoZIhvcNAQcCoIIQMDCCECwCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg2UMIIE6DCCA9CgAwIBAgIOSBtqCRO9gCTKXSLwFPMwDQYJKoZIhvcNAQELBQAwTDEgMB4GA1UE CxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMT Ckdsb2JhbFNpZ24wHhcNMTYwNjE1MDAwMDAwWhcNMjQwNjE1MDAwMDAwWjBdMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEzMDEGA1UEAxMqR2xvYmFsU2lnbiBQZXJzb25h bFNpZ24gMiBDQSAtIFNIQTI1NiAtIEczMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA tpZok2X9LAHsYqMNVL+Ly6RDkaKar7GD8rVtb9nw6tzPFnvXGeOEA4X5xh9wjx9sScVpGR5wkTg1 fgJIXTlrGESmaqXIdPRd9YQ+Yx9xRIIIPu3Jp/bpbiZBKYDJSbr/2Xago7sb9nnfSyjTSnucUcIP ZVChn6hKneVGBI2DT9yyyD3PmCEJmEzA8Y96qT83JmVH2GaPSSbCw0C+Zj1s/zqtKUbwE5zh8uuZ p4vC019QbaIOb8cGlzgvTqGORwK0gwDYpOO6QQdg5d03WvIHwTunnJdoLrfvqUg2vOlpqJmqR+nH 9lHS+bEstsVJtZieU1Pa+3LzfA/4cT7XA/pnwwIDAQABo4IBtTCCAbEwDgYDVR0PAQH/BAQDAgEG MGoGA1UdJQRjMGEGCCsGAQUFBwMCBggrBgEFBQcDBAYIKwYBBQUHAwkGCisGAQQBgjcUAgIGCisG AQQBgjcKAwQGCSsGAQQBgjcVBgYKKwYBBAGCNwoDDAYIKwYBBQUHAwcGCCsGAQUFBwMRMBIGA1Ud EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFGlygmIxZ5VEhXeRgMQENkmdewthMB8GA1UdIwQYMBaA FI/wS3+oLkUkrk1Q+mOai97i3Ru8MD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcwAYYiaHR0cDov L29jc3AyLmdsb2JhbHNpZ24uY29tL3Jvb3RyMzA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3Js Lmdsb2JhbHNpZ24uY29tL3Jvb3QtcjMuY3JsMGcGA1UdIARgMF4wCwYJKwYBBAGgMgEoMAwGCisG AQQBoDIBKAowQQYJKwYBBAGgMgFfMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNp Z24uY29tL3JlcG9zaXRvcnkvMA0GCSqGSIb3DQEBCwUAA4IBAQConc0yzHxn4gtQ16VccKNm4iXv 6rS2UzBuhxI3XDPiwihW45O9RZXzWNgVcUzz5IKJFL7+pcxHvesGVII+5r++9eqI9XnEKCILjHr2 DgvjKq5Jmg6bwifybLYbVUoBthnhaFB0WLwSRRhPrt5eGxMw51UmNICi/hSKBKsHhGFSEaJQALZy 4HL0EWduE6ILYAjX6BSXRDtHFeUPddb46f5Hf5rzITGLsn9BIpoOVrgS878O4JnfUWQi29yBfn75 HajifFvPC+uqn+rcVnvrpLgsLOYG/64kWX/FRH8+mhVe+mcSX3xsUpcxK9q9vLTVtroU/yJUmEC4 OcH5dQsbHBqjMIIDXzCCAkegAwIBAgILBAAAAAABIVhTCKIwDQYJKoZIhvcNAQELBQAwTDEgMB4G A1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNV BAMTCkdsb2JhbFNpZ24wHhcNMDkwMzE4MTAwMDAwWhcNMjkwMzE4MTAwMDAwWjBMMSAwHgYDVQQL ExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMK R2xvYmFsU2lnbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMwldpB5BngiFvXAg7aE yiie/QV2EcWtiHL8RgJDx7KKnQRfJMsuS+FggkbhUqsMgUdwbN1k0ev1LKMPgj0MK66X17YUhhB5 uzsTgHeMCOFJ0mpiLx9e+pZo34knlTifBtc+ycsmWQ1z3rDI6SYOgxXG71uL0gRgykmmKPZpO/bL yCiR5Z2KYVc3rHQU3HTgOu5yLy6c+9C7v/U9AOEGM+iCK65TpjoWc4zdQQ4gOsC0p6Hpsk+QLjJg 6VfLuQSSaGjlOCZgdbKfd/+RFO+uIEn8rUAVSNECMWEZXriX7613t2Saer9fwRPvm2L7DWzgVGkW qQPabumDk3F2xmmFghcCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8w HQYDVR0OBBYEFI/wS3+oLkUkrk1Q+mOai97i3Ru8MA0GCSqGSIb3DQEBCwUAA4IBAQBLQNvAUKr+ yAzv95ZURUm7lgAJQayzE4aGKAczymvmdLm6AC2upArT9fHxD4q/c2dKg8dEe3jgr25sbwMpjjM5 RcOO5LlXbKr8EpbsU8Yt5CRsuZRj+9xTaGdWPoO4zzUhw8lo/s7awlOqzJCK6fBdRoyV3XpYKBov Hd7NADdBj+1EbddTKJd+82cEHhXXipa0095MJ6RMG3NzdvQXmcIfeg7jLQitChws/zyrVQ4PkX42 68NXSb7hLi18YIvDQVETI53O9zJrlAGomecsMx86OyXShkDOOyyGeMlhLxS67ttVb9+E7gUJTb0o 2HLO02JQZR7rkpeDMdmztcpHWD9fMIIFQTCCBCmgAwIBAgIML7TfFWHfxluS5m0sMA0GCSqGSIb3 DQEBCwUAMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQD EypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMwHhcNMjAwODMxMDgw OTQ5WhcNMjIwOTAxMDgwOTQ5WjCBjDELMAkGA1UEBhMCSU4xEjAQBgNVBAgTCUthcm5hdGFrYTES MBAGA1UEBxMJQmFuZ2Fsb3JlMRYwFAYDVQQKEw1Ccm9hZGNvbSBJbmMuMRQwEgYDVQQDEwtCb2Ig TWNNYWhvbjEnMCUGCSqGSIb3DQEJARYYYm9iLm1jbWFob25AYnJvYWRjb20uY29tMIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzK1kauAQxAkeif97AJqv3QgJVDztfPjduswFJbcT0VbZ hr+E3gpqCnUm9TM4HZ2aviVSu9A9SkcpQBOaiOUPUkMrD6PoviodpwV2C5W+i7e/lexjsUdSuTZe h7GRKEOlhPnsW7RgKFR5+3Rrm1kAyEJ2x2ueyn3UhupxoNrYxZOss/+dPwQJ28kP7ICHBBJbcfmh doMU83sfJazBMsp+pMArApFdMwXOBL3dT4ZPkniaMilO+q2y7xQ212K+KlhaRWCWcImC/+pqZCyH /b1zx6vfBbdB/WD9+oi9rnXyd2ikVU2bvJ15VinB52kz7Kpj4A1e7zHgKxqNjFFycWXn2QIDAQAB o4IBzzCCAcswDgYDVR0PAQH/BAQDAgWgMIGeBggrBgEFBQcBAQSBkTCBjjBNBggrBgEFBQcwAoZB aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yc2hhMmcz b2NzcC5jcnQwPQYIKwYBBQUHMAGGMWh0dHA6Ly9vY3NwMi5nbG9iYWxzaWduLmNvbS9nc3BlcnNv bmFsc2lnbjJzaGEyZzMwTQYDVR0gBEYwRDBCBgorBgEEAaAyASgKMDQwMgYIKwYBBQUHAgEWJmh0 dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMAkGA1UdEwQCMAAwRAYDVR0fBD0w OzA5oDegNYYzaHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9nc3BlcnNvbmFsc2lnbjJzaGEyZzMu Y3JsMCMGA1UdEQQcMBqBGGJvYi5tY21haG9uQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEF BQcDBDAfBgNVHSMEGDAWgBRpcoJiMWeVRIV3kYDEBDZJnXsLYTAdBgNVHQ4EFgQUQ1s1Ocvee2Eh o0qY50aFmX9fHS0wDQYJKoZIhvcNAQELBQADggEBAFygcUiFPJ7uRXexxLYonpGCsztO0YcmuWCE jef/oS4oMYJyf+pv/dgdywaJ9U4FhYHg9wIPEcWwS1JpsUrEhx1zQ2JX8+HNHyuCiR11tgtzCr0z nxeMm3RAJPmM3cAy5mS7pDz1Ox081EqDQQzeesvMwwONjIpV1ORg17fpIHxz4KaKg8X5Yv6xYs0Z L+UVLQlmIJqoOHqzOelCjNTCSucYx/sYygzQcvUuX7qUgLOJq+o4JXhqu2dUJ6wu/NPApqyd8hSc YVBMvygG2Y3xBjxYC5Dr4+E1cmcOGEUQhkKo4cwv217+W4FXb4K4B/EM208EsUMXGTb7tV5zJ0rO IH4xggJvMIICawIBATBtMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNh MTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMCDC+0 3xVh38ZbkuZtLDANBglghkgBZQMEAgEFAKCB1DAvBgkqhkiG9w0BCQQxIgQgWkjrotreMSGTtA0t /kIvutxq6GQrNShQSiw6i8MR1bUwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0B CQUxDxcNMjAxMDA3MDAxMDE3WjBpBgkqhkiG9w0BCQ8xXDBaMAsGCWCGSAFlAwQBKjALBglghkgB ZQMEARYwCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMAsGCSqGSIb3DQEBCjALBgkqhkiG9w0BAQcw CwYJYIZIAWUDBAIBMA0GCSqGSIb3DQEBAQUABIIBAD8x30JiGeGcWCIBhXtCnb9hTJpYw1IFUWas t0osSqy8d87pWWvWXisC0MlBWmMroUMtT/BEN8OcGy1t0rjJazp9h1wluNhPICR4HgT/RcAEUPV1 nwrxv2XYTC/orS4j57zci7oF5eaoNtIzsNo9jhsjPtJ06jG1vWVMQmUXiPQrg1SyuFrxQyI7CnQ+ Ti5hZNUQxz3mGjAHQ7kQANSvpdZhA0bFT5ClBCnzjYM0Mg/h1uMQ2IteVJmzLTnAOfsi7uTCHYmq AXS4+p3jynNDZuuwLqwmrf3bez13tVMus5aSbK5oEToSxIlfJz7Tp5ER+2tTuaClta15W9vCy/Zo qmc= --0000000000005d42a405b1098a4c--