You might want to try iperf 2.0.14 --trip-times option to measure your links.  Below a run with a PC w/a 1G NIC connected to the 1G cable MODEM no WiFi, i.e. wired, full duplex socket.  The uplink tcp write latencies are larger than one second.  Downlinks just under 100 ms.  Slow down the uplink traffic to less than the congestion point and write to read latency drops to ~65 ms.  I think this suggests my CABLE MODEM is full of bloat.

UPLINK ONLY
--------------------

Server:
[rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25408
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58158 (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval        Transfer    Bandwidth    Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  4] 0.00-1.00 sec  1.25 MBytes  10.5 Mbits/sec  59.680/51.222/96.282/34.950 ms (10/131072) 80.9 KByte 21.96  305=305:0:0:0:0:0:0:0
[  4] 1.00-2.00 sec  1.12 MBytes  9.44 Mbits/sec  65.218/51.951/87.895/10.877 ms (9/131072) 79.6 KByte 18.09  262=262:0:0:0:0:0:0:0
[  4] 2.00-3.00 sec  1.25 MBytes  10.5 Mbits/sec  62.479/51.170/67.962/ 5.189 ms (10/131072) 76.3 KByte 20.98  289=289:0:0:0:0:0:0:0
[  4] 3.00-4.00 sec  1.12 MBytes  9.44 Mbits/sec  62.861/52.478/68.339/ 5.212 ms (9/131072) 76.7 KByte 18.77  262=262:0:0:0:0:0:0:0
[  4] 4.00-5.00 sec  1.25 MBytes  10.5 Mbits/sec  64.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23  292=292:0:0:0:0:0:0:0
[  4] 5.00-6.00 sec  1.14 MBytes  9.60 Mbits/sec  64.618/50.914/68.277/ 5.696 ms (9/133334) 80.2 KByte 18.57  259=259:0:0:0:0:0:0:0
[  4] 6.00-7.00 sec  1.23 MBytes  10.3 Mbits/sec  66.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29  279=279:0:0:0:0:0:0:0
[  4] 7.00-8.00 sec  1.16 MBytes  9.70 Mbits/sec  68.954/51.902/100.783/15.733 ms (9/134782) 86.6 KByte 17.59  275=274:1:0:0:0:0:0:0
[  4] 8.00-9.00 sec  1.22 MBytes  10.2 Mbits/sec  63.855/51.386/69.919/ 5.284 ms (9/141851) 84.4 KByte 19.99  283=283:0:0:0:0:0:0:0
[  4] 9.00-10.00 sec  1.18 MBytes  9.86 Mbits/sec  67.975/50.996/88.151/11.318 ms (10/123252) 78.0 KByte 18.13  275=275:0:0:0:0:0:0:0
[  4] 10.00-10.14 sec   205 KBytes  11.7 Mbits/sec  99.865/55.342/77.370/ 7.319 ms (2/104962) 97.6 KByte 14.61  43=43:0:0:0:0:0:0:0
[  4] 0.00-10.14 sec  12.1 MBytes  10.0 Mbits/sec  59.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04  2824=2823:1:0:0:0:0:0:0

Client:

[root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times -Z bbr -b 10m -e
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 48006 (1 flows)
Write buffer size:  128 KByte
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.35%enp2s0 port 58160 connected with 45.33.58.123 port 5001 (trip-times) (MSS=1448) (ct=13.05 ms)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  3] 0.00-1.00 sec  1.25 MBytes  10.5 Mbits/sec  11/0          0      135K/49251 us  26.61
[  3] 1.00-2.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0      141K/24791 us  52.87
[  3] 2.00-3.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0      135K/24973 us  47.24
[  3] 3.00-4.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0      158K/25021 us  52.38
[  3] 4.00-5.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0      141K/21247 us  55.52
[  3] 5.00-6.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0      144K/23535 us  55.69
[  3] 6.00-7.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0      135K/27130 us  43.48
[  3] 7.00-8.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0      135K/24813 us  52.82
[  3] 8.00-9.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0      155K/22697 us  51.97
[  3] 9.00-10.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0      124K/21954 us  59.70
[  3] 10.00-10.07 sec   256 KBytes  31.6 Mbits/sec  2/0          0      124K/21954 us  179.64
[  3] 0.00-10.07 sec  12.3 MBytes  10.2 Mbits/sec  99/0          0      124K/21954 us  58.12

FULL DUPLEX:
---------------------
Server:

[rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25379
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58152 (full-duplex) (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval        Transfer    Bandwidth    Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  4] 0.00-1.00 sec  1.05 MBytes  8.79 Mbits/sec  547.494/68.304/904.670/301.250 ms (8/137379) 9.75 MByte 2.01  345=344:1:0:0:0:0:0:0
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  4] 0.00-1.00 sec  40.3 MBytes   338 Mbits/sec  323/0          0     1267K/20319 us  2077.13
[ ID] Interval       Transfer     Bandwidth
[FD4] 0.00-1.00 sec  41.3 MBytes   346 Mbits/sec
[  4] 1.00-2.00 sec  1.34 MBytes  11.3 Mbits/sec  1307.675/904.670/1454.073/145.211 ms (11/127950) 2.30 MByte 1.08  277=262:5:4:6:0:0:0:0
[  4] 1.00-2.00 sec  38.8 MBytes   325 Mbits/sec  310/0          0     1001K/24428 us  1663.35
[FD4] 1.00-2.00 sec  40.1 MBytes   336 Mbits/sec
[  4] 2.00-3.00 sec  38.0 MBytes   319 Mbits/sec  304/0          0     1156K/26355 us  1511.89
[  4] 2.00-3.00 sec  1.33 MBytes  11.1 Mbits/sec  1298.825/1139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07  372=366:2:1:3:0:0:0:0
[FD4] 2.00-3.00 sec  39.3 MBytes   330 Mbits/sec
[  4] 3.00-4.00 sec  1.32 MBytes  11.1 Mbits/sec  1302.239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06  217=199:4:8:4:2:0:0:0
[  4] 3.00-4.00 sec  39.6 MBytes   332 Mbits/sec  317/0          0      970K/19064 us  2179.49
[FD4] 3.00-4.00 sec  40.9 MBytes   343 Mbits/sec
[  4] 4.00-5.00 sec  38.5 MBytes   323 Mbits/sec  308/0          0      998K/20599 us  1959.81
[  4] 4.00-5.00 sec  1.39 MBytes  11.7 Mbits/sec  1310.588/1172.769/1233.522/21.090 ms (11/132426) 1.79 MByte 1.11  223=196:18:5:4:0:0:0:0
[FD4] 4.00-5.00 sec  39.9 MBytes   335 Mbits/sec
[  4] 5.00-6.00 sec  38.6 MBytes   324 Mbits/sec  309/0          0     1340K/17717 us  2286.01
[  4] 5.00-6.00 sec  1.33 MBytes  11.2 Mbits/sec  1305.731/1173.179/1220.196/17.125 ms (11/127160) 1.72 MByte 1.07  158=136:10:5:6:1:0:0:0
[FD4] 5.00-6.00 sec  40.0 MBytes   335 Mbits/sec
[  4] 6.00-7.00 sec  1.36 MBytes  11.4 Mbits/sec  1301.735/1162.410/1216.840/20.348 ms (10/142193) 1.87 MByte 1.09  140=116:7:8:6:3:0:0:0
[  4] 6.00-7.00 sec  39.4 MBytes   330 Mbits/sec  315/0          0     1074K/20040 us  2060.26
[FD4] 6.00-7.00 sec  40.7 MBytes   342 Mbits/sec
[  4] 7.00-8.00 sec  1.35 MBytes  11.3 Mbits/sec  1291.605/1160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10  216=198:5:6:7:0:0:0:0
[  4] 7.00-8.00 sec  38.9 MBytes   326 Mbits/sec  311/0          0     1074K/30895 us  1319.42
[FD4] 7.00-8.00 sec  40.2 MBytes   337 Mbits/sec
[  4] 8.00-9.00 sec  37.4 MBytes   314 Mbits/sec  299/0          0     1264K/29185 us  1342.83
[  4] 8.00-9.00 sec  1.27 MBytes  10.7 Mbits/sec  1326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.01  110=86:9:7:5:1:0:2:0
[FD4] 8.00-9.00 sec  38.6 MBytes   324 Mbits/sec
[  4] 9.00-10.00 sec  1.34 MBytes  11.2 Mbits/sec  1312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByte 1.07  124=101:12:6:2:1:0:2:0
[  4] 9.00-10.00 sec  39.2 MBytes   329 Mbits/sec  314/0          0      967K/21976 us  1872.80
[FD4] 9.00-10.00 sec  40.6 MBytes   341 Mbits/sec
[  4] 10.00-10.00 sec   256 KBytes   477 Mbits/sec  2/0          0      967K/21976 us  2711.06
[  4] 0.00-10.00 sec   389 MBytes   326 Mbits/sec  3112/0          0      967K/21976 us  1854.69
[  4] 10.00-11.00 sec   963 KBytes  7.89 Mbits/sec  1624.535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61  215=210:3:0:0:1:1:0:0
[  4] 11.00-11.41 sec   622 KBytes  12.6 Mbits/sec  1780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0.88  29=11:11:4:3:0:0:0:0
[  4] 0.00-11.41 sec  14.6 MBytes  10.8 Mbits/sec  1184.903/68.304/1548.718/215.041 ms (117/131072) 1.52 MByte 1.13  2426=2225:87:54:46:9:1:4:0
[FD4] 10.00-1602028944.71 sec  1.80 MBytes  0.009 bits/sec
[FD4] 0.00-1602028944.71 sec   404 MBytes  2.11 bits/sec

Client

[root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times --full-duplex -Z bbr
WARNING: tcp congestion control will only be applied on transmit traffic, use -Z on the server
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 47994 (1 flows)
Write buffer size:  128 KByte
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.35%enp2s0 port 58154 connected with 45.33.58.123 port 5001 (full-duplex) (trip-times) (MSS=1448) (ct=12.42 ms)
[ ID] Interval        Transfer    Bandwidth    Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  3] 0.00-1.00 sec  35.8 MBytes   301 Mbits/sec  89.806/29.604/103.341/16.204 ms (286/131398) 3.58 MByte 418.46  17764=17726:35:3:0:0:0:0:0
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  3] 0.00-1.00 sec  2.50 MBytes  21.0 Mbits/sec  21/0         91       62K/44390 us  59.06
[ ID] Interval       Transfer     Bandwidth
[FD3] 0.00-1.00 sec  38.3 MBytes   322 Mbits/sec
[  3] 1.00-2.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          8       56K/36570 us  35.84
[  3] 1.00-2.00 sec  38.7 MBytes   325 Mbits/sec  94.258/61.108/104.117/ 9.454 ms (310/130882) 3.63 MByte 430.45  20775=20768:5:2:0:0:0:0:0
[FD3] 1.00-2.00 sec  39.9 MBytes   335 Mbits/sec
[  3] 2.00-3.00 sec  38.6 MBytes   324 Mbits/sec  94.554/62.312/105.451/ 9.324 ms (308/131379) 3.69 MByte 427.96  18929=18913:15:1:0:0:0:0:0
[  3] 2.00-3.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         12       65K/54911 us  26.26
[FD3] 2.00-3.00 sec  40.0 MBytes   335 Mbits/sec
[  3] 3.00-4.00 sec  37.3 MBytes   313 Mbits/sec  95.362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71  17567=17537:27:3:0:0:0:0:0
[  3] 3.00-4.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         20       67K/46946 us  30.71
[FD3] 3.00-4.00 sec  38.6 MBytes   324 Mbits/sec
[  3] 4.00-5.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0         51       59K/67662 us  19.37
[  3] 4.00-5.00 sec  38.4 MBytes   322 Mbits/sec  90.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.99  19095=19077:16:2:0:0:0:0:0
[FD3] 4.00-5.00 sec  39.6 MBytes   333 Mbits/sec
[  3] 5.00-6.00 sec  38.5 MBytes   323 Mbits/sec  88.832/64.621/102.534/ 9.742 ms (308/130979) 3.40 MByte 454.13  17674=17657:17:0:0:0:0:0:0
[  3] 5.00-6.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         48       65K/64456 us  22.37
[FD3] 5.00-6.00 sec  39.8 MBytes   334 Mbits/sec
[  3] 6.00-7.00 sec  38.4 MBytes   322 Mbits/sec  87.121/62.694/102.124/10.260 ms (307/131199) 3.24 MByte 462.33  19853=19852:1:0:0:0:0:0:0
[  3] 6.00-7.00 sec  3.75 MBytes  31.5 Mbits/sec  30/0         59       67K/151435 us  25.97
[FD3] 6.00-7.00 sec  42.2 MBytes   354 Mbits/sec
[  3] 7.00-8.00 sec  38.5 MBytes   323 Mbits/sec  87.936/62.694/102.226/10.016 ms (308/131222) 3.41 MByte 459.61  19067=19052:15:0:0:0:0:0:0
[  3] 7.00-8.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0         60       73K/56735 us  20.79
[FD3] 7.00-8.00 sec  39.7 MBytes   333 Mbits/sec
[  3] 8.00-9.00 sec  35.9 MBytes   301 Mbits/sec  91.441/56.797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54  18900=18881:10:2:0:0:0:0:7
[  3] 8.00-9.00 sec  1.50 MBytes  12.6 Mbits/sec  13/0         26       82K/49077 us  32.05
[FD3] 8.00-9.00 sec  37.4 MBytes   314 Mbits/sec
[  3] 9.00-10.00 sec  36.3 MBytes   305 Mbits/sec  90.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88  17888=17871:17:0:0:0:0:0:0
[  3] 10.00-10.07 sec  2.72 MBytes   310 Mbits/sec  91.852/69.979/102.633/ 8.280 ms (22/129636) 5.96 MByte 421.94  1418=1414:4:0:0:0:0:0:0
[  3] 0.00-10.07 sec   379 MBytes   316 Mbits/sec  90.712/29.604/148.892/12.195 ms (3033/131072) 3.41 MByte 435.04  188930=188748:162:13:0:0:0:0:7
[  3] 9.00-10.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0          0        5K/55760 us  25.86
[FD3] 9.00-10.00 sec  40.4 MBytes   339 Mbits/sec
[  3] 10.00-10.49 sec   223 KBytes  3.69 Mbits/sec  2/0          0        5K/55760 us  8.28
[  3] 0.00-10.49 sec  17.1 MBytes  13.7 Mbits/sec  139/0        424        5K/55760 us  30.63
[FD3] 10.00-10.49 sec   223 KBytes  3.77 Mbits/sec
[FD3] 0.00-10.49 sec   396 MBytes   317 Mbits/sec

Bob

On Tue, Oct 6, 2020 at 6:24 AM Michael Welzl <michawe@ifi.uio.no> wrote:
Thanks a lot - just the info I was looking for  (also from others who have responded in the meantime, thanks!)


> On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Michael Welzl <michawe@ifi.uio.no> writes:
>
>> Hi, and thanks for a quick answer!
>>
>> But, it's not quite what I was looking for.... see below:
>>
>>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Michael Welzl <michawe@ifi.uio.no> writes:
>>>
>>>> Hi all,
>>>>
>>>> A simple question to y'all who spent so much time on Cake and things
>>>> ... in a household using WiFi, which buffer is usually bloated? Where
>>>> does the latency really come from?
>>>>
>>>> Is it:
>>>> 1. the access point's downlink queue, feeding into the WiFi network,
>>>
>>> This we mostly fixed, but only if you're on a recent OpenWrt with the
>>> right WiFi drivers.
>>
>> Well okay... I was curious about where the bottleneck is. I can
>> translate my question into: "if Cake is installed everywhere, where
>> does it have the most work to do?".
>
> Well, CAKE only runs on the upstream link, so that's where it does its
> work. The software shaper model doesn't really work that well on WiFi,
> so we generally encourage people to just run fixed drivers there
> instead. That being said I have heard of one or two WiFi deployments
> where that was not an option, and where CAKE was used as a shaper
> instead. This was for a fixed WiFi backhaul, though, and even so they
> had to set the shaper quite a lot lower than the max bandwidth to get
> reliable performance.
>
>>> Otherwise, this is a major source of latency *if*
>>> the WiFi link is faster than the downlink from the internet.
>>
>> Huh? Slower, you mean?
>
> No, if the WiFi link is faster, the upstream link becomes the bottleneck
> and CAKE has work to do :)
>
>>> This
>>> depends on both the internet connection and the current rate each WiFi
>>> station operates at, so it can vary wildly, and on very short time
>>> scales.
>>
>> Sure... I was asking for the "if" in your statement above - since this
>> is an operationally-oriented list: what do people see? What is the
>> more common case?
>
> Right, well as you can probably tell that might not have been entirely
> clear from your initial post ;)
>
>>> The extent to which this happens depends on where you are in the
>>> world; personally I've been bottlenecked on the WiFi link ever since
>>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps,
>>> maybe that'll change again?).
>>
>> THIS is what I was after :) one data point, cool - so far, so good...
>
> Ah, you're after anecdotes - well why didn't you say so? ;)
>
> In that case I'll add that my own connection is the only one I've come
> across where the WiFi link is *never* the bottleneck. In Denmark we are
> finally (slowly) getting out of the dark ages as far as fibre
> deployments are concerned, but most operators will sell connections
> capped at 100Mbps or 250Mbps, which is still less than the throughout of
> a 802.11ac link with good signal conditions (my phone consistently gets
> ~250-350 Mbps on a speedtest.net run).
>
> DSL connections tend to have awful latency, and are still quite common,
> but they are pretty easy to fix with CAKE. Cable connections likewise,
> or so is my impression (those are not so common around these parts).
>
> The worst are definitely LTE/mobile broadband connections. Wildly
> varying link speeds, and awful over-buffering; so you really have to
> clamp them down to get anything useful out of CAKE.
>
> -Toke

_______________________________________________
Make-wifi-fast mailing list
Make-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wifi-fast