[Make-wifi-fast] Where is the bloat in WiFi?
Bob McMahon
bob.mcmahon at broadcom.com
Tue Oct 6 20:10:06 EDT 2020
You might want to try iperf 2.0.14 --trip-times option to measure your
links. Below a run with a PC w/a 1G NIC connected to the 1G cable MODEM no
WiFi, i.e. wired, full duplex socket. The uplink tcp write latencies are
larger than one second. Downlinks just under 100 ms. Slow down the uplink
traffic to less than the congestion point and write to read latency drops
to ~65 ms. I think this suggests my CABLE MODEM is full of bloat.
UPLINK ONLY
--------------------
Server:
[rjmcmahon at bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25408
Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port
58158 (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval Transfer Bandwidth Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
[ 4] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec
59.680/51.222/96.282/34.950 ms (10/131072) 80.9 KByte 21.96
305=305:0:0:0:0:0:0:0
[ 4] 1.00-2.00 sec 1.12 MBytes 9.44 Mbits/sec
65.218/51.951/87.895/10.877 ms (9/131072) 79.6 KByte 18.09
262=262:0:0:0:0:0:0:0
[ 4] 2.00-3.00 sec 1.25 MBytes 10.5 Mbits/sec 62.479/51.170/67.962/
5.189 ms (10/131072) 76.3 KByte 20.98 289=289:0:0:0:0:0:0:0
[ 4] 3.00-4.00 sec 1.12 MBytes 9.44 Mbits/sec 62.861/52.478/68.339/
5.212 ms (9/131072) 76.7 KByte 18.77 262=262:0:0:0:0:0:0:0
[ 4] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec
64.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23
292=292:0:0:0:0:0:0:0
[ 4] 5.00-6.00 sec 1.14 MBytes 9.60 Mbits/sec 64.618/50.914/68.277/
5.696 ms (9/133334) 80.2 KByte 18.57 259=259:0:0:0:0:0:0:0
[ 4] 6.00-7.00 sec 1.23 MBytes 10.3 Mbits/sec
66.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29
279=279:0:0:0:0:0:0:0
[ 4] 7.00-8.00 sec 1.16 MBytes 9.70 Mbits/sec
68.954/51.902/100.783/15.733 ms (9/134782) 86.6 KByte 17.59
275=274:1:0:0:0:0:0:0
[ 4] 8.00-9.00 sec 1.22 MBytes 10.2 Mbits/sec 63.855/51.386/69.919/
5.284 ms (9/141851) 84.4 KByte 19.99 283=283:0:0:0:0:0:0:0
[ 4] 9.00-10.00 sec 1.18 MBytes 9.86 Mbits/sec
67.975/50.996/88.151/11.318 ms (10/123252) 78.0 KByte 18.13
275=275:0:0:0:0:0:0:0
[ 4] 10.00-10.14 sec 205 KBytes 11.7 Mbits/sec 99.865/55.342/77.370/
7.319 ms (2/104962) 97.6 KByte 14.61 43=43:0:0:0:0:0:0:0
[ 4] 0.00-10.14 sec 12.1 MBytes 10.0 Mbits/sec
59.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04
2824=2823:1:0:0:0:0:0:0
Client:
[root at localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1
--trip-times -Z bbr -b 10m -e
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 48006 (1
flows)
Write buffer size: 128 KByte
TCP congestion control set to bbr
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.35%enp2s0 port 58160 connected with 45.33.58.123 port
5001 (trip-times) (MSS=1448) (ct=13.05 ms)
[ ID] Interval Transfer Bandwidth Write/Err Rtry
Cwnd/RTT NetPwr
[ 3] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 11/0 0
135K/49251 us 26.61
[ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0
141K/24791 us 52.87
[ 3] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
135K/24973 us 47.24
[ 3] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0
158K/25021 us 52.38
[ 3] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
141K/21247 us 55.52
[ 3] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0
144K/23535 us 55.69
[ 3] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
135K/27130 us 43.48
[ 3] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0
135K/24813 us 52.82
[ 3] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0
155K/22697 us 51.97
[ 3] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0
124K/21954 us 59.70
[ 3] 10.00-10.07 sec 256 KBytes 31.6 Mbits/sec 2/0 0
124K/21954 us 179.64
[ 3] 0.00-10.07 sec 12.3 MBytes 10.2 Mbits/sec 99/0 0
124K/21954 us 58.12
FULL DUPLEX:
---------------------
Server:
[rjmcmahon at bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25379
Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port
58152 (full-duplex) (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval Transfer Bandwidth Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
[ 4] 0.00-1.00 sec 1.05 MBytes 8.79 Mbits/sec
547.494/68.304/904.670/301.250 ms (8/137379) 9.75 MByte 2.01
345=344:1:0:0:0:0:0:0
[ ID] Interval Transfer Bandwidth Write/Err Rtry
Cwnd/RTT NetPwr
[ 4] 0.00-1.00 sec 40.3 MBytes 338 Mbits/sec 323/0 0
1267K/20319 us 2077.13
[ ID] Interval Transfer Bandwidth
[FD4] 0.00-1.00 sec 41.3 MBytes 346 Mbits/sec
[ 4] 1.00-2.00 sec 1.34 MBytes 11.3 Mbits/sec
1307.675/904.670/1454.073/145.211
ms (11/127950) 2.30 MByte 1.08 277=262:5:4:6:0:0:0:0
[ 4] 1.00-2.00 sec 38.8 MBytes 325 Mbits/sec 310/0 0
1001K/24428 us 1663.35
[FD4] 1.00-2.00 sec 40.1 MBytes 336 Mbits/sec
[ 4] 2.00-3.00 sec 38.0 MBytes 319 Mbits/sec 304/0 0
1156K/26355 us 1511.89
[ 4] 2.00-3.00 sec 1.33 MBytes 11.1 Mbits/sec
1298.825/1139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07
372=366:2:1:3:0:0:0:0
[FD4] 2.00-3.00 sec 39.3 MBytes 330 Mbits/sec
[ 4] 3.00-4.00 sec 1.32 MBytes 11.1 Mbits/sec
1302.239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06
217=199:4:8:4:2:0:0:0
[ 4] 3.00-4.00 sec 39.6 MBytes 332 Mbits/sec 317/0 0
970K/19064 us 2179.49
[FD4] 3.00-4.00 sec 40.9 MBytes 343 Mbits/sec
[ 4] 4.00-5.00 sec 38.5 MBytes 323 Mbits/sec 308/0 0
998K/20599 us 1959.81
[ 4] 4.00-5.00 sec 1.39 MBytes 11.7 Mbits/sec
1310.588/1172.769/1233.522/21.090 ms (11/132426) 1.79 MByte 1.11
223=196:18:5:4:0:0:0:0
[FD4] 4.00-5.00 sec 39.9 MBytes 335 Mbits/sec
[ 4] 5.00-6.00 sec 38.6 MBytes 324 Mbits/sec 309/0 0
1340K/17717 us 2286.01
[ 4] 5.00-6.00 sec 1.33 MBytes 11.2 Mbits/sec
1305.731/1173.179/1220.196/17.125 ms (11/127160) 1.72 MByte 1.07
158=136:10:5:6:1:0:0:0
[FD4] 5.00-6.00 sec 40.0 MBytes 335 Mbits/sec
[ 4] 6.00-7.00 sec 1.36 MBytes 11.4 Mbits/sec
1301.735/1162.410/1216.840/20.348 ms (10/142193) 1.87 MByte 1.09
140=116:7:8:6:3:0:0:0
[ 4] 6.00-7.00 sec 39.4 MBytes 330 Mbits/sec 315/0 0
1074K/20040 us 2060.26
[FD4] 6.00-7.00 sec 40.7 MBytes 342 Mbits/sec
[ 4] 7.00-8.00 sec 1.35 MBytes 11.3 Mbits/sec
1291.605/1160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10
216=198:5:6:7:0:0:0:0
[ 4] 7.00-8.00 sec 38.9 MBytes 326 Mbits/sec 311/0 0
1074K/30895 us 1319.42
[FD4] 7.00-8.00 sec 40.2 MBytes 337 Mbits/sec
[ 4] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec 299/0 0
1264K/29185 us 1342.83
[ 4] 8.00-9.00 sec 1.27 MBytes 10.7 Mbits/sec
1326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.01
110=86:9:7:5:1:0:2:0
[FD4] 8.00-9.00 sec 38.6 MBytes 324 Mbits/sec
[ 4] 9.00-10.00 sec 1.34 MBytes 11.2 Mbits/sec
1312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByte 1.07
124=101:12:6:2:1:0:2:0
[ 4] 9.00-10.00 sec 39.2 MBytes 329 Mbits/sec 314/0 0
967K/21976 us 1872.80
[FD4] 9.00-10.00 sec 40.6 MBytes 341 Mbits/sec
[ 4] 10.00-10.00 sec 256 KBytes 477 Mbits/sec 2/0 0
967K/21976 us 2711.06
[ 4] 0.00-10.00 sec 389 MBytes 326 Mbits/sec 3112/0 0
967K/21976 us 1854.69
[ 4] 10.00-11.00 sec 963 KBytes 7.89 Mbits/sec
1624.535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61
215=210:3:0:0:1:1:0:0
[ 4] 11.00-11.41 sec 622 KBytes 12.6 Mbits/sec
1780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0.88
29=11:11:4:3:0:0:0:0
[ 4] 0.00-11.41 sec 14.6 MBytes 10.8 Mbits/sec
1184.903/68.304/1548.718/215.041 ms (117/131072) 1.52 MByte 1.13
2426=2225:87:54:46:9:1:4:0
[FD4] 10.00-1602028944.71 sec 1.80 MBytes 0.009 bits/sec
[FD4] 0.00-1602028944.71 sec 404 MBytes 2.11 bits/sec
Client
[root at localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1
--trip-times --full-duplex -Z bbr
WARNING: tcp congestion control will only be applied on transmit traffic,
use -Z on the server
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 47994 (1
flows)
Write buffer size: 128 KByte
TCP congestion control set to bbr
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.35%enp2s0 port 58154 connected with 45.33.58.123 port
5001 (full-duplex) (trip-times) (MSS=1448) (ct=12.42 ms)
[ ID] Interval Transfer Bandwidth Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
[ 3] 0.00-1.00 sec 35.8 MBytes 301 Mbits/sec
89.806/29.604/103.341/16.204 ms (286/131398) 3.58 MByte 418.46
17764=17726:35:3:0:0:0:0:0
[ ID] Interval Transfer Bandwidth Write/Err Rtry
Cwnd/RTT NetPwr
[ 3] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec 21/0 91
62K/44390 us 59.06
[ ID] Interval Transfer Bandwidth
[FD3] 0.00-1.00 sec 38.3 MBytes 322 Mbits/sec
[ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 8
56K/36570 us 35.84
[ 3] 1.00-2.00 sec 38.7 MBytes 325 Mbits/sec 94.258/61.108/104.117/
9.454 ms (310/130882) 3.63 MByte 430.45 20775=20768:5:2:0:0:0:0:0
[FD3] 1.00-2.00 sec 39.9 MBytes 335 Mbits/sec
[ 3] 2.00-3.00 sec 38.6 MBytes 324 Mbits/sec 94.554/62.312/105.451/
9.324 ms (308/131379) 3.69 MByte 427.96 18929=18913:15:1:0:0:0:0:0
[ 3] 2.00-3.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 12
65K/54911 us 26.26
[FD3] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec
[ 3] 3.00-4.00 sec 37.3 MBytes 313 Mbits/sec
95.362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71
17567=17537:27:3:0:0:0:0:0
[ 3] 3.00-4.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 20
67K/46946 us 30.71
[FD3] 3.00-4.00 sec 38.6 MBytes 324 Mbits/sec
[ 3] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 51
59K/67662 us 19.37
[ 3] 4.00-5.00 sec 38.4 MBytes 322 Mbits/sec
90.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.99
19095=19077:16:2:0:0:0:0:0
[FD3] 4.00-5.00 sec 39.6 MBytes 333 Mbits/sec
[ 3] 5.00-6.00 sec 38.5 MBytes 323 Mbits/sec 88.832/64.621/102.534/
9.742 ms (308/130979) 3.40 MByte 454.13 17674=17657:17:0:0:0:0:0:0
[ 3] 5.00-6.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 48
65K/64456 us 22.37
[FD3] 5.00-6.00 sec 39.8 MBytes 334 Mbits/sec
[ 3] 6.00-7.00 sec 38.4 MBytes 322 Mbits/sec
87.121/62.694/102.124/10.260 ms (307/131199) 3.24 MByte 462.33
19853=19852:1:0:0:0:0:0:0
[ 3] 6.00-7.00 sec 3.75 MBytes 31.5 Mbits/sec 30/0 59
67K/151435 us 25.97
[FD3] 6.00-7.00 sec 42.2 MBytes 354 Mbits/sec
[ 3] 7.00-8.00 sec 38.5 MBytes 323 Mbits/sec
87.936/62.694/102.226/10.016 ms (308/131222) 3.41 MByte 459.61
19067=19052:15:0:0:0:0:0:0
[ 3] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 60
73K/56735 us 20.79
[FD3] 7.00-8.00 sec 39.7 MBytes 333 Mbits/sec
[ 3] 8.00-9.00 sec 35.9 MBytes 301 Mbits/sec
91.441/56.797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54
18900=18881:10:2:0:0:0:0:7
[ 3] 8.00-9.00 sec 1.50 MBytes 12.6 Mbits/sec 13/0 26
82K/49077 us 32.05
[FD3] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec
[ 3] 9.00-10.00 sec 36.3 MBytes 305 Mbits/sec
90.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88
17888=17871:17:0:0:0:0:0:0
[ 3] 10.00-10.07 sec 2.72 MBytes 310 Mbits/sec 91.852/69.979/102.633/
8.280 ms (22/129636) 5.96 MByte 421.94 1418=1414:4:0:0:0:0:0:0
[ 3] 0.00-10.07 sec 379 MBytes 316 Mbits/sec
90.712/29.604/148.892/12.195 ms (3033/131072) 3.41 MByte 435.04
188930=188748:162:13:0:0:0:0:7
[ 3] 9.00-10.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 0
5K/55760 us 25.86
[FD3] 9.00-10.00 sec 40.4 MBytes 339 Mbits/sec
[ 3] 10.00-10.49 sec 223 KBytes 3.69 Mbits/sec 2/0 0
5K/55760 us 8.28
[ 3] 0.00-10.49 sec 17.1 MBytes 13.7 Mbits/sec 139/0 424
5K/55760 us 30.63
[FD3] 10.00-10.49 sec 223 KBytes 3.77 Mbits/sec
[FD3] 0.00-10.49 sec 396 MBytes 317 Mbits/sec
Bob
On Tue, Oct 6, 2020 at 6:24 AM Michael Welzl <michawe at ifi.uio.no> wrote:
> Thanks a lot - just the info I was looking for (also from others who have
> responded in the meantime, thanks!)
>
>
> > On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen <toke at toke.dk> wrote:
> >
> > Michael Welzl <michawe at ifi.uio.no> writes:
> >
> >> Hi, and thanks for a quick answer!
> >>
> >> But, it's not quite what I was looking for.... see below:
> >>
> >>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke at toke.dk> wrote:
> >>>
> >>> Michael Welzl <michawe at ifi.uio.no> writes:
> >>>
> >>>> Hi all,
> >>>>
> >>>> A simple question to y'all who spent so much time on Cake and things
> >>>> ... in a household using WiFi, which buffer is usually bloated? Where
> >>>> does the latency really come from?
> >>>>
> >>>> Is it:
> >>>> 1. the access point's downlink queue, feeding into the WiFi network,
> >>>
> >>> This we mostly fixed, but only if you're on a recent OpenWrt with the
> >>> right WiFi drivers.
> >>
> >> Well okay... I was curious about where the bottleneck is. I can
> >> translate my question into: "if Cake is installed everywhere, where
> >> does it have the most work to do?".
> >
> > Well, CAKE only runs on the upstream link, so that's where it does its
> > work. The software shaper model doesn't really work that well on WiFi,
> > so we generally encourage people to just run fixed drivers there
> > instead. That being said I have heard of one or two WiFi deployments
> > where that was not an option, and where CAKE was used as a shaper
> > instead. This was for a fixed WiFi backhaul, though, and even so they
> > had to set the shaper quite a lot lower than the max bandwidth to get
> > reliable performance.
> >
> >>> Otherwise, this is a major source of latency *if*
> >>> the WiFi link is faster than the downlink from the internet.
> >>
> >> Huh? Slower, you mean?
> >
> > No, if the WiFi link is faster, the upstream link becomes the bottleneck
> > and CAKE has work to do :)
> >
> >>> This
> >>> depends on both the internet connection and the current rate each WiFi
> >>> station operates at, so it can vary wildly, and on very short time
> >>> scales.
> >>
> >> Sure... I was asking for the "if" in your statement above - since this
> >> is an operationally-oriented list: what do people see? What is the
> >> more common case?
> >
> > Right, well as you can probably tell that might not have been entirely
> > clear from your initial post ;)
> >
> >>> The extent to which this happens depends on where you are in the
> >>> world; personally I've been bottlenecked on the WiFi link ever since
> >>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps,
> >>> maybe that'll change again?).
> >>
> >> THIS is what I was after :) one data point, cool - so far, so good...
> >
> > Ah, you're after anecdotes - well why didn't you say so? ;)
> >
> > In that case I'll add that my own connection is the only one I've come
> > across where the WiFi link is *never* the bottleneck. In Denmark we are
> > finally (slowly) getting out of the dark ages as far as fibre
> > deployments are concerned, but most operators will sell connections
> > capped at 100Mbps or 250Mbps, which is still less than the throughout of
> > a 802.11ac link with good signal conditions (my phone consistently gets
> > ~250-350 Mbps on a speedtest.net run).
> >
> > DSL connections tend to have awful latency, and are still quite common,
> > but they are pretty easy to fix with CAKE. Cable connections likewise,
> > or so is my impression (those are not so common around these parts).
> >
> > The worst are definitely LTE/mobile broadband connections. Wildly
> > varying link speeds, and awful over-buffering; so you really have to
> > clamp them down to get anything useful out of CAKE.
> >
> > -Toke
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/make-wifi-fast/attachments/20201006/a710d090/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4163 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.bufferbloat.net/pipermail/make-wifi-fast/attachments/20201006/a710d090/attachment-0001.bin>
More information about the Make-wifi-fast
mailing list