[Make-wifi-fast] Where is the bloat in WiFi?

[Make-wifi-fast] Where is the bloat in WiFi?

[Michael Welzl]

Hi all,

A simple question to y'all who spent so much time on Cake and things ...   in a household using WiFi, which buffer is usually bloated? Where does the latency really come from?

Is it:
1. the access point's downlink queue, feeding into the WiFi network,
2. the modem's downlink queue, feeding into the access point,
3. the modem's uplink queue,
4. the access point's uplink queue towards the modem   (hm, that seems silly, surely the AP-modem connection is fast... so perhaps, instead: the queue in the host, as it wants to send data towards the access point)

or is it a combination of these?

I guess that, with openwrt, Cake is operating on the queue that's feeding the wifi network, as the modem's queue is out of its control... so: is this where the bottleneck usually is?

Just wondering about your views and experiences.

Thanks in advance!

Cheers,
Michael

--
PS: my personal guess: 1 and 3 above are the most common. But that's *pure* guesswork!

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Toke Høiland-Jørgensen]

Michael Welzl <michawe@ifi.uio.no> writes:

> Hi all,
>
> A simple question to y'all who spent so much time on Cake and things
> ... in a household using WiFi, which buffer is usually bloated? Where
> does the latency really come from?
>
> Is it:
> 1. the access point's downlink queue, feeding into the WiFi network,

This we mostly fixed, but only if you're on a recent OpenWrt with the
right WiFi drivers. Otherwise, this is a major source of latency *if*
the WiFi link is faster than the downlink from the internet. This
depends on both the internet connection and the current rate each WiFi
station operates at, so it can vary wildly, and on very short time
scales.

> 2. the modem's downlink queue, feeding into the access point,

If your internet (downlink) connection is slower than your WiFi link,
this is where you'll get the queueing.

> 3. the modem's uplink queue,

As above, but in the other direction - but as uplinks tend to be
asymmetric, this direction is often more of a problem.

> 4. the access point's uplink queue towards the modem   (hm, that seems
> silly, surely the AP-modem connection is fast... so perhaps, instead:
> the queue in the host, as it wants to send data towards the access
> point)

Yeah, that would be in the host; but host drivers can suffer from severe
bufferbloat as well, especially as rates drop (since the buffers are
often tuned for the maximum throughput the device can deliver in
best-case signal conditions).

> or is it a combination of these?

Usually it's a combination; especially since the WiFi capacity varies
wildly with signal conditions (as devices move around relative to the
AP), general link usage (more devices active mean less available
capacity for each device, exacerbated by airtime unfairness), and
interference. Also there are things like excessive retries causing HoL
blocking.

> I guess that, with openwrt, Cake is operating on the queue that's
> feeding the wifi network, as the modem's queue is out of its
> control... so: is this where the bottleneck usually is?

It certainly used to be; but as uplink connection speeds improve, the
bottleneck moves to the WiFi link. The extent to which this happens
depends on where you are in the world; personally I've been bottlenecked
on the WiFi link ever since I got a fibre upstream (and with 802.11ax
rates maxing at >1Gbps, maybe that'll change again?).

-Toke

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Michael Welzl]

Hi, and thanks for a quick answer!

But, it's not quite what I was looking for.... see below:

> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Michael Welzl <michawe@ifi.uio.no> writes:
> 
>> Hi all,
>> 
>> A simple question to y'all who spent so much time on Cake and things
>> ... in a household using WiFi, which buffer is usually bloated? Where
>> does the latency really come from?
>> 
>> Is it:
>> 1. the access point's downlink queue, feeding into the WiFi network,
> 
> This we mostly fixed, but only if you're on a recent OpenWrt with the
> right WiFi drivers.

Well okay... I was curious about where the bottleneck is. I can translate my question into: "if Cake is installed everywhere, where does it have the most work to do?".


> Otherwise, this is a major source of latency *if*
> the WiFi link is faster than the downlink from the internet.

Huh? Slower, you mean?


> This
> depends on both the internet connection and the current rate each WiFi
> station operates at, so it can vary wildly, and on very short time
> scales.

Sure... I was asking for the "if" in your statement above - since this is an operationally-oriented list: what do people see? What is the more common case?


>> 2. the modem's downlink queue, feeding into the access point,
> 
> If your internet (downlink) connection is slower than your WiFi link,
> this is where you'll get the queueing.

Yes sure  :-)     see above: I wanted to know what the more common case is, in households that people on this list have dealt with.


>> 3. the modem's uplink queue,
> 
> As above, but in the other direction - but as uplinks tend to be
> asymmetric, this direction is often more of a problem.
> 
>> 4. the access point's uplink queue towards the modem   (hm, that seems
>> silly, surely the AP-modem connection is fast... so perhaps, instead:
>> the queue in the host, as it wants to send data towards the access
>> point)
> 
> Yeah, that would be in the host; but host drivers can suffer from severe
> bufferbloat as well, especially as rates drop (since the buffers are
> often tuned for the maximum throughput the device can deliver in
> best-case signal conditions).
> 
>> or is it a combination of these?
> 
> Usually it's a combination; especially since the WiFi capacity varies
> wildly with signal conditions (as devices move around relative to the
> AP), general link usage (more devices active mean less available
> capacity for each device, exacerbated by airtime unfairness), and
> interference. Also there are things like excessive retries causing HoL
> blocking.

Man, what an academic answer!    Makes me think you have a PhD, or something!  What *theoretically* can happen is not what I was fishing for   :-D


>> I guess that, with openwrt, Cake is operating on the queue that's
>> feeding the wifi network, as the modem's queue is out of its
>> control... so: is this where the bottleneck usually is?
> 
> It certainly used to be; but as uplink connection speeds improve, the
> bottleneck moves to the WiFi link.

Yessss, that's why I was asking....


> The extent to which this happens
> depends on where you are in the world; personally I've been bottlenecked
> on the WiFi link ever since I got a fibre upstream (and with 802.11ax
> rates maxing at >1Gbps, maybe that'll change again?).

THIS is what I was after   :)   one data point, cool - so far, so good...

Cheers,
Michael

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Sebastian Moeller]

Hi Michael,

just to add another anecdotal data point on a nominal 100/40 access link (synced at 100/32, cake-shaped to 95/31, wifi currenrly ath10K @ 2.4 and 5 GHGz). The access link is the bottleneck most of the time. WiFi typically only shows up in dedicated testing by odd side-effects, like cyclic latency spikes when a station running the speedtest just starts scanning other channels (macos seems prone to do this if location services are somehow triggered), or airtime starvation, when a station (ab)uses AC_VI or AC_VO and the others do not manage to squeeze packets in... 
	Other than that even without the latest wifi drivers (router still on OpenWrt 19.07.4) WiFi mostly works without nasty latency issues, but I have a number of machines connected via wires, so already behaviorally try to keep WiFi only lightly loaded. Not sure whether you where looking for reports like this though. And I have to admit my family is mostly latency tolerant, at least to the lever we encounter it, not sure how that would look without cake (I currently run a speedtest automatically every two hours from the router, but from just using the network I can not tell when these run, of even fully loaded cake keeps the access link quite usable, hurrah for cake's per internal IP address fairness modes, but that is orthogonal to your wifi question).

Best Regards
        Sebastian

P.S.: Do to typical end-user access link asymmetry, I agree with Toke, that WiFi bufferbloat will mostly be seen in the download direction, which is actually not bad, as that means that an air-time fair AP can help a lot. For egress, that is going to be much harder, to sufficiently coordinate all stations to behave well, short of tracking all stations air time usage and reprogram the airtime access parameters individually for each station, but I digress)

> On Oct 6, 2020, at 14:15, Michael Welzl <michawe@ifi.uio.no> wrote:
> 
> Hi, and thanks for a quick answer!
> 
> But, it's not quite what I was looking for.... see below:
> 
>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Michael Welzl <michawe@ifi.uio.no> writes:
>> 
>>> Hi all,
>>> 
>>> A simple question to y'all who spent so much time on Cake and things
>>> ... in a household using WiFi, which buffer is usually bloated? Where
>>> does the latency really come from?
>>> 
>>> Is it:
>>> 1. the access point's downlink queue, feeding into the WiFi network,
>> 
>> This we mostly fixed, but only if you're on a recent OpenWrt with the
>> right WiFi drivers.
> 
> Well okay... I was curious about where the bottleneck is. I can translate my question into: "if Cake is installed everywhere, where does it have the most work to do?".
> 
> 
>> Otherwise, this is a major source of latency *if*
>> the WiFi link is faster than the downlink from the internet.
> 
> Huh? Slower, you mean?
> 
> 
>> This
>> depends on both the internet connection and the current rate each WiFi
>> station operates at, so it can vary wildly, and on very short time
>> scales.
> 
> Sure... I was asking for the "if" in your statement above - since this is an operationally-oriented list: what do people see? What is the more common case?
> 
> 
>>> 2. the modem's downlink queue, feeding into the access point,
>> 
>> If your internet (downlink) connection is slower than your WiFi link,
>> this is where you'll get the queueing.
> 
> Yes sure  :-)     see above: I wanted to know what the more common case is, in households that people on this list have dealt with.
> 
> 
>>> 3. the modem's uplink queue,
>> 
>> As above, but in the other direction - but as uplinks tend to be
>> asymmetric, this direction is often more of a problem.
>> 
>>> 4. the access point's uplink queue towards the modem   (hm, that seems
>>> silly, surely the AP-modem connection is fast... so perhaps, instead:
>>> the queue in the host, as it wants to send data towards the access
>>> point)
>> 
>> Yeah, that would be in the host; but host drivers can suffer from severe
>> bufferbloat as well, especially as rates drop (since the buffers are
>> often tuned for the maximum throughput the device can deliver in
>> best-case signal conditions).
>> 
>>> or is it a combination of these?
>> 
>> Usually it's a combination; especially since the WiFi capacity varies
>> wildly with signal conditions (as devices move around relative to the
>> AP), general link usage (more devices active mean less available
>> capacity for each device, exacerbated by airtime unfairness), and
>> interference. Also there are things like excessive retries causing HoL
>> blocking.
> 
> Man, what an academic answer!    Makes me think you have a PhD, or something!  What *theoretically* can happen is not what I was fishing for   :-D
> 
> 
>>> I guess that, with openwrt, Cake is operating on the queue that's
>>> feeding the wifi network, as the modem's queue is out of its
>>> control... so: is this where the bottleneck usually is?
>> 
>> It certainly used to be; but as uplink connection speeds improve, the
>> bottleneck moves to the WiFi link.
> 
> Yessss, that's why I was asking....
> 
> 
>> The extent to which this happens
>> depends on where you are in the world; personally I've been bottlenecked
>> on the WiFi link ever since I got a fibre upstream (and with 802.11ax
>> rates maxing at >1Gbps, maybe that'll change again?).
> 
> THIS is what I was after   :)   one data point, cool - so far, so good...
> 
> Cheers,
> Michael
> 
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Toke Høiland-Jørgensen]

Michael Welzl <michawe@ifi.uio.no> writes:

> Hi, and thanks for a quick answer!
>
> But, it's not quite what I was looking for.... see below:
>
>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 
>> Michael Welzl <michawe@ifi.uio.no> writes:
>> 
>>> Hi all,
>>> 
>>> A simple question to y'all who spent so much time on Cake and things
>>> ... in a household using WiFi, which buffer is usually bloated? Where
>>> does the latency really come from?
>>> 
>>> Is it:
>>> 1. the access point's downlink queue, feeding into the WiFi network,
>> 
>> This we mostly fixed, but only if you're on a recent OpenWrt with the
>> right WiFi drivers.
>
> Well okay... I was curious about where the bottleneck is. I can
> translate my question into: "if Cake is installed everywhere, where
> does it have the most work to do?".

Well, CAKE only runs on the upstream link, so that's where it does its
work. The software shaper model doesn't really work that well on WiFi,
so we generally encourage people to just run fixed drivers there
instead. That being said I have heard of one or two WiFi deployments
where that was not an option, and where CAKE was used as a shaper
instead. This was for a fixed WiFi backhaul, though, and even so they
had to set the shaper quite a lot lower than the max bandwidth to get
reliable performance.

>> Otherwise, this is a major source of latency *if*
>> the WiFi link is faster than the downlink from the internet.
>
> Huh? Slower, you mean?

No, if the WiFi link is faster, the upstream link becomes the bottleneck
and CAKE has work to do :)

>> This
>> depends on both the internet connection and the current rate each WiFi
>> station operates at, so it can vary wildly, and on very short time
>> scales.
>
> Sure... I was asking for the "if" in your statement above - since this
> is an operationally-oriented list: what do people see? What is the
> more common case?

Right, well as you can probably tell that might not have been entirely
clear from your initial post ;)

>> The extent to which this happens depends on where you are in the
>> world; personally I've been bottlenecked on the WiFi link ever since
>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps,
>> maybe that'll change again?).
>
> THIS is what I was after :) one data point, cool - so far, so good...

Ah, you're after anecdotes - well why didn't you say so? ;)

In that case I'll add that my own connection is the only one I've come
across where the WiFi link is *never* the bottleneck. In Denmark we are
finally (slowly) getting out of the dark ages as far as fibre
deployments are concerned, but most operators will sell connections
capped at 100Mbps or 250Mbps, which is still less than the throughout of
a 802.11ac link with good signal conditions (my phone consistently gets
~250-350 Mbps on a speedtest.net run).

DSL connections tend to have awful latency, and are still quite common,
but they are pretty easy to fix with CAKE. Cable connections likewise,
or so is my impression (those are not so common around these parts).

The worst are definitely LTE/mobile broadband connections. Wildly
varying link speeds, and awful over-buffering; so you really have to
clamp them down to get anything useful out of CAKE.

-Toke

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Jonathan Morton]

> On 6 Oct, 2020, at 3:44 pm, Toke Høiland-Jørgensen via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
> The worst are definitely LTE/mobile broadband connections. Wildly
> varying link speeds, and awful over-buffering; so you really have to
> clamp them down to get anything useful out of CAKE.

Out here in rural Finland, that's what I have to deal with.  Meanwhile the wifi bands are very quiet and I'm never far from the AP, so there is plenty of throughput there.  I'm sure there are cases with wired WAN and noisier wifi airspace where the wifi could often be the bottleneck.  The balance will vary for different people at different times.

But also you should consider the case of transferring files over wifi to/from some other machine on the LAN.  In that case the Internet connection doesn't matter, and the wifi will always be the bottleneck.

The lesson is that you need to address bloat everywhere in order to cover all possible use cases.

 - Jonathan Morton

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Michael Welzl]

Thanks a lot - just the info I was looking for  (also from others who have responded in the meantime, thanks!)


> On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Michael Welzl <michawe@ifi.uio.no> writes:
> 
>> Hi, and thanks for a quick answer!
>> 
>> But, it's not quite what I was looking for.... see below:
>> 
>>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>> 
>>> Michael Welzl <michawe@ifi.uio.no> writes:
>>> 
>>>> Hi all,
>>>> 
>>>> A simple question to y'all who spent so much time on Cake and things
>>>> ... in a household using WiFi, which buffer is usually bloated? Where
>>>> does the latency really come from?
>>>> 
>>>> Is it:
>>>> 1. the access point's downlink queue, feeding into the WiFi network,
>>> 
>>> This we mostly fixed, but only if you're on a recent OpenWrt with the
>>> right WiFi drivers.
>> 
>> Well okay... I was curious about where the bottleneck is. I can
>> translate my question into: "if Cake is installed everywhere, where
>> does it have the most work to do?".
> 
> Well, CAKE only runs on the upstream link, so that's where it does its
> work. The software shaper model doesn't really work that well on WiFi,
> so we generally encourage people to just run fixed drivers there
> instead. That being said I have heard of one or two WiFi deployments
> where that was not an option, and where CAKE was used as a shaper
> instead. This was for a fixed WiFi backhaul, though, and even so they
> had to set the shaper quite a lot lower than the max bandwidth to get
> reliable performance.
> 
>>> Otherwise, this is a major source of latency *if*
>>> the WiFi link is faster than the downlink from the internet.
>> 
>> Huh? Slower, you mean?
> 
> No, if the WiFi link is faster, the upstream link becomes the bottleneck
> and CAKE has work to do :)
> 
>>> This
>>> depends on both the internet connection and the current rate each WiFi
>>> station operates at, so it can vary wildly, and on very short time
>>> scales.
>> 
>> Sure... I was asking for the "if" in your statement above - since this
>> is an operationally-oriented list: what do people see? What is the
>> more common case?
> 
> Right, well as you can probably tell that might not have been entirely
> clear from your initial post ;)
> 
>>> The extent to which this happens depends on where you are in the
>>> world; personally I've been bottlenecked on the WiFi link ever since
>>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps,
>>> maybe that'll change again?).
>> 
>> THIS is what I was after :) one data point, cool - so far, so good...
> 
> Ah, you're after anecdotes - well why didn't you say so? ;)
> 
> In that case I'll add that my own connection is the only one I've come
> across where the WiFi link is *never* the bottleneck. In Denmark we are
> finally (slowly) getting out of the dark ages as far as fibre
> deployments are concerned, but most operators will sell connections
> capped at 100Mbps or 250Mbps, which is still less than the throughout of
> a 802.11ac link with good signal conditions (my phone consistently gets
> ~250-350 Mbps on a speedtest.net run).
> 
> DSL connections tend to have awful latency, and are still quite common,
> but they are pretty easy to fix with CAKE. Cable connections likewise,
> or so is my impression (those are not so common around these parts).
> 
> The worst are definitely LTE/mobile broadband connections. Wildly
> varying link speeds, and awful over-buffering; so you really have to
> clamp them down to get anything useful out of CAKE.
> 
> -Toke

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 17207 bytes --]

You might want to try iperf 2.0.14 --trip-times option to measure your
links.  Below a run with a PC w/a 1G NIC connected to the 1G cable MODEM no
WiFi, i.e. wired, full duplex socket.  The uplink tcp write latencies are
larger than one second.  Downlinks just under 100 ms.  Slow down the uplink
traffic to less than the congestion point and write to read latency drops
to ~65 ms.  I think this suggests my CABLE MODEM is full of bloat.

UPLINK ONLY
--------------------

Server:
[rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25408
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port
58158 (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval        Transfer    Bandwidth    Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  4] 0.00-1.00 sec  1.25 MBytes  10.5 Mbits/sec
 59.680/51.222/96.282/34.950 ms (10/131072) 80.9 KByte 21.96
 305=305:0:0:0:0:0:0:0
[  4] 1.00-2.00 sec  1.12 MBytes  9.44 Mbits/sec
 65.218/51.951/87.895/10.877 ms (9/131072) 79.6 KByte 18.09
 262=262:0:0:0:0:0:0:0
[  4] 2.00-3.00 sec  1.25 MBytes  10.5 Mbits/sec  62.479/51.170/67.962/
5.189 ms (10/131072) 76.3 KByte 20.98  289=289:0:0:0:0:0:0:0
[  4] 3.00-4.00 sec  1.12 MBytes  9.44 Mbits/sec  62.861/52.478/68.339/
5.212 ms (9/131072) 76.7 KByte 18.77  262=262:0:0:0:0:0:0:0
[  4] 4.00-5.00 sec  1.25 MBytes  10.5 Mbits/sec
 64.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23
 292=292:0:0:0:0:0:0:0
[  4] 5.00-6.00 sec  1.14 MBytes  9.60 Mbits/sec  64.618/50.914/68.277/
5.696 ms (9/133334) 80.2 KByte 18.57  259=259:0:0:0:0:0:0:0
[  4] 6.00-7.00 sec  1.23 MBytes  10.3 Mbits/sec
 66.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29
 279=279:0:0:0:0:0:0:0
[  4] 7.00-8.00 sec  1.16 MBytes  9.70 Mbits/sec
 68.954/51.902/100.783/15.733 ms (9/134782) 86.6 KByte 17.59
 275=274:1:0:0:0:0:0:0
[  4] 8.00-9.00 sec  1.22 MBytes  10.2 Mbits/sec  63.855/51.386/69.919/
5.284 ms (9/141851) 84.4 KByte 19.99  283=283:0:0:0:0:0:0:0
[  4] 9.00-10.00 sec  1.18 MBytes  9.86 Mbits/sec
 67.975/50.996/88.151/11.318 ms (10/123252) 78.0 KByte 18.13
 275=275:0:0:0:0:0:0:0
[  4] 10.00-10.14 sec   205 KBytes  11.7 Mbits/sec  99.865/55.342/77.370/
7.319 ms (2/104962) 97.6 KByte 14.61  43=43:0:0:0:0:0:0:0
[  4] 0.00-10.14 sec  12.1 MBytes  10.0 Mbits/sec
 59.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04
 2824=2823:1:0:0:0:0:0:0

Client:

[root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1
--trip-times -Z bbr -b 10m -e
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 48006 (1
flows)
Write buffer size:  128 KByte
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.35%enp2s0 port 58160 connected with 45.33.58.123 port
5001 (trip-times) (MSS=1448) (ct=13.05 ms)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry
Cwnd/RTT        NetPwr
[  3] 0.00-1.00 sec  1.25 MBytes  10.5 Mbits/sec  11/0          0
 135K/49251 us  26.61
[  3] 1.00-2.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0
 141K/24791 us  52.87
[  3] 2.00-3.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0
 135K/24973 us  47.24
[  3] 3.00-4.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0
 158K/25021 us  52.38
[  3] 4.00-5.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0
 141K/21247 us  55.52
[  3] 5.00-6.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0
 144K/23535 us  55.69
[  3] 6.00-7.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0
 135K/27130 us  43.48
[  3] 7.00-8.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0
 135K/24813 us  52.82
[  3] 8.00-9.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0          0
 155K/22697 us  51.97
[  3] 9.00-10.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          0
 124K/21954 us  59.70
[  3] 10.00-10.07 sec   256 KBytes  31.6 Mbits/sec  2/0          0
 124K/21954 us  179.64
[  3] 0.00-10.07 sec  12.3 MBytes  10.2 Mbits/sec  99/0          0
 124K/21954 us  58.12

FULL DUPLEX:
---------------------
Server:

[rjmcmahon@bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr
------------------------------------------------------------
Server listening on TCP port 5001 with pid 25379
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port
58152 (full-duplex) (trip-times) (MSS=1448) (peer 2.0.14-alpha)
[ ID] Interval        Transfer    Bandwidth    Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  4] 0.00-1.00 sec  1.05 MBytes  8.79 Mbits/sec
 547.494/68.304/904.670/301.250 ms (8/137379) 9.75 MByte 2.01
 345=344:1:0:0:0:0:0:0
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry
Cwnd/RTT        NetPwr
[  4] 0.00-1.00 sec  40.3 MBytes   338 Mbits/sec  323/0          0
1267K/20319 us  2077.13
[ ID] Interval       Transfer     Bandwidth
[FD4] 0.00-1.00 sec  41.3 MBytes   346 Mbits/sec
[  4] 1.00-2.00 sec  1.34 MBytes  11.3 Mbits/sec
1307.675/904.670/1454.073/145.211
ms (11/127950) 2.30 MByte 1.08  277=262:5:4:6:0:0:0:0
[  4] 1.00-2.00 sec  38.8 MBytes   325 Mbits/sec  310/0          0
1001K/24428 us  1663.35
[FD4] 1.00-2.00 sec  40.1 MBytes   336 Mbits/sec
[  4] 2.00-3.00 sec  38.0 MBytes   319 Mbits/sec  304/0          0
1156K/26355 us  1511.89
[  4] 2.00-3.00 sec  1.33 MBytes  11.1 Mbits/sec
 1298.825/1139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07
 372=366:2:1:3:0:0:0:0
[FD4] 2.00-3.00 sec  39.3 MBytes   330 Mbits/sec
[  4] 3.00-4.00 sec  1.32 MBytes  11.1 Mbits/sec
 1302.239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06
 217=199:4:8:4:2:0:0:0
[  4] 3.00-4.00 sec  39.6 MBytes   332 Mbits/sec  317/0          0
 970K/19064 us  2179.49
[FD4] 3.00-4.00 sec  40.9 MBytes   343 Mbits/sec
[  4] 4.00-5.00 sec  38.5 MBytes   323 Mbits/sec  308/0          0
 998K/20599 us  1959.81
[  4] 4.00-5.00 sec  1.39 MBytes  11.7 Mbits/sec
 1310.588/1172.769/1233.522/21.090 ms (11/132426) 1.79 MByte 1.11
 223=196:18:5:4:0:0:0:0
[FD4] 4.00-5.00 sec  39.9 MBytes   335 Mbits/sec
[  4] 5.00-6.00 sec  38.6 MBytes   324 Mbits/sec  309/0          0
1340K/17717 us  2286.01
[  4] 5.00-6.00 sec  1.33 MBytes  11.2 Mbits/sec
 1305.731/1173.179/1220.196/17.125 ms (11/127160) 1.72 MByte 1.07
 158=136:10:5:6:1:0:0:0
[FD4] 5.00-6.00 sec  40.0 MBytes   335 Mbits/sec
[  4] 6.00-7.00 sec  1.36 MBytes  11.4 Mbits/sec
 1301.735/1162.410/1216.840/20.348 ms (10/142193) 1.87 MByte 1.09
 140=116:7:8:6:3:0:0:0
[  4] 6.00-7.00 sec  39.4 MBytes   330 Mbits/sec  315/0          0
1074K/20040 us  2060.26
[FD4] 6.00-7.00 sec  40.7 MBytes   342 Mbits/sec
[  4] 7.00-8.00 sec  1.35 MBytes  11.3 Mbits/sec
 1291.605/1160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10
 216=198:5:6:7:0:0:0:0
[  4] 7.00-8.00 sec  38.9 MBytes   326 Mbits/sec  311/0          0
1074K/30895 us  1319.42
[FD4] 7.00-8.00 sec  40.2 MBytes   337 Mbits/sec
[  4] 8.00-9.00 sec  37.4 MBytes   314 Mbits/sec  299/0          0
1264K/29185 us  1342.83
[  4] 8.00-9.00 sec  1.27 MBytes  10.7 Mbits/sec
 1326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.01
 110=86:9:7:5:1:0:2:0
[FD4] 8.00-9.00 sec  38.6 MBytes   324 Mbits/sec
[  4] 9.00-10.00 sec  1.34 MBytes  11.2 Mbits/sec
 1312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByte 1.07
 124=101:12:6:2:1:0:2:0
[  4] 9.00-10.00 sec  39.2 MBytes   329 Mbits/sec  314/0          0
 967K/21976 us  1872.80
[FD4] 9.00-10.00 sec  40.6 MBytes   341 Mbits/sec
[  4] 10.00-10.00 sec   256 KBytes   477 Mbits/sec  2/0          0
 967K/21976 us  2711.06
[  4] 0.00-10.00 sec   389 MBytes   326 Mbits/sec  3112/0          0
 967K/21976 us  1854.69
[  4] 10.00-11.00 sec   963 KBytes  7.89 Mbits/sec
 1624.535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61
 215=210:3:0:0:1:1:0:0
[  4] 11.00-11.41 sec   622 KBytes  12.6 Mbits/sec
 1780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0.88
 29=11:11:4:3:0:0:0:0
[  4] 0.00-11.41 sec  14.6 MBytes  10.8 Mbits/sec
 1184.903/68.304/1548.718/215.041 ms (117/131072) 1.52 MByte 1.13
 2426=2225:87:54:46:9:1:4:0
[FD4] 10.00-1602028944.71 sec  1.80 MBytes  0.009 bits/sec
[FD4] 0.00-1602028944.71 sec   404 MBytes  2.11 bits/sec

Client

[root@localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1
--trip-times --full-duplex -Z bbr
WARNING: tcp congestion control will only be applied on transmit traffic,
use -Z on the server
------------------------------------------------------------
Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 47994 (1
flows)
Write buffer size:  128 KByte
TCP congestion control set to bbr
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.35%enp2s0 port 58154 connected with 45.33.58.123 port
5001 (full-duplex) (trip-times) (MSS=1448) (ct=12.42 ms)
[ ID] Interval        Transfer    Bandwidth    Burst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  3] 0.00-1.00 sec  35.8 MBytes   301 Mbits/sec
 89.806/29.604/103.341/16.204 ms (286/131398) 3.58 MByte 418.46
 17764=17726:35:3:0:0:0:0:0
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry
Cwnd/RTT        NetPwr
[  3] 0.00-1.00 sec  2.50 MBytes  21.0 Mbits/sec  21/0         91
62K/44390 us  59.06
[ ID] Interval       Transfer     Bandwidth
[FD3] 0.00-1.00 sec  38.3 MBytes   322 Mbits/sec
[  3] 1.00-2.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0          8
56K/36570 us  35.84
[  3] 1.00-2.00 sec  38.7 MBytes   325 Mbits/sec  94.258/61.108/104.117/
9.454 ms (310/130882) 3.63 MByte 430.45  20775=20768:5:2:0:0:0:0:0
[FD3] 1.00-2.00 sec  39.9 MBytes   335 Mbits/sec
[  3] 2.00-3.00 sec  38.6 MBytes   324 Mbits/sec  94.554/62.312/105.451/
9.324 ms (308/131379) 3.69 MByte 427.96  18929=18913:15:1:0:0:0:0:0
[  3] 2.00-3.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         12
65K/54911 us  26.26
[FD3] 2.00-3.00 sec  40.0 MBytes   335 Mbits/sec
[  3] 3.00-4.00 sec  37.3 MBytes   313 Mbits/sec
 95.362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71
 17567=17537:27:3:0:0:0:0:0
[  3] 3.00-4.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         20
67K/46946 us  30.71
[FD3] 3.00-4.00 sec  38.6 MBytes   324 Mbits/sec
[  3] 4.00-5.00 sec  1.25 MBytes  10.5 Mbits/sec  10/0         51
59K/67662 us  19.37
[  3] 4.00-5.00 sec  38.4 MBytes   322 Mbits/sec
 90.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.99
 19095=19077:16:2:0:0:0:0:0
[FD3] 4.00-5.00 sec  39.6 MBytes   333 Mbits/sec
[  3] 5.00-6.00 sec  38.5 MBytes   323 Mbits/sec  88.832/64.621/102.534/
9.742 ms (308/130979) 3.40 MByte 454.13  17674=17657:17:0:0:0:0:0:0
[  3] 5.00-6.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0         48
65K/64456 us  22.37
[FD3] 5.00-6.00 sec  39.8 MBytes   334 Mbits/sec
[  3] 6.00-7.00 sec  38.4 MBytes   322 Mbits/sec
 87.121/62.694/102.124/10.260 ms (307/131199) 3.24 MByte 462.33
 19853=19852:1:0:0:0:0:0:0
[  3] 6.00-7.00 sec  3.75 MBytes  31.5 Mbits/sec  30/0         59
67K/151435 us  25.97
[FD3] 6.00-7.00 sec  42.2 MBytes   354 Mbits/sec
[  3] 7.00-8.00 sec  38.5 MBytes   323 Mbits/sec
 87.936/62.694/102.226/10.016 ms (308/131222) 3.41 MByte 459.61
 19067=19052:15:0:0:0:0:0:0
[  3] 7.00-8.00 sec  1.12 MBytes  9.44 Mbits/sec  9/0         60
73K/56735 us  20.79
[FD3] 7.00-8.00 sec  39.7 MBytes   333 Mbits/sec
[  3] 8.00-9.00 sec  35.9 MBytes   301 Mbits/sec
 91.441/56.797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54
 18900=18881:10:2:0:0:0:0:7
[  3] 8.00-9.00 sec  1.50 MBytes  12.6 Mbits/sec  13/0         26
82K/49077 us  32.05
[FD3] 8.00-9.00 sec  37.4 MBytes   314 Mbits/sec
[  3] 9.00-10.00 sec  36.3 MBytes   305 Mbits/sec
 90.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88
 17888=17871:17:0:0:0:0:0:0
[  3] 10.00-10.07 sec  2.72 MBytes   310 Mbits/sec  91.852/69.979/102.633/
8.280 ms (22/129636) 5.96 MByte 421.94  1418=1414:4:0:0:0:0:0:0
[  3] 0.00-10.07 sec   379 MBytes   316 Mbits/sec
 90.712/29.604/148.892/12.195 ms (3033/131072) 3.41 MByte 435.04
 188930=188748:162:13:0:0:0:0:7
[  3] 9.00-10.00 sec  1.38 MBytes  11.5 Mbits/sec  11/0          0
 5K/55760 us  25.86
[FD3] 9.00-10.00 sec  40.4 MBytes   339 Mbits/sec
[  3] 10.00-10.49 sec   223 KBytes  3.69 Mbits/sec  2/0          0
 5K/55760 us  8.28
[  3] 0.00-10.49 sec  17.1 MBytes  13.7 Mbits/sec  139/0        424
 5K/55760 us  30.63
[FD3] 10.00-10.49 sec   223 KBytes  3.77 Mbits/sec
[FD3] 0.00-10.49 sec   396 MBytes   317 Mbits/sec

Bob

On Tue, Oct 6, 2020 at 6:24 AM Michael Welzl <michawe@ifi.uio.no> wrote:

> Thanks a lot - just the info I was looking for  (also from others who have
> responded in the meantime, thanks!)
>
>
> > On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >
> > Michael Welzl <michawe@ifi.uio.no> writes:
> >
> >> Hi, and thanks for a quick answer!
> >>
> >> But, it's not quite what I was looking for.... see below:
> >>
> >>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >>>
> >>> Michael Welzl <michawe@ifi.uio.no> writes:
> >>>
> >>>> Hi all,
> >>>>
> >>>> A simple question to y'all who spent so much time on Cake and things
> >>>> ... in a household using WiFi, which buffer is usually bloated? Where
> >>>> does the latency really come from?
> >>>>
> >>>> Is it:
> >>>> 1. the access point's downlink queue, feeding into the WiFi network,
> >>>
> >>> This we mostly fixed, but only if you're on a recent OpenWrt with the
> >>> right WiFi drivers.
> >>
> >> Well okay... I was curious about where the bottleneck is. I can
> >> translate my question into: "if Cake is installed everywhere, where
> >> does it have the most work to do?".
> >
> > Well, CAKE only runs on the upstream link, so that's where it does its
> > work. The software shaper model doesn't really work that well on WiFi,
> > so we generally encourage people to just run fixed drivers there
> > instead. That being said I have heard of one or two WiFi deployments
> > where that was not an option, and where CAKE was used as a shaper
> > instead. This was for a fixed WiFi backhaul, though, and even so they
> > had to set the shaper quite a lot lower than the max bandwidth to get
> > reliable performance.
> >
> >>> Otherwise, this is a major source of latency *if*
> >>> the WiFi link is faster than the downlink from the internet.
> >>
> >> Huh? Slower, you mean?
> >
> > No, if the WiFi link is faster, the upstream link becomes the bottleneck
> > and CAKE has work to do :)
> >
> >>> This
> >>> depends on both the internet connection and the current rate each WiFi
> >>> station operates at, so it can vary wildly, and on very short time
> >>> scales.
> >>
> >> Sure... I was asking for the "if" in your statement above - since this
> >> is an operationally-oriented list: what do people see? What is the
> >> more common case?
> >
> > Right, well as you can probably tell that might not have been entirely
> > clear from your initial post ;)
> >
> >>> The extent to which this happens depends on where you are in the
> >>> world; personally I've been bottlenecked on the WiFi link ever since
> >>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps,
> >>> maybe that'll change again?).
> >>
> >> THIS is what I was after :) one data point, cool - so far, so good...
> >
> > Ah, you're after anecdotes - well why didn't you say so? ;)
> >
> > In that case I'll add that my own connection is the only one I've come
> > across where the WiFi link is *never* the bottleneck. In Denmark we are
> > finally (slowly) getting out of the dark ages as far as fibre
> > deployments are concerned, but most operators will sell connections
> > capped at 100Mbps or 250Mbps, which is still less than the throughout of
> > a 802.11ac link with good signal conditions (my phone consistently gets
> > ~250-350 Mbps on a speedtest.net run).
> >
> > DSL connections tend to have awful latency, and are still quite common,
> > but they are pretty easy to fix with CAKE. Cable connections likewise,
> > or so is my impression (those are not so common around these parts).
> >
> > The worst are definitely LTE/mobile broadband connections. Wildly
> > varying link speeds, and awful over-buffering; so you really have to
> > clamp them down to get anything useful out of CAKE.
> >
> > -Toke
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #1.2: Type: text/html, Size: 20260 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4163 bytes --]

Re: [Make-wifi-fast] Where is the bloat in WiFi?

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 5298 bytes --]

Hi Michael,

In my case (WAN = GPON DL/UP 2Gbps/500Mbps) some devices are bottlenecked
at the AP, others are
bottlenecked at the switch, like my NAS which is multi-homed with 2 x 1GE
ports to the switch.
WAN downlink is hardly ever the bottleneck. However, backhaul links are
10GE and there may be
congestion there depending on the level of over subscription.

(From a container in the NAS)

     Server: ORANGE FRANCE - Paris (id = 24215)
        ISP: Orange
    Latency:     0.92 ms   (0.03 ms jitter)
   Download:   935.31 Mbps (data used: 433.1 MB)
     Upload:   599.02 Mbps (data used: 645.4 MB)
Packet Loss:     0.0%

For a laptop connected to one AP (line of sight available) I get 500Mbps
DL/UL speedtest.
For the uplink it is hard to say if the bottleneck is the uplink of the
router or one of the three APs,
or the switch? At 5GHz no device goes below ~700Mbps (PHY) rate according
to the WLC.

My case is not really common now, but with fiber adoption growth and more
people working from
home I expect this to become more common. WMM helps with all the online
meetings we do these days.
For video end-points (like Cisco DX) I only use ethernet.

With fiber growth in the home, investments in the backhaul become critical
to keep up with demand.

For xDSL the bottleneck is the WAN with high probability.

Luca




On Tue, Oct 6, 2020 at 2:15 PM Michael Welzl <michawe@ifi.uio.no> wrote:

> Hi, and thanks for a quick answer!
>
> But, it's not quite what I was looking for.... see below:
>
> > On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >
> > Michael Welzl <michawe@ifi.uio.no> writes:
> >
> >> Hi all,
> >>
> >> A simple question to y'all who spent so much time on Cake and things
> >> ... in a household using WiFi, which buffer is usually bloated? Where
> >> does the latency really come from?
> >>
> >> Is it:
> >> 1. the access point's downlink queue, feeding into the WiFi network,
> >
> > This we mostly fixed, but only if you're on a recent OpenWrt with the
> > right WiFi drivers.
>
> Well okay... I was curious about where the bottleneck is. I can translate
> my question into: "if Cake is installed everywhere, where does it have the
> most work to do?".
>
>
> > Otherwise, this is a major source of latency *if*
> > the WiFi link is faster than the downlink from the internet.
>
> Huh? Slower, you mean?
>
>
> > This
> > depends on both the internet connection and the current rate each WiFi
> > station operates at, so it can vary wildly, and on very short time
> > scales.
>
> Sure... I was asking for the "if" in your statement above - since this is
> an operationally-oriented list: what do people see? What is the more common
> case?
>
>
> >> 2. the modem's downlink queue, feeding into the access point,
> >
> > If your internet (downlink) connection is slower than your WiFi link,
> > this is where you'll get the queueing.
>
> Yes sure  :-)     see above: I wanted to know what the more common case
> is, in households that people on this list have dealt with.
>
>
> >> 3. the modem's uplink queue,
> >
> > As above, but in the other direction - but as uplinks tend to be
> > asymmetric, this direction is often more of a problem.
> >
> >> 4. the access point's uplink queue towards the modem   (hm, that seems
> >> silly, surely the AP-modem connection is fast... so perhaps, instead:
> >> the queue in the host, as it wants to send data towards the access
> >> point)
> >
> > Yeah, that would be in the host; but host drivers can suffer from severe
> > bufferbloat as well, especially as rates drop (since the buffers are
> > often tuned for the maximum throughput the device can deliver in
> > best-case signal conditions).
> >
> >> or is it a combination of these?
> >
> > Usually it's a combination; especially since the WiFi capacity varies
> > wildly with signal conditions (as devices move around relative to the
> > AP), general link usage (more devices active mean less available
> > capacity for each device, exacerbated by airtime unfairness), and
> > interference. Also there are things like excessive retries causing HoL
> > blocking.
>
> Man, what an academic answer!    Makes me think you have a PhD, or
> something!  What *theoretically* can happen is not what I was fishing for
>  :-D
>
>
> >> I guess that, with openwrt, Cake is operating on the queue that's
> >> feeding the wifi network, as the modem's queue is out of its
> >> control... so: is this where the bottleneck usually is?
> >
> > It certainly used to be; but as uplink connection speeds improve, the
> > bottleneck moves to the WiFi link.
>
> Yessss, that's why I was asking....
>
>
> > The extent to which this happens
> > depends on where you are in the world; personally I've been bottlenecked
> > on the WiFi link ever since I got a fibre upstream (and with 802.11ax
> > rates maxing at >1Gbps, maybe that'll change again?).
>
> THIS is what I was after   :)   one data point, cool - so far, so good...
>
> Cheers,
> Michael
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #2: Type: text/html, Size: 8313 bytes --]