From michawe at ifi.uio.no Tue Oct 6 07:25:36 2020 From: michawe at ifi.uio.no (Michael Welzl) Date: Tue, 6 Oct 2020 13:25:36 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? Message-ID: Hi all, A simple question to y'all who spent so much time on Cake and things ... in a household using WiFi, which buffer is usually bloated? Where does the latency really come from? Is it: 1. the access point's downlink queue, feeding into the WiFi network, 2. the modem's downlink queue, feeding into the access point, 3. the modem's uplink queue, 4. the access point's uplink queue towards the modem (hm, that seems silly, surely the AP-modem connection is fast... so perhaps, instead: the queue in the host, as it wants to send data towards the access point) or is it a combination of these? I guess that, with openwrt, Cake is operating on the queue that's feeding the wifi network, as the modem's queue is out of its control... so: is this where the bottleneck usually is? Just wondering about your views and experiences. Thanks in advance! Cheers, Michael -- PS: my personal guess: 1 and 3 above are the most common. But that's *pure* guesswork! From toke at toke.dk Tue Oct 6 07:47:47 2020 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Tue, 06 Oct 2020 13:47:47 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: References: Message-ID: <87d01vfue4.fsf@toke.dk> Michael Welzl writes: > Hi all, > > A simple question to y'all who spent so much time on Cake and things > ... in a household using WiFi, which buffer is usually bloated? Where > does the latency really come from? > > Is it: > 1. the access point's downlink queue, feeding into the WiFi network, This we mostly fixed, but only if you're on a recent OpenWrt with the right WiFi drivers. Otherwise, this is a major source of latency *if* the WiFi link is faster than the downlink from the internet. This depends on both the internet connection and the current rate each WiFi station operates at, so it can vary wildly, and on very short time scales. > 2. the modem's downlink queue, feeding into the access point, If your internet (downlink) connection is slower than your WiFi link, this is where you'll get the queueing. > 3. the modem's uplink queue, As above, but in the other direction - but as uplinks tend to be asymmetric, this direction is often more of a problem. > 4. the access point's uplink queue towards the modem (hm, that seems > silly, surely the AP-modem connection is fast... so perhaps, instead: > the queue in the host, as it wants to send data towards the access > point) Yeah, that would be in the host; but host drivers can suffer from severe bufferbloat as well, especially as rates drop (since the buffers are often tuned for the maximum throughput the device can deliver in best-case signal conditions). > or is it a combination of these? Usually it's a combination; especially since the WiFi capacity varies wildly with signal conditions (as devices move around relative to the AP), general link usage (more devices active mean less available capacity for each device, exacerbated by airtime unfairness), and interference. Also there are things like excessive retries causing HoL blocking. > I guess that, with openwrt, Cake is operating on the queue that's > feeding the wifi network, as the modem's queue is out of its > control... so: is this where the bottleneck usually is? It certainly used to be; but as uplink connection speeds improve, the bottleneck moves to the WiFi link. The extent to which this happens depends on where you are in the world; personally I've been bottlenecked on the WiFi link ever since I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps, maybe that'll change again?). -Toke From michawe at ifi.uio.no Tue Oct 6 08:15:30 2020 From: michawe at ifi.uio.no (Michael Welzl) Date: Tue, 6 Oct 2020 14:15:30 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: <87d01vfue4.fsf@toke.dk> References: <87d01vfue4.fsf@toke.dk> Message-ID: Hi, and thanks for a quick answer! But, it's not quite what I was looking for.... see below: > On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: > > Michael Welzl writes: > >> Hi all, >> >> A simple question to y'all who spent so much time on Cake and things >> ... in a household using WiFi, which buffer is usually bloated? Where >> does the latency really come from? >> >> Is it: >> 1. the access point's downlink queue, feeding into the WiFi network, > > This we mostly fixed, but only if you're on a recent OpenWrt with the > right WiFi drivers. Well okay... I was curious about where the bottleneck is. I can translate my question into: "if Cake is installed everywhere, where does it have the most work to do?". > Otherwise, this is a major source of latency *if* > the WiFi link is faster than the downlink from the internet. Huh? Slower, you mean? > This > depends on both the internet connection and the current rate each WiFi > station operates at, so it can vary wildly, and on very short time > scales. Sure... I was asking for the "if" in your statement above - since this is an operationally-oriented list: what do people see? What is the more common case? >> 2. the modem's downlink queue, feeding into the access point, > > If your internet (downlink) connection is slower than your WiFi link, > this is where you'll get the queueing. Yes sure :-) see above: I wanted to know what the more common case is, in households that people on this list have dealt with. >> 3. the modem's uplink queue, > > As above, but in the other direction - but as uplinks tend to be > asymmetric, this direction is often more of a problem. > >> 4. the access point's uplink queue towards the modem (hm, that seems >> silly, surely the AP-modem connection is fast... so perhaps, instead: >> the queue in the host, as it wants to send data towards the access >> point) > > Yeah, that would be in the host; but host drivers can suffer from severe > bufferbloat as well, especially as rates drop (since the buffers are > often tuned for the maximum throughput the device can deliver in > best-case signal conditions). > >> or is it a combination of these? > > Usually it's a combination; especially since the WiFi capacity varies > wildly with signal conditions (as devices move around relative to the > AP), general link usage (more devices active mean less available > capacity for each device, exacerbated by airtime unfairness), and > interference. Also there are things like excessive retries causing HoL > blocking. Man, what an academic answer! Makes me think you have a PhD, or something! What *theoretically* can happen is not what I was fishing for :-D >> I guess that, with openwrt, Cake is operating on the queue that's >> feeding the wifi network, as the modem's queue is out of its >> control... so: is this where the bottleneck usually is? > > It certainly used to be; but as uplink connection speeds improve, the > bottleneck moves to the WiFi link. Yessss, that's why I was asking.... > The extent to which this happens > depends on where you are in the world; personally I've been bottlenecked > on the WiFi link ever since I got a fibre upstream (and with 802.11ax > rates maxing at >1Gbps, maybe that'll change again?). THIS is what I was after :) one data point, cool - so far, so good... Cheers, Michael From moeller0 at gmx.de Tue Oct 6 08:44:02 2020 From: moeller0 at gmx.de (Sebastian Moeller) Date: Tue, 6 Oct 2020 14:44:02 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: References: <87d01vfue4.fsf@toke.dk> Message-ID: <7BFF2AA7-AB58-46D2-91F2-9BEE2D34AA28@gmx.de> Hi Michael, just to add another anecdotal data point on a nominal 100/40 access link (synced at 100/32, cake-shaped to 95/31, wifi currenrly ath10K @ 2.4 and 5 GHGz). The access link is the bottleneck most of the time. WiFi typically only shows up in dedicated testing by odd side-effects, like cyclic latency spikes when a station running the speedtest just starts scanning other channels (macos seems prone to do this if location services are somehow triggered), or airtime starvation, when a station (ab)uses AC_VI or AC_VO and the others do not manage to squeeze packets in... Other than that even without the latest wifi drivers (router still on OpenWrt 19.07.4) WiFi mostly works without nasty latency issues, but I have a number of machines connected via wires, so already behaviorally try to keep WiFi only lightly loaded. Not sure whether you where looking for reports like this though. And I have to admit my family is mostly latency tolerant, at least to the lever we encounter it, not sure how that would look without cake (I currently run a speedtest automatically every two hours from the router, but from just using the network I can not tell when these run, of even fully loaded cake keeps the access link quite usable, hurrah for cake's per internal IP address fairness modes, but that is orthogonal to your wifi question). Best Regards Sebastian P.S.: Do to typical end-user access link asymmetry, I agree with Toke, that WiFi bufferbloat will mostly be seen in the download direction, which is actually not bad, as that means that an air-time fair AP can help a lot. For egress, that is going to be much harder, to sufficiently coordinate all stations to behave well, short of tracking all stations air time usage and reprogram the airtime access parameters individually for each station, but I digress) > On Oct 6, 2020, at 14:15, Michael Welzl wrote: > > Hi, and thanks for a quick answer! > > But, it's not quite what I was looking for.... see below: > >> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: >> >> Michael Welzl writes: >> >>> Hi all, >>> >>> A simple question to y'all who spent so much time on Cake and things >>> ... in a household using WiFi, which buffer is usually bloated? Where >>> does the latency really come from? >>> >>> Is it: >>> 1. the access point's downlink queue, feeding into the WiFi network, >> >> This we mostly fixed, but only if you're on a recent OpenWrt with the >> right WiFi drivers. > > Well okay... I was curious about where the bottleneck is. I can translate my question into: "if Cake is installed everywhere, where does it have the most work to do?". > > >> Otherwise, this is a major source of latency *if* >> the WiFi link is faster than the downlink from the internet. > > Huh? Slower, you mean? > > >> This >> depends on both the internet connection and the current rate each WiFi >> station operates at, so it can vary wildly, and on very short time >> scales. > > Sure... I was asking for the "if" in your statement above - since this is an operationally-oriented list: what do people see? What is the more common case? > > >>> 2. the modem's downlink queue, feeding into the access point, >> >> If your internet (downlink) connection is slower than your WiFi link, >> this is where you'll get the queueing. > > Yes sure :-) see above: I wanted to know what the more common case is, in households that people on this list have dealt with. > > >>> 3. the modem's uplink queue, >> >> As above, but in the other direction - but as uplinks tend to be >> asymmetric, this direction is often more of a problem. >> >>> 4. the access point's uplink queue towards the modem (hm, that seems >>> silly, surely the AP-modem connection is fast... so perhaps, instead: >>> the queue in the host, as it wants to send data towards the access >>> point) >> >> Yeah, that would be in the host; but host drivers can suffer from severe >> bufferbloat as well, especially as rates drop (since the buffers are >> often tuned for the maximum throughput the device can deliver in >> best-case signal conditions). >> >>> or is it a combination of these? >> >> Usually it's a combination; especially since the WiFi capacity varies >> wildly with signal conditions (as devices move around relative to the >> AP), general link usage (more devices active mean less available >> capacity for each device, exacerbated by airtime unfairness), and >> interference. Also there are things like excessive retries causing HoL >> blocking. > > Man, what an academic answer! Makes me think you have a PhD, or something! What *theoretically* can happen is not what I was fishing for :-D > > >>> I guess that, with openwrt, Cake is operating on the queue that's >>> feeding the wifi network, as the modem's queue is out of its >>> control... so: is this where the bottleneck usually is? >> >> It certainly used to be; but as uplink connection speeds improve, the >> bottleneck moves to the WiFi link. > > Yessss, that's why I was asking.... > > >> The extent to which this happens >> depends on where you are in the world; personally I've been bottlenecked >> on the WiFi link ever since I got a fibre upstream (and with 802.11ax >> rates maxing at >1Gbps, maybe that'll change again?). > > THIS is what I was after :) one data point, cool - so far, so good... > > Cheers, > Michael > > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast From toke at toke.dk Tue Oct 6 08:44:27 2020 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Tue, 06 Oct 2020 14:44:27 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: References: <87d01vfue4.fsf@toke.dk> Message-ID: <87a6wzfrro.fsf@toke.dk> Michael Welzl writes: > Hi, and thanks for a quick answer! > > But, it's not quite what I was looking for.... see below: > >> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: >> >> Michael Welzl writes: >> >>> Hi all, >>> >>> A simple question to y'all who spent so much time on Cake and things >>> ... in a household using WiFi, which buffer is usually bloated? Where >>> does the latency really come from? >>> >>> Is it: >>> 1. the access point's downlink queue, feeding into the WiFi network, >> >> This we mostly fixed, but only if you're on a recent OpenWrt with the >> right WiFi drivers. > > Well okay... I was curious about where the bottleneck is. I can > translate my question into: "if Cake is installed everywhere, where > does it have the most work to do?". Well, CAKE only runs on the upstream link, so that's where it does its work. The software shaper model doesn't really work that well on WiFi, so we generally encourage people to just run fixed drivers there instead. That being said I have heard of one or two WiFi deployments where that was not an option, and where CAKE was used as a shaper instead. This was for a fixed WiFi backhaul, though, and even so they had to set the shaper quite a lot lower than the max bandwidth to get reliable performance. >> Otherwise, this is a major source of latency *if* >> the WiFi link is faster than the downlink from the internet. > > Huh? Slower, you mean? No, if the WiFi link is faster, the upstream link becomes the bottleneck and CAKE has work to do :) >> This >> depends on both the internet connection and the current rate each WiFi >> station operates at, so it can vary wildly, and on very short time >> scales. > > Sure... I was asking for the "if" in your statement above - since this > is an operationally-oriented list: what do people see? What is the > more common case? Right, well as you can probably tell that might not have been entirely clear from your initial post ;) >> The extent to which this happens depends on where you are in the >> world; personally I've been bottlenecked on the WiFi link ever since >> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps, >> maybe that'll change again?). > > THIS is what I was after :) one data point, cool - so far, so good... Ah, you're after anecdotes - well why didn't you say so? ;) In that case I'll add that my own connection is the only one I've come across where the WiFi link is *never* the bottleneck. In Denmark we are finally (slowly) getting out of the dark ages as far as fibre deployments are concerned, but most operators will sell connections capped at 100Mbps or 250Mbps, which is still less than the throughout of a 802.11ac link with good signal conditions (my phone consistently gets ~250-350 Mbps on a speedtest.net run). DSL connections tend to have awful latency, and are still quite common, but they are pretty easy to fix with CAKE. Cable connections likewise, or so is my impression (those are not so common around these parts). The worst are definitely LTE/mobile broadband connections. Wildly varying link speeds, and awful over-buffering; so you really have to clamp them down to get anything useful out of CAKE. -Toke From muscariello at ieee.org Tue Oct 6 09:09:58 2020 From: muscariello at ieee.org (Luca Muscariello) Date: Tue, 6 Oct 2020 15:09:58 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: References: <87d01vfue4.fsf@toke.dk> Message-ID: Hi Michael, In my case (WAN = GPON DL/UP 2Gbps/500Mbps) some devices are bottlenecked at the AP, others are bottlenecked at the switch, like my NAS which is multi-homed with 2 x 1GE ports to the switch. WAN downlink is hardly ever the bottleneck. However, backhaul links are 10GE and there may be congestion there depending on the level of over subscription. (From a container in the NAS) Server: ORANGE FRANCE - Paris (id = 24215) ISP: Orange Latency: 0.92 ms (0.03 ms jitter) Download: 935.31 Mbps (data used: 433.1 MB) Upload: 599.02 Mbps (data used: 645.4 MB) Packet Loss: 0.0% For a laptop connected to one AP (line of sight available) I get 500Mbps DL/UL speedtest. For the uplink it is hard to say if the bottleneck is the uplink of the router or one of the three APs, or the switch? At 5GHz no device goes below ~700Mbps (PHY) rate according to the WLC. My case is not really common now, but with fiber adoption growth and more people working from home I expect this to become more common. WMM helps with all the online meetings we do these days. For video end-points (like Cisco DX) I only use ethernet. With fiber growth in the home, investments in the backhaul become critical to keep up with demand. For xDSL the bottleneck is the WAN with high probability. Luca On Tue, Oct 6, 2020 at 2:15 PM Michael Welzl wrote: > Hi, and thanks for a quick answer! > > But, it's not quite what I was looking for.... see below: > > > On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: > > > > Michael Welzl writes: > > > >> Hi all, > >> > >> A simple question to y'all who spent so much time on Cake and things > >> ... in a household using WiFi, which buffer is usually bloated? Where > >> does the latency really come from? > >> > >> Is it: > >> 1. the access point's downlink queue, feeding into the WiFi network, > > > > This we mostly fixed, but only if you're on a recent OpenWrt with the > > right WiFi drivers. > > Well okay... I was curious about where the bottleneck is. I can translate > my question into: "if Cake is installed everywhere, where does it have the > most work to do?". > > > > Otherwise, this is a major source of latency *if* > > the WiFi link is faster than the downlink from the internet. > > Huh? Slower, you mean? > > > > This > > depends on both the internet connection and the current rate each WiFi > > station operates at, so it can vary wildly, and on very short time > > scales. > > Sure... I was asking for the "if" in your statement above - since this is > an operationally-oriented list: what do people see? What is the more common > case? > > > >> 2. the modem's downlink queue, feeding into the access point, > > > > If your internet (downlink) connection is slower than your WiFi link, > > this is where you'll get the queueing. > > Yes sure :-) see above: I wanted to know what the more common case > is, in households that people on this list have dealt with. > > > >> 3. the modem's uplink queue, > > > > As above, but in the other direction - but as uplinks tend to be > > asymmetric, this direction is often more of a problem. > > > >> 4. the access point's uplink queue towards the modem (hm, that seems > >> silly, surely the AP-modem connection is fast... so perhaps, instead: > >> the queue in the host, as it wants to send data towards the access > >> point) > > > > Yeah, that would be in the host; but host drivers can suffer from severe > > bufferbloat as well, especially as rates drop (since the buffers are > > often tuned for the maximum throughput the device can deliver in > > best-case signal conditions). > > > >> or is it a combination of these? > > > > Usually it's a combination; especially since the WiFi capacity varies > > wildly with signal conditions (as devices move around relative to the > > AP), general link usage (more devices active mean less available > > capacity for each device, exacerbated by airtime unfairness), and > > interference. Also there are things like excessive retries causing HoL > > blocking. > > Man, what an academic answer! Makes me think you have a PhD, or > something! What *theoretically* can happen is not what I was fishing for > :-D > > > >> I guess that, with openwrt, Cake is operating on the queue that's > >> feeding the wifi network, as the modem's queue is out of its > >> control... so: is this where the bottleneck usually is? > > > > It certainly used to be; but as uplink connection speeds improve, the > > bottleneck moves to the WiFi link. > > Yessss, that's why I was asking.... > > > > The extent to which this happens > > depends on where you are in the world; personally I've been bottlenecked > > on the WiFi link ever since I got a fibre upstream (and with 802.11ax > > rates maxing at >1Gbps, maybe that'll change again?). > > THIS is what I was after :) one data point, cool - so far, so good... > > Cheers, > Michael > > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -------------- next part -------------- An HTML attachment was scrubbed... URL: From chromatix99 at gmail.com Tue Oct 6 09:21:29 2020 From: chromatix99 at gmail.com (Jonathan Morton) Date: Tue, 6 Oct 2020 16:21:29 +0300 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: <87a6wzfrro.fsf@toke.dk> References: <87d01vfue4.fsf@toke.dk> <87a6wzfrro.fsf@toke.dk> Message-ID: > On 6 Oct, 2020, at 3:44 pm, Toke Høiland-Jørgensen via Make-wifi-fast wrote: > > The worst are definitely LTE/mobile broadband connections. Wildly > varying link speeds, and awful over-buffering; so you really have to > clamp them down to get anything useful out of CAKE. Out here in rural Finland, that's what I have to deal with. Meanwhile the wifi bands are very quiet and I'm never far from the AP, so there is plenty of throughput there. I'm sure there are cases with wired WAN and noisier wifi airspace where the wifi could often be the bottleneck. The balance will vary for different people at different times. But also you should consider the case of transferring files over wifi to/from some other machine on the LAN. In that case the Internet connection doesn't matter, and the wifi will always be the bottleneck. The lesson is that you need to address bloat everywhere in order to cover all possible use cases. - Jonathan Morton From michawe at ifi.uio.no Tue Oct 6 09:24:26 2020 From: michawe at ifi.uio.no (Michael Welzl) Date: Tue, 6 Oct 2020 15:24:26 +0200 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: <87a6wzfrro.fsf@toke.dk> References: <87d01vfue4.fsf@toke.dk> <87a6wzfrro.fsf@toke.dk> Message-ID: Thanks a lot - just the info I was looking for (also from others who have responded in the meantime, thanks!) > On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen wrote: > > Michael Welzl writes: > >> Hi, and thanks for a quick answer! >> >> But, it's not quite what I was looking for.... see below: >> >>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: >>> >>> Michael Welzl writes: >>> >>>> Hi all, >>>> >>>> A simple question to y'all who spent so much time on Cake and things >>>> ... in a household using WiFi, which buffer is usually bloated? Where >>>> does the latency really come from? >>>> >>>> Is it: >>>> 1. the access point's downlink queue, feeding into the WiFi network, >>> >>> This we mostly fixed, but only if you're on a recent OpenWrt with the >>> right WiFi drivers. >> >> Well okay... I was curious about where the bottleneck is. I can >> translate my question into: "if Cake is installed everywhere, where >> does it have the most work to do?". > > Well, CAKE only runs on the upstream link, so that's where it does its > work. The software shaper model doesn't really work that well on WiFi, > so we generally encourage people to just run fixed drivers there > instead. That being said I have heard of one or two WiFi deployments > where that was not an option, and where CAKE was used as a shaper > instead. This was for a fixed WiFi backhaul, though, and even so they > had to set the shaper quite a lot lower than the max bandwidth to get > reliable performance. > >>> Otherwise, this is a major source of latency *if* >>> the WiFi link is faster than the downlink from the internet. >> >> Huh? Slower, you mean? > > No, if the WiFi link is faster, the upstream link becomes the bottleneck > and CAKE has work to do :) > >>> This >>> depends on both the internet connection and the current rate each WiFi >>> station operates at, so it can vary wildly, and on very short time >>> scales. >> >> Sure... I was asking for the "if" in your statement above - since this >> is an operationally-oriented list: what do people see? What is the >> more common case? > > Right, well as you can probably tell that might not have been entirely > clear from your initial post ;) > >>> The extent to which this happens depends on where you are in the >>> world; personally I've been bottlenecked on the WiFi link ever since >>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps, >>> maybe that'll change again?). >> >> THIS is what I was after :) one data point, cool - so far, so good... > > Ah, you're after anecdotes - well why didn't you say so? ;) > > In that case I'll add that my own connection is the only one I've come > across where the WiFi link is *never* the bottleneck. In Denmark we are > finally (slowly) getting out of the dark ages as far as fibre > deployments are concerned, but most operators will sell connections > capped at 100Mbps or 250Mbps, which is still less than the throughout of > a 802.11ac link with good signal conditions (my phone consistently gets > ~250-350 Mbps on a speedtest.net run). > > DSL connections tend to have awful latency, and are still quite common, > but they are pretty easy to fix with CAKE. Cable connections likewise, > or so is my impression (those are not so common around these parts). > > The worst are definitely LTE/mobile broadband connections. Wildly > varying link speeds, and awful over-buffering; so you really have to > clamp them down to get anything useful out of CAKE. > > -Toke From bob.mcmahon at broadcom.com Tue Oct 6 20:10:06 2020 From: bob.mcmahon at broadcom.com (Bob McMahon) Date: Tue, 6 Oct 2020 17:10:06 -0700 Subject: [Make-wifi-fast] Where is the bloat in WiFi? In-Reply-To: References: <87d01vfue4.fsf@toke.dk> <87a6wzfrro.fsf@toke.dk> Message-ID: You might want to try iperf 2.0.14 --trip-times option to measure your links. Below a run with a PC w/a 1G NIC connected to the 1G cable MODEM no WiFi, i.e. wired, full duplex socket. The uplink tcp write latencies are larger than one second. Downlinks just under 100 ms. Slow down the uplink traffic to less than the congestion point and write to read latency drops to ~65 ms. I think this suggests my CABLE MODEM is full of bloat. UPLINK ONLY -------------------- Server: [rjmcmahon at bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr ------------------------------------------------------------ Server listening on TCP port 5001 with pid 25408 Read buffer size: 128 KByte (Dist bin width=16.0 KByte) TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58158 (trip-times) (MSS=1448) (peer 2.0.14-alpha) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist [ 4] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 59.680/51.222/96.282/34.950 ms (10/131072) 80.9 KByte 21.96 305=305:0:0:0:0:0:0:0 [ 4] 1.00-2.00 sec 1.12 MBytes 9.44 Mbits/sec 65.218/51.951/87.895/10.877 ms (9/131072) 79.6 KByte 18.09 262=262:0:0:0:0:0:0:0 [ 4] 2.00-3.00 sec 1.25 MBytes 10.5 Mbits/sec 62.479/51.170/67.962/ 5.189 ms (10/131072) 76.3 KByte 20.98 289=289:0:0:0:0:0:0:0 [ 4] 3.00-4.00 sec 1.12 MBytes 9.44 Mbits/sec 62.861/52.478/68.339/ 5.212 ms (9/131072) 76.7 KByte 18.77 262=262:0:0:0:0:0:0:0 [ 4] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec 64.789/50.914/91.907/12.235 ms (10/131072) 79.1 KByte 20.23 292=292:0:0:0:0:0:0:0 [ 4] 5.00-6.00 sec 1.14 MBytes 9.60 Mbits/sec 64.618/50.914/68.277/ 5.696 ms (9/133334) 80.2 KByte 18.57 259=259:0:0:0:0:0:0:0 [ 4] 6.00-7.00 sec 1.23 MBytes 10.3 Mbits/sec 66.905/50.936/87.335/10.856 ms (10/129035) 80.4 KByte 19.29 279=279:0:0:0:0:0:0:0 [ 4] 7.00-8.00 sec 1.16 MBytes 9.70 Mbits/sec 68.954/51.902/100.783/15.733 ms (9/134782) 86.6 KByte 17.59 275=274:1:0:0:0:0:0:0 [ 4] 8.00-9.00 sec 1.22 MBytes 10.2 Mbits/sec 63.855/51.386/69.919/ 5.284 ms (9/141851) 84.4 KByte 19.99 283=283:0:0:0:0:0:0:0 [ 4] 9.00-10.00 sec 1.18 MBytes 9.86 Mbits/sec 67.975/50.996/88.151/11.318 ms (10/123252) 78.0 KByte 18.13 275=275:0:0:0:0:0:0:0 [ 4] 10.00-10.14 sec 205 KBytes 11.7 Mbits/sec 99.865/55.342/77.370/ 7.319 ms (2/104962) 97.6 KByte 14.61 43=43:0:0:0:0:0:0:0 [ 4] 0.00-10.14 sec 12.1 MBytes 10.0 Mbits/sec 59.580/50.914/100.783/14.104 ms (97/131072) 72.9 KByte 21.04 2824=2823:1:0:0:0:0:0:0 Client: [root at localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times -Z bbr -b 10m -e ------------------------------------------------------------ Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 48006 (1 flows) Write buffer size: 128 KByte TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.35%enp2s0 port 58160 connected with 45.33.58.123 port 5001 (trip-times) (MSS=1448) (ct=13.05 ms) [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 3] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 11/0 0 135K/49251 us 26.61 [ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 141K/24791 us 52.87 [ 3] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 135K/24973 us 47.24 [ 3] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 158K/25021 us 52.38 [ 3] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 141K/21247 us 55.52 [ 3] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 144K/23535 us 55.69 [ 3] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 135K/27130 us 43.48 [ 3] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 135K/24813 us 52.82 [ 3] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 0 155K/22697 us 51.97 [ 3] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 0 124K/21954 us 59.70 [ 3] 10.00-10.07 sec 256 KBytes 31.6 Mbits/sec 2/0 0 124K/21954 us 179.64 [ 3] 0.00-10.07 sec 12.3 MBytes 10.2 Mbits/sec 99/0 0 124K/21954 us 58.12 FULL DUPLEX: --------------------- Server: [rjmcmahon at bobcat iperf2-code]$ src/iperf -s -i 1 -e -Z bbr ------------------------------------------------------------ Server listening on TCP port 5001 with pid 25379 Read buffer size: 128 KByte (Dist bin width=16.0 KByte) TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 45.33.58.123%eth0 port 5001 connected with 73.92.17.76 port 58152 (full-duplex) (trip-times) (MSS=1448) (peer 2.0.14-alpha) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist [ 4] 0.00-1.00 sec 1.05 MBytes 8.79 Mbits/sec 547.494/68.304/904.670/301.250 ms (8/137379) 9.75 MByte 2.01 345=344:1:0:0:0:0:0:0 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 4] 0.00-1.00 sec 40.3 MBytes 338 Mbits/sec 323/0 0 1267K/20319 us 2077.13 [ ID] Interval Transfer Bandwidth [FD4] 0.00-1.00 sec 41.3 MBytes 346 Mbits/sec [ 4] 1.00-2.00 sec 1.34 MBytes 11.3 Mbits/sec 1307.675/904.670/1454.073/145.211 ms (11/127950) 2.30 MByte 1.08 277=262:5:4:6:0:0:0:0 [ 4] 1.00-2.00 sec 38.8 MBytes 325 Mbits/sec 310/0 0 1001K/24428 us 1663.35 [FD4] 1.00-2.00 sec 40.1 MBytes 336 Mbits/sec [ 4] 2.00-3.00 sec 38.0 MBytes 319 Mbits/sec 304/0 0 1156K/26355 us 1511.89 [ 4] 2.00-3.00 sec 1.33 MBytes 11.1 Mbits/sec 1298.825/1139.072/1220.561/27.582 ms (10/139008) 1.81 MByte 1.07 372=366:2:1:3:0:0:0:0 [FD4] 2.00-3.00 sec 39.3 MBytes 330 Mbits/sec [ 4] 3.00-4.00 sec 1.32 MBytes 11.1 Mbits/sec 1302.239/1150.168/1234.313/22.922 ms (11/125844) 1.76 MByte 1.06 217=199:4:8:4:2:0:0:0 [ 4] 3.00-4.00 sec 39.6 MBytes 332 Mbits/sec 317/0 0 970K/19064 us 2179.49 [FD4] 3.00-4.00 sec 40.9 MBytes 343 Mbits/sec [ 4] 4.00-5.00 sec 38.5 MBytes 323 Mbits/sec 308/0 0 998K/20599 us 1959.81 [ 4] 4.00-5.00 sec 1.39 MBytes 11.7 Mbits/sec 1310.588/1172.769/1233.522/21.090 ms (11/132426) 1.79 MByte 1.11 223=196:18:5:4:0:0:0:0 [FD4] 4.00-5.00 sec 39.9 MBytes 335 Mbits/sec [ 4] 5.00-6.00 sec 38.6 MBytes 324 Mbits/sec 309/0 0 1340K/17717 us 2286.01 [ 4] 5.00-6.00 sec 1.33 MBytes 11.2 Mbits/sec 1305.731/1173.179/1220.196/17.125 ms (11/127160) 1.72 MByte 1.07 158=136:10:5:6:1:0:0:0 [FD4] 5.00-6.00 sec 40.0 MBytes 335 Mbits/sec [ 4] 6.00-7.00 sec 1.36 MBytes 11.4 Mbits/sec 1301.735/1162.410/1216.840/20.348 ms (10/142193) 1.87 MByte 1.09 140=116:7:8:6:3:0:0:0 [ 4] 6.00-7.00 sec 39.4 MBytes 330 Mbits/sec 315/0 0 1074K/20040 us 2060.26 [FD4] 6.00-7.00 sec 40.7 MBytes 342 Mbits/sec [ 4] 7.00-8.00 sec 1.35 MBytes 11.3 Mbits/sec 1291.605/1160.382/1220.122/18.092 ms (11/128608) 1.70 MByte 1.10 216=198:5:6:7:0:0:0:0 [ 4] 7.00-8.00 sec 38.9 MBytes 326 Mbits/sec 311/0 0 1074K/30895 us 1319.42 [FD4] 7.00-8.00 sec 40.2 MBytes 337 Mbits/sec [ 4] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec 299/0 0 1264K/29185 us 1342.83 [ 4] 8.00-9.00 sec 1.27 MBytes 10.7 Mbits/sec 1326.329/1173.254/1244.452/23.637 ms (10/133360) 1.73 MByte 1.01 110=86:9:7:5:1:0:2:0 [FD4] 8.00-9.00 sec 38.6 MBytes 324 Mbits/sec [ 4] 9.00-10.00 sec 1.34 MBytes 11.2 Mbits/sec 1312.714/1134.091/1279.147/46.480 ms (11/127818) 1.69 MByte 1.07 124=101:12:6:2:1:0:2:0 [ 4] 9.00-10.00 sec 39.2 MBytes 329 Mbits/sec 314/0 0 967K/21976 us 1872.80 [FD4] 9.00-10.00 sec 40.6 MBytes 341 Mbits/sec [ 4] 10.00-10.00 sec 256 KBytes 477 Mbits/sec 2/0 0 967K/21976 us 2711.06 [ 4] 0.00-10.00 sec 389 MBytes 326 Mbits/sec 3112/0 0 967K/21976 us 1854.69 [ 4] 10.00-11.00 sec 963 KBytes 7.89 Mbits/sec 1624.535/1134.091/1548.718/32.733 ms (8/123261) 2.10 MByte 0.61 215=210:3:0:0:1:1:0:0 [ 4] 11.00-11.41 sec 622 KBytes 12.6 Mbits/sec 1780.227/1454.172/1511.515/16.223 ms (5/127355) 2.58 MByte 0.88 29=11:11:4:3:0:0:0:0 [ 4] 0.00-11.41 sec 14.6 MBytes 10.8 Mbits/sec 1184.903/68.304/1548.718/215.041 ms (117/131072) 1.52 MByte 1.13 2426=2225:87:54:46:9:1:4:0 [FD4] 10.00-1602028944.71 sec 1.80 MBytes 0.009 bits/sec [FD4] 0.00-1602028944.71 sec 404 MBytes 2.11 bits/sec Client [root at localhost iperf2-code]# iperf -c bobcat.rjmcmahon.com -i1 --trip-times --full-duplex -Z bbr WARNING: tcp congestion control will only be applied on transmit traffic, use -Z on the server ------------------------------------------------------------ Client connecting to bobcat.rjmcmahon.com, TCP port 5001 with pid 47994 (1 flows) Write buffer size: 128 KByte TCP congestion control set to bbr TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.35%enp2s0 port 58154 connected with 45.33.58.123 port 5001 (full-duplex) (trip-times) (MSS=1448) (ct=12.42 ms) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist [ 3] 0.00-1.00 sec 35.8 MBytes 301 Mbits/sec 89.806/29.604/103.341/16.204 ms (286/131398) 3.58 MByte 418.46 17764=17726:35:3:0:0:0:0:0 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr [ 3] 0.00-1.00 sec 2.50 MBytes 21.0 Mbits/sec 21/0 91 62K/44390 us 59.06 [ ID] Interval Transfer Bandwidth [FD3] 0.00-1.00 sec 38.3 MBytes 322 Mbits/sec [ 3] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 8 56K/36570 us 35.84 [ 3] 1.00-2.00 sec 38.7 MBytes 325 Mbits/sec 94.258/61.108/104.117/ 9.454 ms (310/130882) 3.63 MByte 430.45 20775=20768:5:2:0:0:0:0:0 [FD3] 1.00-2.00 sec 39.9 MBytes 335 Mbits/sec [ 3] 2.00-3.00 sec 38.6 MBytes 324 Mbits/sec 94.554/62.312/105.451/ 9.324 ms (308/131379) 3.69 MByte 427.96 18929=18913:15:1:0:0:0:0:0 [ 3] 2.00-3.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 12 65K/54911 us 26.26 [FD3] 2.00-3.00 sec 40.0 MBytes 335 Mbits/sec [ 3] 3.00-4.00 sec 37.3 MBytes 313 Mbits/sec 95.362/62.161/117.696/11.285 ms (299/130672) 3.55 MByte 409.71 17567=17537:27:3:0:0:0:0:0 [ 3] 3.00-4.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 20 67K/46946 us 30.71 [FD3] 3.00-4.00 sec 38.6 MBytes 324 Mbits/sec [ 3] 4.00-5.00 sec 1.25 MBytes 10.5 Mbits/sec 10/0 51 59K/67662 us 19.37 [ 3] 4.00-5.00 sec 38.4 MBytes 322 Mbits/sec 90.268/64.571/104.215/10.430 ms (307/131134) 3.45 MByte 445.99 19095=19077:16:2:0:0:0:0:0 [FD3] 4.00-5.00 sec 39.6 MBytes 333 Mbits/sec [ 3] 5.00-6.00 sec 38.5 MBytes 323 Mbits/sec 88.832/64.621/102.534/ 9.742 ms (308/130979) 3.40 MByte 454.13 17674=17657:17:0:0:0:0:0:0 [ 3] 5.00-6.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 48 65K/64456 us 22.37 [FD3] 5.00-6.00 sec 39.8 MBytes 334 Mbits/sec [ 3] 6.00-7.00 sec 38.4 MBytes 322 Mbits/sec 87.121/62.694/102.124/10.260 ms (307/131199) 3.24 MByte 462.33 19853=19852:1:0:0:0:0:0:0 [ 3] 6.00-7.00 sec 3.75 MBytes 31.5 Mbits/sec 30/0 59 67K/151435 us 25.97 [FD3] 6.00-7.00 sec 42.2 MBytes 354 Mbits/sec [ 3] 7.00-8.00 sec 38.5 MBytes 323 Mbits/sec 87.936/62.694/102.226/10.016 ms (308/131222) 3.41 MByte 459.61 19067=19052:15:0:0:0:0:0:0 [ 3] 7.00-8.00 sec 1.12 MBytes 9.44 Mbits/sec 9/0 60 73K/56735 us 20.79 [FD3] 7.00-8.00 sec 39.7 MBytes 333 Mbits/sec [ 3] 8.00-9.00 sec 35.9 MBytes 301 Mbits/sec 91.441/56.797/123.585/14.616 ms (287/131122) 3.40 MByte 411.54 18900=18881:10:2:0:0:0:0:7 [ 3] 8.00-9.00 sec 1.50 MBytes 12.6 Mbits/sec 13/0 26 82K/49077 us 32.05 [FD3] 8.00-9.00 sec 37.4 MBytes 314 Mbits/sec [ 3] 9.00-10.00 sec 36.3 MBytes 305 Mbits/sec 90.461/60.571/148.892/15.939 ms (291/130834) 3.17 MByte 420.88 17888=17871:17:0:0:0:0:0:0 [ 3] 10.00-10.07 sec 2.72 MBytes 310 Mbits/sec 91.852/69.979/102.633/ 8.280 ms (22/129636) 5.96 MByte 421.94 1418=1414:4:0:0:0:0:0:0 [ 3] 0.00-10.07 sec 379 MBytes 316 Mbits/sec 90.712/29.604/148.892/12.195 ms (3033/131072) 3.41 MByte 435.04 188930=188748:162:13:0:0:0:0:7 [ 3] 9.00-10.00 sec 1.38 MBytes 11.5 Mbits/sec 11/0 0 5K/55760 us 25.86 [FD3] 9.00-10.00 sec 40.4 MBytes 339 Mbits/sec [ 3] 10.00-10.49 sec 223 KBytes 3.69 Mbits/sec 2/0 0 5K/55760 us 8.28 [ 3] 0.00-10.49 sec 17.1 MBytes 13.7 Mbits/sec 139/0 424 5K/55760 us 30.63 [FD3] 10.00-10.49 sec 223 KBytes 3.77 Mbits/sec [FD3] 0.00-10.49 sec 396 MBytes 317 Mbits/sec Bob On Tue, Oct 6, 2020 at 6:24 AM Michael Welzl wrote: > Thanks a lot - just the info I was looking for (also from others who have > responded in the meantime, thanks!) > > > > On 6 Oct 2020, at 14:44, Toke Høiland-Jørgensen wrote: > > > > Michael Welzl writes: > > > >> Hi, and thanks for a quick answer! > >> > >> But, it's not quite what I was looking for.... see below: > >> > >>> On 6 Oct 2020, at 13:47, Toke Høiland-Jørgensen wrote: > >>> > >>> Michael Welzl writes: > >>> > >>>> Hi all, > >>>> > >>>> A simple question to y'all who spent so much time on Cake and things > >>>> ... in a household using WiFi, which buffer is usually bloated? Where > >>>> does the latency really come from? > >>>> > >>>> Is it: > >>>> 1. the access point's downlink queue, feeding into the WiFi network, > >>> > >>> This we mostly fixed, but only if you're on a recent OpenWrt with the > >>> right WiFi drivers. > >> > >> Well okay... I was curious about where the bottleneck is. I can > >> translate my question into: "if Cake is installed everywhere, where > >> does it have the most work to do?". > > > > Well, CAKE only runs on the upstream link, so that's where it does its > > work. The software shaper model doesn't really work that well on WiFi, > > so we generally encourage people to just run fixed drivers there > > instead. That being said I have heard of one or two WiFi deployments > > where that was not an option, and where CAKE was used as a shaper > > instead. This was for a fixed WiFi backhaul, though, and even so they > > had to set the shaper quite a lot lower than the max bandwidth to get > > reliable performance. > > > >>> Otherwise, this is a major source of latency *if* > >>> the WiFi link is faster than the downlink from the internet. > >> > >> Huh? Slower, you mean? > > > > No, if the WiFi link is faster, the upstream link becomes the bottleneck > > and CAKE has work to do :) > > > >>> This > >>> depends on both the internet connection and the current rate each WiFi > >>> station operates at, so it can vary wildly, and on very short time > >>> scales. > >> > >> Sure... I was asking for the "if" in your statement above - since this > >> is an operationally-oriented list: what do people see? What is the > >> more common case? > > > > Right, well as you can probably tell that might not have been entirely > > clear from your initial post ;) > > > >>> The extent to which this happens depends on where you are in the > >>> world; personally I've been bottlenecked on the WiFi link ever since > >>> I got a fibre upstream (and with 802.11ax rates maxing at >1Gbps, > >>> maybe that'll change again?). > >> > >> THIS is what I was after :) one data point, cool - so far, so good... > > > > Ah, you're after anecdotes - well why didn't you say so? ;) > > > > In that case I'll add that my own connection is the only one I've come > > across where the WiFi link is *never* the bottleneck. In Denmark we are > > finally (slowly) getting out of the dark ages as far as fibre > > deployments are concerned, but most operators will sell connections > > capped at 100Mbps or 250Mbps, which is still less than the throughout of > > a 802.11ac link with good signal conditions (my phone consistently gets > > ~250-350 Mbps on a speedtest.net run). > > > > DSL connections tend to have awful latency, and are still quite common, > > but they are pretty easy to fix with CAKE. Cable connections likewise, > > or so is my impression (those are not so common around these parts). > > > > The worst are definitely LTE/mobile broadband connections. Wildly > > varying link speeds, and awful over-buffering; so you really have to > > clamp them down to get anything useful out of CAKE. > > > > -Toke > > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4163 bytes Desc: S/MIME Cryptographic Signature URL: From bob.mcmahon at broadcom.com Wed Oct 21 13:54:18 2020 From: bob.mcmahon at broadcom.com (Bob McMahon) Date: Wed, 21 Oct 2020 10:54:18 -0700 Subject: [Make-wifi-fast] pressure for better clocks Message-ID: Hi All, Iperf 2.0.14 provides end/end latency measurements but requires sync'ed clocks. The lack of attention to end/end latency is a major deficit to the networking industry and impairs user experience. Ping isn't the real deal either. I find one of the many problems with the tech industry is that cheap mass market stuff proliferates. An example of this is the oscillators used for time keeping which run things like CPUs. There are many ways to get better time without spending too much. The GPS atomic clocks are basically available for free over most of the planet. Cellular clocks, synced to GPS, are available indoors. Here is a good article: https://www.pcworld.com/article/2891892/why-computers-still-struggle-to-tell-the-time.html *But computer makers often use inexpensive crystals costing only a few cents each, which can compromise accuracy. “If you buy server-class hardware, you will get cheap crystal, and time will wander if you don’t do something about it,” Neville-Neil said.The average crystal ends up being about as accurate as a mechanical watch* Bob McMahon -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4163 bytes Desc: S/MIME Cryptographic Signature URL: