[Cake] Ubiquity (Unifi ) Smart Queues
dave seddon
dave.seddon.ca at gmail.com
Tue Jan 2 16:15:18 EST 2024
Thanks Sebastian!
Now I see the rates!!
I actually reduced the rates to ensure this device is the bottleneck 80/10
Mb/s
[image: image.png]
root at USG-Pro-4:~# tc -d class show dev eth2
class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil
9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b
level 0
class fq_codel 100:12c parent 100:
class fq_codel 100:213 parent 100:
class fq_codel 100:22e parent 100:
root at USG-Pro-4:~# tc -d class show dev ifb_eth2
class htb 1:10 root leaf 100: prio 0 quantum 200000 rate 76000Kbit ceil
76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead
0b level 0
class fq_codel 100:2c8 parent 100:
class fq_codel 100:3df parent 100:
On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller <moeller0 at gmx.de> wrote:
> Hi Dave.
>
> just a few comments from the peanut gallery...
>
>
> > On Jan 2, 2024, at 19:59, dave seddon via Cake <
> cake at lists.bufferbloat.net> wrote:
> >
> > G'day,
> >
> > Happy new year y'all
>
> +1
>
> >
> > I thought people might be interested to see what Ubiquity/Unifi is doing
> with "Smart Queues" on their devices. The documentation on their website
> is not very informative.
> >
> > Hopefully, this is vaguely interesting because Ubiquity is widely
> deployed and apparently they have a market cap of >$8 billion, so you would
> hope they do a "good job" (... Seems like they might be a target customer
> for libreqos )
> >
> > <image.png>
> > https://finance.yahoo.com/quote/ui/
> >
> > ( I use Unifi because their wifi stuff seems ok, and all the
> switching/routing/wifi is all integrated into the single gui control
> system. Also honestly, I'm not sure I know how to do prefix delegation
> stuff on Linux by hand. )
> >
> > Network diagram
> >
> > Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <--->
> [Switches] <----> Access points
> >
> > "Smart Queue" Configuration
> > Ubiquity doesn't have many knobs, you just enable "smart queues" and set
> the bandwidth.
> >
> >
> >
> >
> > "Smart Queue" Implementation
> >
> > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT
> cake, but fq_codel.
> >
> > And cake isn't available :(
> >
> > root at USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
> > Unknown qdisc "cake", hence option "bandwidth" is unparsable
> >
> > Outbound eth2
> >
> > root at USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver
> 3.17
> > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078
> requeues 0) <---- OVERLIMITS?
> > backlog 0b 0p requeues 0
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514
> target 5.0ms interval 100.0ms ecn
> > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues
> 0) <----- DROPS
> > backlog 0b 0p requeues 0
> > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
> > new_flows_len 1 old_flows_len 1
> > qdisc ingress ffff: parent ffff:fff1 ----------------
> > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues
> 0)
> > backlog 0b 0p requeues 0
> > • target 5.0ms is the default (
> https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder
> if they did much testing on this hardware?
>
> [SM] Not sure whether playing with target in isolation would be much use,
> in codel theory target should be 5-10% of interval ans interval should be
> in the order of magnitude of to be handled RTTs (the default is 100ms wich
> works reasonably well even across the Atlantic, but you probably knew all
> that).
>
> > • ( I actually have a spare "wan" ethernet port, so I
> guess I could hook up a PC and perform a flent test. )
> > • It's unclear to me what the "htb" is doing, because I would have
> expected the download/upload rates to be configured here, but they appear
> not to be
>
> [SM] Likely because HTB does not reveal this when asked with the `-s`
> option, try `-q` instead and not as qdisc but as class (so maybe `tc -d
> class show dev eth2`).
>
> > • I'm not really sure what "overlimits" means or what that does,
> and tried looking this up, but I guess the kernel source is likely the
> "best" documentation for this. Maybe this means it's dropping? Or is it
> ECN?
>
> I think this text about TBF explains this reasonably well (HTB is
> essentially a hierarchical version of TBF):
>
> see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html
>
> 9.2.2. Token Bucket Filter
>
> The Token Bucket Filter (TBF) is a simple qdisc that only passes packets
> arriving at a rate which is not exceeding some administratively set rate,
> but with the possibility to allow short bursts in excess of this rate.
>
> TBF is very precise, network- and processor friendly. It should be your
> first choice if you simply want to slow an interface down!
>
> The TBF implementation consists of a buffer (bucket), constantly filled by
> some virtual pieces of information called tokens, at a specific rate (token
> rate). The most important parameter of the bucket is its size, that is the
> number of tokens it can store.
>
> Each arriving token collects one incoming data packet from the data queue
> and is then deleted from the bucket. Associating this algorithm with the
> two flows -- token and data, gives us three possible scenarios:
>
> • The data arrives in TBF at a rate that's equal to the rate of
> incoming tokens. In this case each incoming packet has its matching token
> and passes the queue without delay.
>
> • The data arrives in TBF at a rate that's smaller than the token
> rate. Only a part of the tokens are deleted at output of each data packet
> that's sent out the queue, so the tokens accumulate, up to the bucket size.
> The unused tokens can then be used to send data a a speed that's exceeding
> the standard token rate, in case short data bursts occur.
>
> • The data arrives in TBF at a rate bigger than the token rate.
> This means that the bucket will soon be devoid of tokens, which causes the
> TBF to throttle itself for a while. This is called an 'overlimit
> situation'. If packets keep coming in, packets will start to get dropped.
>
> The last scenario is very important, because it allows to administratively
> shape the bandwidth available to data that's passing the filter.
>
> The accumulation of tokens allows a short burst of overlimit data to be
> still passed without loss, but any lasting overload will cause packets to
> be constantly delayed, and then dropped.
>
> Please note that in the actual implementation, tokens correspond to bytes,
> not packets.
>
>
> >
> > Inbound eth2 via ifb
> >
> > root at USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver
> 3.17
> > Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339
> requeues 0) <---- OVERLIMITS?
> > backlog 0b 0p requeues 0
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514
> target 5.0ms interval 100.0ms ecn
> > Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0
> requeues 0) <---- WOW. DROPS!!
> > backlog 0b 0p requeues 0
> > maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0
> > new_flows_len 0 old_flows_len 2
> >
> > Apparently rather than applying the tc qdsic on the outbound path on the
> LAN side ( eth0 ), they are applying it inbound on the the eth2 via
> ifb_eth2.
>
> [SM] Same approach that sqm-scripts takes, if you attach the ingress
> shaper to the LAN port egress all internet traffic not traversing that
> interface will not be shaped, e.g. traffic to/from the router itself or
> WiFi traffic. If you are sure that such by-pass traffic does not exist,
> putting the ingress shaper on lan-egress can save the cost of the ifb
> indirection, but for full WiFi routers that is generally not true.
>
>
> > Initially, I was pretty surprised to see so many drops on the inbound
> path, but maybe this is actually normal?
>
> [SM] Depends on your traffic and whether ECN is used or not. In your case
> it appears ECN is not used and then DROPS are the only way fq_codel can
> tell a flow to step on the brakes....
>
> >
> > I could imagine the upstream CDNs pushing pretty hard with low RTTs, but
> I would probably have expected the bottlenecks to form at the access
> points. e.g. It's gigabit all the way until it reaches the air interface of
> the access points. .... Or do I have a problem in my LAN network?
>
> [SM] The idea is to create an artificial bittleneck (using HTB) so the
> most relevant queue is under AQM control...
>
> >
> > I wonder if I can log into the access points to look at them too?....
> >
> > ( BTW - to get to root on these devices you can SSH in as an "admin"
> users, and then just "sudo su" )
> >
> > ifconfig
> >
> > root at USG-Pro-4:~# ifconfig -a
> > eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> <---- queue len 0? Maybe this is a driver issue?
> > RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)
> >
> > eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0
> > inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0
> > inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link
> > inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:154930 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)
> >
> > eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1
> > inet addr:172.88.[SNIP] Bcast:255.255.255.255
> Mask:255.255.240.0
> > inet6 addr: [SNIP]:d474:3d71/128 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)
> >
> > eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2
> > BROADCAST MULTICAST MTU:1500 Metric:1
> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
> >
> > eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:782123 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)
> >
> > eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2695 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)
> >
> > ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27
> > inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link
> > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
> > RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0
> > TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:32
> <----- queue len 32? Curious
> > RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1
> GiB)
> >
> >
> > System info
> >
> > This has a prehistoric kernel, I guess because they have some stuff that
> taints the kernel
> >
> > root at USG-Pro-4:~# uname -a
> > Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64
> GNU/Linux
>
> [SM] I remember the time we felt great about using a series 3 kernel
> instead of old series 2 gunk, but upstream is at series 6 by now (but it
> also started to increase major numbers more aggressively after series 2)
>
> >
> > root at USG-Pro-4:~# cat /var/log/dmesg | grep taint
> > ubnt_platform: module license 'Proprietary' taints kernel.
> > Disabling lock debugging due to kernel taint
> >
> > I also notice this module, but I'm not sure it is in use.
> > /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko
> >
> >
> > root at USG-Pro-4:~# cat /proc/cpuinfo
> > system type : UBNT_E220
> > machine : Unknown
> > processor : 0
> > cpu model : Cavium Octeon II V0.1
> > BogoMIPS : 2000.00
> > wait instruction : yes
> > microsecond timers : yes
> > tlb_entries : 128
> > extra interrupt vector : yes
> > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
> > isa : mips1 mips2 mips3 mips4 mips5 mips64r2
> > ASEs implemented :
> > shadow register sets : 1
> > kscratch registers : 3
> > core : 0
> > VCED exceptions : not available
> > VCEI exceptions : not available
> >
> > processor : 1
> > cpu model : Cavium Octeon II V0.1
> > BogoMIPS : 2000.00
> > wait instruction : yes
> > microsecond timers : yes
> > tlb_entries : 128
> > extra interrupt vector : yes
> > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
> > isa : mips1 mips2 mips3 mips4 mips5 mips64r2
> > ASEs implemented :
> > shadow register sets : 1
> > kscratch registers : 3
> > core : 1
> > VCED exceptions : not available
> > VCEI exceptions : not available
> >
> >
> >
> > root at USG-Pro-4:~# ethtool -i eth2
> > driver: octeon-ethernet
> > version: 2.0
> > firmware-version:
> > bus-info: Builtin
> > supports-statistics: no
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: no
> > supports-priv-flags: no
> >
> > root at USG-Pro-4:~# ethtool -S eth2
> > no stats available
> >
> > ( Oh great! Thanks guys! )
> >
> > root at USG-Pro-4:~# netstat -ia
> > Kernel Interface table
> > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP
> TX-OVR Flg
> > eth0 1500 0 11340913 0 0 0 21612063 0 0
> 0 BMRU
> > eth1 1500 0 154902 0 0 0 233236 0 0
> 0 BMRU
> > eth2 1500 0 60898610 0 0 0 10544414 0 0
> 0 BMRU
> > eth3 1500 0 0 0 0 0 0 0 0
> 0 BM
> > eth0.20 1500 0 781992 0 0 0 480214 0 0
> 0 BMRU
> > eth0.40 1500 0 2695 0 0 0 194260 0 0
> 0 BMRU
> > ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0
> 0 BORU <---- RX drops?
> > imq0 16000 0 0 0 0 0 0 0 0
> 0 ORU
> > lo 65536 0 9255 0 0 0 9255 0 0
> 0 LRU
> > loop0 1500 0 0 0 0 0 0 0 0
> 0 BM
> > loop1 1500 0 0 0 0 0 0 0 0
> 0 BM
> > loop2 1500 0 0 0 0 0 0 0 0
> 0 BM
> > loop3 1500 0 0 0 0 0 0 0 0
> 0 BM
> > npi0 1500 0 0 0 0 0 0 0 0
> 0 BM
> > npi1 1500 0 0 0 0 0 0 0 0
> 0 BM
> > npi2 1500 0 0 0 0 0 0 0 0
> 0 BM
> > npi3 1500 0 0 0 0 0 0 0 0
> 0 BM
> >
> > root at USG-Pro-4:/opt/vyatta/etc# cat version
> > Version: v4.4.57.5578372.230112.0824
> >
> > --
> > Regards,
> > Dave Seddon
> > _______________________________________________
> > Cake mailing list
> > Cake at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
--
Regards,
Dave Seddon
+1 415 857 5102
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20240102/b6210227/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6574 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20240102/b6210227/attachment-0001.png>
More information about the Cake
mailing list