[Cake] Ubiquity (Unifi ) Smart Queues
dave seddon
dave.seddon.ca at gmail.com
Tue Jan 2 13:59:39 EST 2024
G'day,
Happy new year y'all
I thought people might be interested to see what Ubiquity/Unifi is doing
with "Smart Queues" on their devices. The documentation on their website
is not very informative.
Hopefully, this is vaguely interesting because Ubiquity is widely deployed
and apparently they have a market cap of >$8 billion, so you would hope
they do a "good job" (... Seems like they might be a target customer for
libreqos )
[image: image.png]
https://finance.yahoo.com/quote/ui/
( I use Unifi because their wifi stuff seems ok, and all the
switching/routing/wifi is all integrated into the single gui control
system. Also honestly, I'm not sure I know how to do prefix delegation
stuff on Linux by hand. )
*Network diagram*
Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <--->
[Switches] <----> Access points
*"Smart Queue" Configuration*
Ubiquity doesn't have many knobs, you just enable "smart queues" and set
the bandwidth.
[image: image.png]
*"Smart Queue" Implementation*
Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT
cake, but fq_codel.
And cake isn't available :(
root at USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
Unknown qdisc "cake", hence option "bandwidth" is unparsable
*Outbound eth2*
root at USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues
0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target
5.0ms interval 100.0ms ecn
Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues
0) <----- DROPS
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
new_flows_len 1 old_flows_len 1
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
- target 5.0ms is the default (
https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I
wonder if they did much testing on this hardware?
- ( I actually have a spare "wan" ethernet port, so I guess I could hook
up a PC and perform a flent test. )
- It's unclear to me what the "htb" is doing, because I would have
expected the download/upload rates to be configured here, but they appear
not to be
- I'm not really sure what "overlimits" means or what that does, and
tried looking this up, but I guess the kernel source is likely the "best"
documentation for this. Maybe this means it's dropping? Or is it ECN?
*Inbound eth2 via ifb*
root at USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339
requeues 0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target
5.0ms interval 100.0ms ecn
Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues
0) <---- WOW. DROPS!!
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0
new_flows_len 0 old_flows_len 2
Apparently rather than applying the tc qdsic on the outbound path on the
LAN side ( eth0 ), they are applying it inbound on the the eth2 via
ifb_eth2.
Initially, I was pretty surprised to see so many drops on the inbound path,
but maybe this is actually normal?
I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I
would probably have expected the bottlenecks to form at the access points.
e.g. It's gigabit all the way until it reaches the air interface of the
access points. .... Or do I have a problem in my LAN network?
I wonder if I can log into the access points to look at them too?....
( BTW - to get to root on these devices you can SSH in as an "admin" users,
and then just "sudo su" )
*ifconfig*
root at USG-Pro-4:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0
inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0
TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
<---- queue len 0? Maybe this is a driver issue?
RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)
eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0
inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0
inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link
inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:154930 errors:0 dropped:0 overruns:0 frame:0
TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)
eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1
inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0
inet6 addr: [SNIP]:d474:3d71/128 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0
TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)
eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:782123 errors:0 dropped:0 overruns:0 frame:0
TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)
eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2695 errors:0 dropped:0 overruns:0 frame:0
TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)
ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27
inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0
TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32 <-----
queue len 32? Curious
RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB)
*System info*
This has a prehistoric kernel, I guess because they have some stuff that
taints the kernel
root at USG-Pro-4:~# uname -a
Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64
GNU/Linux
root at USG-Pro-4:~# cat /var/log/dmesg | grep taint
ubnt_platform: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
I also notice this module, but I'm not sure it is in use.
/lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko
root at USG-Pro-4:~# cat /proc/cpuinfo
system type : UBNT_E220
machine : Unknown
processor : 0
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 0
VCED exceptions : not available
VCEI exceptions : not available
processor : 1
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 1
VCED exceptions : not available
VCEI exceptions : not available
root at USG-Pro-4:~# ethtool -i eth2
driver: octeon-ethernet
version: 2.0
firmware-version:
bus-info: Builtin
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root at USG-Pro-4:~# ethtool -S eth2
no stats available
( Oh great! Thanks guys! )
root at USG-Pro-4:~# netstat -ia
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR
Flg
eth0 1500 0 11340913 0 0 0 21612063 0 0
0 BMRU
eth1 1500 0 154902 0 0 0 233236 0 0
0 BMRU
eth2 1500 0 60898610 0 0 0 10544414 0 0
0 BMRU
eth3 1500 0 0 0 0 0 0 0 0
0 BM
eth0.20 1500 0 781992 0 0 0 480214 0 0
0 BMRU
eth0.40 1500 0 2695 0 0 0 194260 0 0
0 BMRU
ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0
0 BORU <---- RX drops?
imq0 16000 0 0 0 0 0 0 0 0
0 ORU
lo 65536 0 9255 0 0 0 9255 0 0
0 LRU
loop0 1500 0 0 0 0 0 0 0 0
0 BM
loop1 1500 0 0 0 0 0 0 0 0
0 BM
loop2 1500 0 0 0 0 0 0 0 0
0 BM
loop3 1500 0 0 0 0 0 0 0 0
0 BM
npi0 1500 0 0 0 0 0 0 0 0
0 BM
npi1 1500 0 0 0 0 0 0 0 0
0 BM
npi2 1500 0 0 0 0 0 0 0 0
0 BM
npi3 1500 0 0 0 0 0 0 0 0
0 BM
root at USG-Pro-4:/opt/vyatta/etc# cat version
Version: v4.4.57.5578372.230112.0824
--
Regards,
Dave Seddon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20240102/770eb5f5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 124401 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20240102/770eb5f5/attachment-0001.png>
More information about the Cake
mailing list