G'day,

Happy new year y'all

I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative.

Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos )

https://finance.yahoo.com/quote/ui/

( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. )

Network diagram

Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points

"Smart Queue" Configuration

Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth.

"Smart Queue" Implementation

Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel.

And cake isn't available :(

root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
Unknown qdisc "cake", hence option "bandwidth" is unparsable

Outbound eth2

root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
new_flows_len 1 old_flows_len 1
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

target 5.0ms is the default ( https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if they did much testing on this hardware?

( I actually have a spare "wan" ethernet port, so I guess I could hook up a PC and perform a flent test. )

It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be
I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN?

Inbound eth2 via ifb

root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!!
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0
new_flows_len 0 old_flows_len 2

Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2.

Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal?

I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network?

I wonder if I can log into the access points to look at them too?....

( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" )

ifconfig

root@USG-Pro-4:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0
inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0
TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue?
RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)

eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0
inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0
inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link
inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:154930 errors:0 dropped:0 overruns:0 frame:0
TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)

eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1
inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0
inet6 addr: [SNIP]:d474:3d71/128 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0
TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)

eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:782123 errors:0 dropped:0 overruns:0 frame:0
TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)

eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2695 errors:0 dropped:0 overruns:0 frame:0
TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)

ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27
inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0
TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32 <----- queue len 32? Curious
RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB)

System info

This has a prehistoric kernel, I guess because they have some stuff that taints the kernel

root@USG-Pro-4:~# uname -a
Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux

root@USG-Pro-4:~# cat /var/log/dmesg | grep taint
ubnt_platform: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint

I also notice this module, but I'm not sure it is in use.

/lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko

root@USG-Pro-4:~# cat /proc/cpuinfo
system type : UBNT_E220
machine : Unknown
processor : 0
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 0
VCED exceptions : not available
VCEI exceptions : not available

processor : 1
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 1
VCED exceptions : not available
VCEI exceptions : not available

root@USG-Pro-4:~# ethtool -i eth2
driver: octeon-ethernet
version: 2.0
firmware-version:
bus-info: Builtin
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

root@USG-Pro-4:~# ethtool -S eth2
no stats available

( Oh great! Thanks guys! )

root@USG-Pro-4:~# netstat -ia
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU
eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU
eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU
eth3 1500 0 0 0 0 0 0 0 0 0 BM
eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU
eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU
ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops?
imq0 16000 0 0 0 0 0 0 0 0 0 ORU
lo 65536 0 9255 0 0 0 9255 0 0 0 LRU
loop0 1500 0 0 0 0 0 0 0 0 0 BM
loop1 1500 0 0 0 0 0 0 0 0 0 BM
loop2 1500 0 0 0 0 0 0 0 0 0 BM
loop3 1500 0 0 0 0 0 0 0 0 0 BM
npi0 1500 0 0 0 0 0 0 0 0 0 BM
npi1 1500 0 0 0 0 0 0 0 0 0 BM
npi2 1500 0 0 0 0 0 0 0 0 0 BM
npi3 1500 0 0 0 0 0 0 0 0 0 BM

root@USG-Pro-4:/opt/vyatta/etc# cat version
Version: v4.4.57.5578372.230112.0824

Regards,

Dave Seddon