G'day, Happy new year y'all I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative. Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos ) [image: image.png] https://finance.yahoo.com/quote/ui/ ( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. ) *Network diagram* Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points *"Smart Queue" Configuration* Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth. [image: image.png] *"Smart Queue" Implementation* Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel. And cake isn't available :( root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms Unknown qdisc "cake", hence option "bandwidth" is unparsable *Outbound eth2* root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS? backlog 0b 0p requeues 0 qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 new_flows_len 1 old_flows_len 1 qdisc ingress ffff: parent ffff:fff1 ---------------- Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 - target 5.0ms is the default ( https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if they did much testing on this hardware? - ( I actually have a spare "wan" ethernet port, so I guess I could hook up a PC and perform a flent test. ) - It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be - I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN? *Inbound eth2 via ifb* root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2 qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) <---- OVERLIMITS? backlog 0b 0p requeues 0 qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!! backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0 new_flows_len 0 old_flows_len 2 Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal? I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network? I wonder if I can log into the access points to look at them too?.... ( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" ) *ifconfig* root@USG-Pro-4:~# ifconfig -a eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0 inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0 TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue? RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB) eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0 inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0 inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:154930 errors:0 dropped:0 overruns:0 frame:0 TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB) eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1 inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0 inet6 addr: [SNIP]:d474:3d71/128 Scope:Global inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0 TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB) eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0 inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:782123 errors:0 dropped:0 overruns:0 frame:0 TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB) eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0 inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2695 errors:0 dropped:0 overruns:0 frame:0 TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB) ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27 inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0 TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 <----- queue len 32? Curious RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB) *System info* This has a prehistoric kernel, I guess because they have some stuff that taints the kernel root@USG-Pro-4:~# uname -a Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux root@USG-Pro-4:~# cat /var/log/dmesg | grep taint ubnt_platform: module license 'Proprietary' taints kernel. Disabling lock debugging due to kernel taint I also notice this module, but I'm not sure it is in use. /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko root@USG-Pro-4:~# cat /proc/cpuinfo system type : UBNT_E220 machine : Unknown processor : 0 cpu model : Cavium Octeon II V0.1 BogoMIPS : 2000.00 wait instruction : yes microsecond timers : yes tlb_entries : 128 extra interrupt vector : yes hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] isa : mips1 mips2 mips3 mips4 mips5 mips64r2 ASEs implemented : shadow register sets : 1 kscratch registers : 3 core : 0 VCED exceptions : not available VCEI exceptions : not available processor : 1 cpu model : Cavium Octeon II V0.1 BogoMIPS : 2000.00 wait instruction : yes microsecond timers : yes tlb_entries : 128 extra interrupt vector : yes hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] isa : mips1 mips2 mips3 mips4 mips5 mips64r2 ASEs implemented : shadow register sets : 1 kscratch registers : 3 core : 1 VCED exceptions : not available VCEI exceptions : not available root@USG-Pro-4:~# ethtool -i eth2 driver: octeon-ethernet version: 2.0 firmware-version: bus-info: Builtin supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no root@USG-Pro-4:~# ethtool -S eth2 no stats available ( Oh great! Thanks guys! ) root@USG-Pro-4:~# netstat -ia Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU eth3 1500 0 0 0 0 0 0 0 0 0 BM eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops? imq0 16000 0 0 0 0 0 0 0 0 0 ORU lo 65536 0 9255 0 0 0 9255 0 0 0 LRU loop0 1500 0 0 0 0 0 0 0 0 0 BM loop1 1500 0 0 0 0 0 0 0 0 0 BM loop2 1500 0 0 0 0 0 0 0 0 0 BM loop3 1500 0 0 0 0 0 0 0 0 0 BM npi0 1500 0 0 0 0 0 0 0 0 0 BM npi1 1500 0 0 0 0 0 0 0 0 0 BM npi2 1500 0 0 0 0 0 0 0 0 0 BM npi3 1500 0 0 0 0 0 0 0 0 0 BM root@USG-Pro-4:/opt/vyatta/etc# cat version Version: v4.4.57.5578372.230112.0824 -- Regards, Dave Seddon