<div dir="ltr"><div id="gmail-:2sw" class="gmail-Ar gmail-Au gmail-Ao" style="display:block"><div id="gmail-:2ss" class="gmail-Am gmail-aiL gmail-Al editable gmail-LW-avf gmail-tS-tW gmail-tS-tY" aria-label="Message Body" role="textbox" aria-multiline="true" style="direction:ltr;min-height:376px" tabindex="1" aria-controls=":2vj" aria-expanded="false"><div>G'day,</div><div><br></div><div>Happy new year y'all</div><div><br></div><div>I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative.</div><div><br></div><div>Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos )<br></div><div><br></div><div><img src="cid:ii_lqwoedy50" alt="image.png" width="558" height="359"><br><a href="https://finance.yahoo.com/quote/ui/">https://finance.yahoo.com/quote/ui/</a><br></div><div><br></div><div>( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. )</div><div><br></div><div><div><b>Network diagram</b><br></div><div><br></div><div>Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points</div></div><div><br></div><div><b>"Smart Queue" Configuration</b></div><div>Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth.<br></div><div><br></div><div><img src="cid:ii_lqv47sq90" alt="image.png" width="434" height="558"></div><div><br></div><br><div><b>"Smart Queue" Implementation</b><br></div><div><br></div><div>Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel.</div><div><br></div><div>And cake isn't available :(</div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms<br>Unknown qdisc "cake", hence option "bandwidth" is unparsable</span></div><div><br></div><div><b>Outbound eth2</b><br></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2<br>qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17<br> Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS?<br> backlog 0b 0p requeues 0 <br>qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn <br> Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS<br> backlog 0b 0p requeues 0 <br> maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0<br> new_flows_len 1 old_flows_len 1<br>qdisc ingress ffff: parent ffff:fff1 ---------------- <br> Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) <br> backlog 0b 0p requeues 0 <br></span></div><div><ul><li>target 5.0ms is the default ( <a href="https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html">https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html</a> ). I wonder if they did much testing on this hardware?<br></li><ul><li>( I actually have a spare "wan" ethernet port, so I
guess I could hook up a PC and perform a flent test. )</li></ul><li>It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be<br></li><li>I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN?<br></li></ul></div><div><br></div><div><b>Inbound eth2 via ifb</b><br></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2<br>qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17<br> Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) </span><span style="font-family:monospace"> <---- OVERLIMITS?</span><br><span style="font-family:monospace"> backlog 0b 0p requeues 0 <br>qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn <br> Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!!<br> backlog 0b 0p requeues 0 <br> maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0<br> new_flows_len 0 old_flows_len 2</span><br></div><div><br></div><div>Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2.<br></div><div><br></div><div>Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal?</div><div><br></div><div>I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network?</div><div><br></div><div>I wonder if I can log into the access points to look at them too?....<br></div><div><br></div><div>( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" )</div><div><br></div><div><b>ifconfig</b><br></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# ifconfig -a<br>eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f <br> inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0<br> inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global<br> inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue? <br> RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)<br><br>eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0 <br> inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0<br> inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link<br> inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:154930 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)<br><br>eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1 <br> inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0<br> inet6 addr: [SNIP]:d474:3d71/128 Scope:Global<br> inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)<br><br>eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2 <br> BROADCAST MULTICAST MTU:1500 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)<br><br>eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f <br> inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0<br> inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global<br> inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:782123 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)<br><br>eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f <br> inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0<br> inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global<br> inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:2695 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)<br><br>ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27 <br> inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link<br> UP BROADCAST RUNNING NOARP MTU:1500 Metric:1<br> RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0<br> TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:32 <----- queue len 32? Curious<br> RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB)<br></span></div><div><span style="font-family:monospace"><br></span></div><div><br></div><div><b>System info</b><br></div><div><br></div><div>This has a prehistoric kernel, I guess because they have some stuff that taints the kernel<br></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">root@USG-Pro-4:~# uname -a<br>Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">root@USG-Pro-4:~# cat /var/log/dmesg | grep taint<br>ubnt_platform: module license 'Proprietary' taints kernel.<br>Disabling lock debugging due to kernel taint</span><br></div><div><br></div><div>I also notice this module, but I'm not sure it is in use.<br></div><div><span style="font-family:monospace">/lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko</span></div><div></div><div><br></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# cat /proc/cpuinfo <br>system type : UBNT_E220<br>machine : Unknown<br>processor : 0<br>cpu model : Cavium Octeon II V0.1<br>BogoMIPS : 2000.00<br>wait instruction : yes<br>microsecond timers : yes<br>tlb_entries : 128<br>extra interrupt vector : yes<br>hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]<br>isa : mips1 mips2 mips3 mips4 mips5 mips64r2<br>ASEs implemented :<br>shadow register sets : 1<br>kscratch registers : 3<br>core : 0<br>VCED exceptions : not available<br>VCEI exceptions : not available<br><br>processor : 1<br>cpu model : Cavium Octeon II V0.1<br>BogoMIPS : 2000.00<br>wait instruction : yes<br>microsecond timers : yes<br>tlb_entries : 128<br>extra interrupt vector : yes<br>hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]<br>isa : mips1 mips2 mips3 mips4 mips5 mips64r2<br>ASEs implemented :<br>shadow register sets : 1<br>kscratch registers : 3<br>core : 1<br>VCED exceptions : not available<br>VCEI exceptions : not available</span></div><div><br></div><div><br></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# ethtool -i eth2<br>driver: octeon-ethernet<br>version: 2.0<br>firmware-version: <br>bus-info: Builtin<br>supports-statistics: no<br>supports-test: no<br>supports-eeprom-access: no<br>supports-register-dump: no<br>supports-priv-flags: no</span></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# ethtool -S eth2<br>no stats available</span></div><div><br></div><div>( Oh great! Thanks guys! )</div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:~# netstat -ia<br>Kernel Interface table<br>Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg<br>eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU<br>eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU<br>eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU<br>eth3 1500 0 0 0 0 0 0 0 0 0 BM<br>eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU<br>eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU<br>ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops?<br>imq0 16000 0 0 0 0 0 0 0 0 0 ORU<br>lo 65536 0 9255 0 0 0 9255 0 0 0 LRU<br>loop0 1500 0 0 0 0 0 0 0 0 0 BM<br>loop1 1500 0 0 0 0 0 0 0 0 0 BM<br>loop2 1500 0 0 0 0 0 0 0 0 0 BM<br>loop3 1500 0 0 0 0 0 0 0 0 0 BM<br>npi0 1500 0 0 0 0 0 0 0 0 0 BM<br>npi1 1500 0 0 0 0 0 0 0 0 0 BM<br>npi2 1500 0 0 0 0 0 0 0 0 0 BM<br>npi3 1500 0 0 0 0 0 0 0 0 0 BM</span></div><div><br></div><div><span style="font-family:monospace">root@USG-Pro-4:/opt/vyatta/etc# cat version <br>Version: v4.4.57.5578372.230112.0824</span><br></div><div><br></div></div></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Regards,<br></div>Dave Seddon<br></div></div></div></div></div></div></div>