[Cake] Ubiquity (Unifi ) Smart Queues
Sebastian Moeller
moeller0 at gmx.de
Tue Jan 2 16:24:18 EST 2024
Hi Dave,
> On Jan 2, 2024, at 22:15, dave seddon <dave.seddon.ca at gmail.com> wrote:
>
> Thanks Sebastian!
>
> Now I see the rates!!
>
> I actually reduced the rates to ensure this device is the bottleneck 80/10 Mb/s
>
> <image.png>
>
> root at USG-Pro-4:~# tc -d class show dev eth2
> class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 0
> class fq_codel 100:12c parent 100:
> class fq_codel 100:213 parent 100:
> class fq_codel 100:22e parent 100:
>
> root at USG-Pro-4:~# tc -d class show dev ifb_eth2
> class htb 1:10 root leaf 100: prio 0 quantum 200000 rate 76000Kbit ceil 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b level 0
> class fq_codel 100:2c8 parent 100:
> class fq_codel 100:3df parent 100:
>
[SM] They apparently have a rather loose translation from 80/10 GUI rates to shaper settings (derated by 5%), and my personal hobby horse seem to ignore the overhead question more or less entirely**...
bad Ubiquity... if you want to be a "good boy" email me ;)
**) If that 5% derating is supposed to handle overhead, oh my... per packet overhead stays constant with packet number while payload size scales with packet size so overhead inversely scales with packet size and hence static rerating for per-packet overhead is approximate at best to stay polite.
Also what is up with the burst values? why do they differ by 32 octets?
Sorry for bothering you with this, these are mostly questions for Ubiquity and the are unlikely to monitor this list ;)
Regards
Sebastian
> On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller <moeller0 at gmx.de> wrote:
> Hi Dave.
>
> just a few comments from the peanut gallery...
>
>
> > On Jan 2, 2024, at 19:59, dave seddon via Cake <cake at lists.bufferbloat.net> wrote:
> >
> > G'day,
> >
> > Happy new year y'all
>
> +1
>
> >
> > I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative.
> >
> > Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos )
> >
> > <image.png>
> > https://finance.yahoo.com/quote/ui/
> >
> > ( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. )
> >
> > Network diagram
> >
> > Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points
> >
> > "Smart Queue" Configuration
> > Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth.
> >
> >
> >
> >
> > "Smart Queue" Implementation
> >
> > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel.
> >
> > And cake isn't available :(
> >
> > root at USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
> > Unknown qdisc "cake", hence option "bandwidth" is unparsable
> >
> > Outbound eth2
> >
> > root at USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
> > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS?
> > backlog 0b 0p requeues 0
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
> > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS
> > backlog 0b 0p requeues 0
> > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
> > new_flows_len 1 old_flows_len 1
> > qdisc ingress ffff: parent ffff:fff1 ----------------
> > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0)
> > backlog 0b 0p requeues 0
> > • target 5.0ms is the default ( https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if they did much testing on this hardware?
>
> [SM] Not sure whether playing with target in isolation would be much use, in codel theory target should be 5-10% of interval ans interval should be in the order of magnitude of to be handled RTTs (the default is 100ms wich works reasonably well even across the Atlantic, but you probably knew all that).
>
> > • ( I actually have a spare "wan" ethernet port, so I guess I could hook up a PC and perform a flent test. )
> > • It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be
>
> [SM] Likely because HTB does not reveal this when asked with the `-s` option, try `-q` instead and not as qdisc but as class (so maybe `tc -d class show dev eth2`).
>
> > • I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN?
>
> I think this text about TBF explains this reasonably well (HTB is essentially a hierarchical version of TBF):
>
> see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html
>
> 9.2.2. Token Bucket Filter
>
> The Token Bucket Filter (TBF) is a simple qdisc that only passes packets arriving at a rate which is not exceeding some administratively set rate, but with the possibility to allow short bursts in excess of this rate.
>
> TBF is very precise, network- and processor friendly. It should be your first choice if you simply want to slow an interface down!
>
> The TBF implementation consists of a buffer (bucket), constantly filled by some virtual pieces of information called tokens, at a specific rate (token rate). The most important parameter of the bucket is its size, that is the number of tokens it can store.
>
> Each arriving token collects one incoming data packet from the data queue and is then deleted from the bucket. Associating this algorithm with the two flows -- token and data, gives us three possible scenarios:
>
> • The data arrives in TBF at a rate that's equal to the rate of incoming tokens. In this case each incoming packet has its matching token and passes the queue without delay.
>
> • The data arrives in TBF at a rate that's smaller than the token rate. Only a part of the tokens are deleted at output of each data packet that's sent out the queue, so the tokens accumulate, up to the bucket size. The unused tokens can then be used to send data a a speed that's exceeding the standard token rate, in case short data bursts occur.
>
> • The data arrives in TBF at a rate bigger than the token rate. This means that the bucket will soon be devoid of tokens, which causes the TBF to throttle itself for a while. This is called an 'overlimit situation'. If packets keep coming in, packets will start to get dropped.
>
> The last scenario is very important, because it allows to administratively shape the bandwidth available to data that's passing the filter.
>
> The accumulation of tokens allows a short burst of overlimit data to be still passed without loss, but any lasting overload will cause packets to be constantly delayed, and then dropped.
>
> Please note that in the actual implementation, tokens correspond to bytes, not packets.
>
>
> >
> > Inbound eth2 via ifb
> >
> > root at USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2
> > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
> > Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) <---- OVERLIMITS?
> > backlog 0b 0p requeues 0
> > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
> > Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!!
> > backlog 0b 0p requeues 0
> > maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0
> > new_flows_len 0 old_flows_len 2
> >
> > Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2.
>
> [SM] Same approach that sqm-scripts takes, if you attach the ingress shaper to the LAN port egress all internet traffic not traversing that interface will not be shaped, e.g. traffic to/from the router itself or WiFi traffic. If you are sure that such by-pass traffic does not exist, putting the ingress shaper on lan-egress can save the cost of the ifb indirection, but for full WiFi routers that is generally not true.
>
>
> > Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal?
>
> [SM] Depends on your traffic and whether ECN is used or not. In your case it appears ECN is not used and then DROPS are the only way fq_codel can tell a flow to step on the brakes....
>
> >
> > I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network?
>
> [SM] The idea is to create an artificial bittleneck (using HTB) so the most relevant queue is under AQM control...
>
> >
> > I wonder if I can log into the access points to look at them too?....
> >
> > ( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" )
> >
> > ifconfig
> >
> > root at USG-Pro-4:~# ifconfig -a
> > eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue?
> > RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)
> >
> > eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0
> > inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0
> > inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link
> > inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:154930 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)
> >
> > eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1
> > inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0
> > inet6 addr: [SNIP]:d474:3d71/128 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)
> >
> > eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2
> > BROADCAST MULTICAST MTU:1500 Metric:1
> > RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
> >
> > eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:782123 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)
> >
> > eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
> > inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0
> > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
> > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2695 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)
> >
> > ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27
> > inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link
> > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
> > RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0
> > TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:32 <----- queue len 32? Curious
> > RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB)
> >
> >
> > System info
> >
> > This has a prehistoric kernel, I guess because they have some stuff that taints the kernel
> >
> > root at USG-Pro-4:~# uname -a
> > Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux
>
> [SM] I remember the time we felt great about using a series 3 kernel instead of old series 2 gunk, but upstream is at series 6 by now (but it also started to increase major numbers more aggressively after series 2)
>
> >
> > root at USG-Pro-4:~# cat /var/log/dmesg | grep taint
> > ubnt_platform: module license 'Proprietary' taints kernel.
> > Disabling lock debugging due to kernel taint
> >
> > I also notice this module, but I'm not sure it is in use.
> > /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko
> >
> >
> > root at USG-Pro-4:~# cat /proc/cpuinfo
> > system type : UBNT_E220
> > machine : Unknown
> > processor : 0
> > cpu model : Cavium Octeon II V0.1
> > BogoMIPS : 2000.00
> > wait instruction : yes
> > microsecond timers : yes
> > tlb_entries : 128
> > extra interrupt vector : yes
> > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
> > isa : mips1 mips2 mips3 mips4 mips5 mips64r2
> > ASEs implemented :
> > shadow register sets : 1
> > kscratch registers : 3
> > core : 0
> > VCED exceptions : not available
> > VCEI exceptions : not available
> >
> > processor : 1
> > cpu model : Cavium Octeon II V0.1
> > BogoMIPS : 2000.00
> > wait instruction : yes
> > microsecond timers : yes
> > tlb_entries : 128
> > extra interrupt vector : yes
> > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
> > isa : mips1 mips2 mips3 mips4 mips5 mips64r2
> > ASEs implemented :
> > shadow register sets : 1
> > kscratch registers : 3
> > core : 1
> > VCED exceptions : not available
> > VCEI exceptions : not available
> >
> >
> >
> > root at USG-Pro-4:~# ethtool -i eth2
> > driver: octeon-ethernet
> > version: 2.0
> > firmware-version:
> > bus-info: Builtin
> > supports-statistics: no
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: no
> > supports-priv-flags: no
> >
> > root at USG-Pro-4:~# ethtool -S eth2
> > no stats available
> >
> > ( Oh great! Thanks guys! )
> >
> > root at USG-Pro-4:~# netstat -ia
> > Kernel Interface table
> > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
> > eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU
> > eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU
> > eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU
> > eth3 1500 0 0 0 0 0 0 0 0 0 BM
> > eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU
> > eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU
> > ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops?
> > imq0 16000 0 0 0 0 0 0 0 0 0 ORU
> > lo 65536 0 9255 0 0 0 9255 0 0 0 LRU
> > loop0 1500 0 0 0 0 0 0 0 0 0 BM
> > loop1 1500 0 0 0 0 0 0 0 0 0 BM
> > loop2 1500 0 0 0 0 0 0 0 0 0 BM
> > loop3 1500 0 0 0 0 0 0 0 0 0 BM
> > npi0 1500 0 0 0 0 0 0 0 0 0 BM
> > npi1 1500 0 0 0 0 0 0 0 0 0 BM
> > npi2 1500 0 0 0 0 0 0 0 0 0 BM
> > npi3 1500 0 0 0 0 0 0 0 0 0 BM
> >
> > root at USG-Pro-4:/opt/vyatta/etc# cat version
> > Version: v4.4.57.5578372.230112.0824
> >
> > --
> > Regards,
> > Dave Seddon
> > _______________________________________________
> > Cake mailing list
> > Cake at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
> Regards,
> Dave Seddon
> +1 415 857 5102
More information about the Cake
mailing list