* [Cake] Ubiquity (Unifi ) Smart Queues
@ 2024-01-02 18:59 dave seddon
2024-01-02 20:52 ` Sebastian Moeller
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: dave seddon @ 2024-01-02 18:59 UTC (permalink / raw)
To: Cake List
[-- Attachment #1.1: Type: text/plain, Size: 11021 bytes --]
G'day,
Happy new year y'all
I thought people might be interested to see what Ubiquity/Unifi is doing
with "Smart Queues" on their devices. The documentation on their website
is not very informative.
Hopefully, this is vaguely interesting because Ubiquity is widely deployed
and apparently they have a market cap of >$8 billion, so you would hope
they do a "good job" (... Seems like they might be a target customer for
libreqos )
[image: image.png]
https://finance.yahoo.com/quote/ui/
( I use Unifi because their wifi stuff seems ok, and all the
switching/routing/wifi is all integrated into the single gui control
system. Also honestly, I'm not sure I know how to do prefix delegation
stuff on Linux by hand. )
*Network diagram*
Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <--->
[Switches] <----> Access points
*"Smart Queue" Configuration*
Ubiquity doesn't have many knobs, you just enable "smart queues" and set
the bandwidth.
[image: image.png]
*"Smart Queue" Implementation*
Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT
cake, but fq_codel.
And cake isn't available :(
root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms
Unknown qdisc "cake", hence option "bandwidth" is unparsable
*Outbound eth2*
root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues
0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target
5.0ms interval 100.0ms ecn
Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues
0) <----- DROPS
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0
new_flows_len 1 old_flows_len 1
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
- target 5.0ms is the default (
https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I
wonder if they did much testing on this hardware?
- ( I actually have a spare "wan" ethernet port, so I guess I could hook
up a PC and perform a flent test. )
- It's unclear to me what the "htb" is doing, because I would have
expected the download/upload rates to be configured here, but they appear
not to be
- I'm not really sure what "overlimits" means or what that does, and
tried looking this up, but I guess the kernel source is likely the "best"
documentation for this. Maybe this means it's dropping? Or is it ECN?
*Inbound eth2 via ifb*
root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339
requeues 0) <---- OVERLIMITS?
backlog 0b 0p requeues 0
qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target
5.0ms interval 100.0ms ecn
Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues
0) <---- WOW. DROPS!!
backlog 0b 0p requeues 0
maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0
new_flows_len 0 old_flows_len 2
Apparently rather than applying the tc qdsic on the outbound path on the
LAN side ( eth0 ), they are applying it inbound on the the eth2 via
ifb_eth2.
Initially, I was pretty surprised to see so many drops on the inbound path,
but maybe this is actually normal?
I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I
would probably have expected the bottlenecks to form at the access points.
e.g. It's gigabit all the way until it reaches the air interface of the
access points. .... Or do I have a problem in my LAN network?
I wonder if I can log into the access points to look at them too?....
( BTW - to get to root on these devices you can SSH in as an "admin" users,
and then just "sudo su" )
*ifconfig*
root@USG-Pro-4:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0
inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0
TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
<---- queue len 0? Maybe this is a driver issue?
RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB)
eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0
inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0
inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link
inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:154930 errors:0 dropped:0 overruns:0 frame:0
TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB)
eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1
inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0
inet6 addr: [SNIP]:d474:3d71/128 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0
TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB)
eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:782123 errors:0 dropped:0 overruns:0 frame:0
TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB)
eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f
inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0
inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global
inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2695 errors:0 dropped:0 overruns:0 frame:0
TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB)
ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27
inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0
TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32 <-----
queue len 32? Curious
RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB)
*System info*
This has a prehistoric kernel, I guess because they have some stuff that
taints the kernel
root@USG-Pro-4:~# uname -a
Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64
GNU/Linux
root@USG-Pro-4:~# cat /var/log/dmesg | grep taint
ubnt_platform: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
I also notice this module, but I'm not sure it is in use.
/lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko
root@USG-Pro-4:~# cat /proc/cpuinfo
system type : UBNT_E220
machine : Unknown
processor : 0
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 0
VCED exceptions : not available
VCEI exceptions : not available
processor : 1
cpu model : Cavium Octeon II V0.1
BogoMIPS : 2000.00
wait instruction : yes
microsecond timers : yes
tlb_entries : 128
extra interrupt vector : yes
hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa : mips1 mips2 mips3 mips4 mips5 mips64r2
ASEs implemented :
shadow register sets : 1
kscratch registers : 3
core : 1
VCED exceptions : not available
VCEI exceptions : not available
root@USG-Pro-4:~# ethtool -i eth2
driver: octeon-ethernet
version: 2.0
firmware-version:
bus-info: Builtin
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root@USG-Pro-4:~# ethtool -S eth2
no stats available
( Oh great! Thanks guys! )
root@USG-Pro-4:~# netstat -ia
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR
Flg
eth0 1500 0 11340913 0 0 0 21612063 0 0
0 BMRU
eth1 1500 0 154902 0 0 0 233236 0 0
0 BMRU
eth2 1500 0 60898610 0 0 0 10544414 0 0
0 BMRU
eth3 1500 0 0 0 0 0 0 0 0
0 BM
eth0.20 1500 0 781992 0 0 0 480214 0 0
0 BMRU
eth0.40 1500 0 2695 0 0 0 194260 0 0
0 BMRU
ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0
0 BORU <---- RX drops?
imq0 16000 0 0 0 0 0 0 0 0
0 ORU
lo 65536 0 9255 0 0 0 9255 0 0
0 LRU
loop0 1500 0 0 0 0 0 0 0 0
0 BM
loop1 1500 0 0 0 0 0 0 0 0
0 BM
loop2 1500 0 0 0 0 0 0 0 0
0 BM
loop3 1500 0 0 0 0 0 0 0 0
0 BM
npi0 1500 0 0 0 0 0 0 0 0
0 BM
npi1 1500 0 0 0 0 0 0 0 0
0 BM
npi2 1500 0 0 0 0 0 0 0 0
0 BM
npi3 1500 0 0 0 0 0 0 0 0
0 BM
root@USG-Pro-4:/opt/vyatta/etc# cat version
Version: v4.4.57.5578372.230112.0824
--
Regards,
Dave Seddon
[-- Attachment #1.2: Type: text/html, Size: 15389 bytes --]
[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 124401 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-02 18:59 [Cake] Ubiquity (Unifi ) Smart Queues dave seddon @ 2024-01-02 20:52 ` Sebastian Moeller 2024-01-02 21:15 ` dave seddon 2024-01-02 21:57 ` Jonathan Morton 2024-01-03 13:44 ` Pete Heist 2 siblings, 1 reply; 16+ messages in thread From: Sebastian Moeller @ 2024-01-02 20:52 UTC (permalink / raw) To: dave seddon; +Cc: Cake List Hi Dave. just a few comments from the peanut gallery... > On Jan 2, 2024, at 19:59, dave seddon via Cake <cake@lists.bufferbloat.net> wrote: > > G'day, > > Happy new year y'all +1 > > I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative. > > Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos ) > > <image.png> > https://finance.yahoo.com/quote/ui/ > > ( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. ) > > Network diagram > > Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points > > "Smart Queue" Configuration > Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth. > > > > > "Smart Queue" Implementation > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel. > > And cake isn't available :( > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > Outbound eth2 > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS? > backlog 0b 0p requeues 0 > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > new_flows_len 1 old_flows_len 1 > qdisc ingress ffff: parent ffff:fff1 ---------------- > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > • target 5.0ms is the default ( https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if they did much testing on this hardware? [SM] Not sure whether playing with target in isolation would be much use, in codel theory target should be 5-10% of interval ans interval should be in the order of magnitude of to be handled RTTs (the default is 100ms wich works reasonably well even across the Atlantic, but you probably knew all that). > • ( I actually have a spare "wan" ethernet port, so I guess I could hook up a PC and perform a flent test. ) > • It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be [SM] Likely because HTB does not reveal this when asked with the `-s` option, try `-q` instead and not as qdisc but as class (so maybe `tc -d class show dev eth2`). > • I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN? I think this text about TBF explains this reasonably well (HTB is essentially a hierarchical version of TBF): see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html 9.2.2. Token Bucket Filter The Token Bucket Filter (TBF) is a simple qdisc that only passes packets arriving at a rate which is not exceeding some administratively set rate, but with the possibility to allow short bursts in excess of this rate. TBF is very precise, network- and processor friendly. It should be your first choice if you simply want to slow an interface down! The TBF implementation consists of a buffer (bucket), constantly filled by some virtual pieces of information called tokens, at a specific rate (token rate). The most important parameter of the bucket is its size, that is the number of tokens it can store. Each arriving token collects one incoming data packet from the data queue and is then deleted from the bucket. Associating this algorithm with the two flows -- token and data, gives us three possible scenarios: • The data arrives in TBF at a rate that's equal to the rate of incoming tokens. In this case each incoming packet has its matching token and passes the queue without delay. • The data arrives in TBF at a rate that's smaller than the token rate. Only a part of the tokens are deleted at output of each data packet that's sent out the queue, so the tokens accumulate, up to the bucket size. The unused tokens can then be used to send data a a speed that's exceeding the standard token rate, in case short data bursts occur. • The data arrives in TBF at a rate bigger than the token rate. This means that the bucket will soon be devoid of tokens, which causes the TBF to throttle itself for a while. This is called an 'overlimit situation'. If packets keep coming in, packets will start to get dropped. The last scenario is very important, because it allows to administratively shape the bandwidth available to data that's passing the filter. The accumulation of tokens allows a short burst of overlimit data to be still passed without loss, but any lasting overload will cause packets to be constantly delayed, and then dropped. Please note that in the actual implementation, tokens correspond to bytes, not packets. > > Inbound eth2 via ifb > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2 > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) <---- OVERLIMITS? > backlog 0b 0p requeues 0 > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn > Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!! > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0 > new_flows_len 0 old_flows_len 2 > > Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. [SM] Same approach that sqm-scripts takes, if you attach the ingress shaper to the LAN port egress all internet traffic not traversing that interface will not be shaped, e.g. traffic to/from the router itself or WiFi traffic. If you are sure that such by-pass traffic does not exist, putting the ingress shaper on lan-egress can save the cost of the ifb indirection, but for full WiFi routers that is generally not true. > Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal? [SM] Depends on your traffic and whether ECN is used or not. In your case it appears ECN is not used and then DROPS are the only way fq_codel can tell a flow to step on the brakes.... > > I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network? [SM] The idea is to create an artificial bittleneck (using HTB) so the most relevant queue is under AQM control... > > I wonder if I can log into the access points to look at them too?.... > > ( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" ) > > ifconfig > > root@USG-Pro-4:~# ifconfig -a > eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0 > inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0 > TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue? > RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB) > > eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0 > inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0 > inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link > inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:154930 errors:0 dropped:0 overruns:0 frame:0 > TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB) > > eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1 > inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0 > inet6 addr: [SNIP]:d474:3d71/128 Scope:Global > inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0 > TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB) > > eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2 > BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0 > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:782123 errors:0 dropped:0 overruns:0 frame:0 > TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB) > > eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0 > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:2695 errors:0 dropped:0 overruns:0 frame:0 > TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB) > > ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27 > inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 > RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0 > TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:32 <----- queue len 32? Curious > RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB) > > > System info > > This has a prehistoric kernel, I guess because they have some stuff that taints the kernel > > root@USG-Pro-4:~# uname -a > Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux [SM] I remember the time we felt great about using a series 3 kernel instead of old series 2 gunk, but upstream is at series 6 by now (but it also started to increase major numbers more aggressively after series 2) > > root@USG-Pro-4:~# cat /var/log/dmesg | grep taint > ubnt_platform: module license 'Proprietary' taints kernel. > Disabling lock debugging due to kernel taint > > I also notice this module, but I'm not sure it is in use. > /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko > > > root@USG-Pro-4:~# cat /proc/cpuinfo > system type : UBNT_E220 > machine : Unknown > processor : 0 > cpu model : Cavium Octeon II V0.1 > BogoMIPS : 2000.00 > wait instruction : yes > microsecond timers : yes > tlb_entries : 128 > extra interrupt vector : yes > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > ASEs implemented : > shadow register sets : 1 > kscratch registers : 3 > core : 0 > VCED exceptions : not available > VCEI exceptions : not available > > processor : 1 > cpu model : Cavium Octeon II V0.1 > BogoMIPS : 2000.00 > wait instruction : yes > microsecond timers : yes > tlb_entries : 128 > extra interrupt vector : yes > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > ASEs implemented : > shadow register sets : 1 > kscratch registers : 3 > core : 1 > VCED exceptions : not available > VCEI exceptions : not available > > > > root@USG-Pro-4:~# ethtool -i eth2 > driver: octeon-ethernet > version: 2.0 > firmware-version: > bus-info: Builtin > supports-statistics: no > supports-test: no > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: no > > root@USG-Pro-4:~# ethtool -S eth2 > no stats available > > ( Oh great! Thanks guys! ) > > root@USG-Pro-4:~# netstat -ia > Kernel Interface table > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU > eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU > eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU > eth3 1500 0 0 0 0 0 0 0 0 0 BM > eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU > eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU > ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops? > imq0 16000 0 0 0 0 0 0 0 0 0 ORU > lo 65536 0 9255 0 0 0 9255 0 0 0 LRU > loop0 1500 0 0 0 0 0 0 0 0 0 BM > loop1 1500 0 0 0 0 0 0 0 0 0 BM > loop2 1500 0 0 0 0 0 0 0 0 0 BM > loop3 1500 0 0 0 0 0 0 0 0 0 BM > npi0 1500 0 0 0 0 0 0 0 0 0 BM > npi1 1500 0 0 0 0 0 0 0 0 0 BM > npi2 1500 0 0 0 0 0 0 0 0 0 BM > npi3 1500 0 0 0 0 0 0 0 0 0 BM > > root@USG-Pro-4:/opt/vyatta/etc# cat version > Version: v4.4.57.5578372.230112.0824 > > -- > Regards, > Dave Seddon > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-02 20:52 ` Sebastian Moeller @ 2024-01-02 21:15 ` dave seddon 2024-01-02 21:24 ` Sebastian Moeller 0 siblings, 1 reply; 16+ messages in thread From: dave seddon @ 2024-01-02 21:15 UTC (permalink / raw) To: Sebastian Moeller; +Cc: Cake List [-- Attachment #1.1: Type: text/plain, Size: 17462 bytes --] Thanks Sebastian! Now I see the rates!! I actually reduced the rates to ensure this device is the bottleneck 80/10 Mb/s [image: image.png] root@USG-Pro-4:~# tc -d class show dev eth2 class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 0 class fq_codel 100:12c parent 100: class fq_codel 100:213 parent 100: class fq_codel 100:22e parent 100: root@USG-Pro-4:~# tc -d class show dev ifb_eth2 class htb 1:10 root leaf 100: prio 0 quantum 200000 rate 76000Kbit ceil 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b level 0 class fq_codel 100:2c8 parent 100: class fq_codel 100:3df parent 100: On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller <moeller0@gmx.de> wrote: > Hi Dave. > > just a few comments from the peanut gallery... > > > > On Jan 2, 2024, at 19:59, dave seddon via Cake < > cake@lists.bufferbloat.net> wrote: > > > > G'day, > > > > Happy new year y'all > > +1 > > > > > I thought people might be interested to see what Ubiquity/Unifi is doing > with "Smart Queues" on their devices. The documentation on their website > is not very informative. > > > > Hopefully, this is vaguely interesting because Ubiquity is widely > deployed and apparently they have a market cap of >$8 billion, so you would > hope they do a "good job" (... Seems like they might be a target customer > for libreqos ) > > > > <image.png> > > https://finance.yahoo.com/quote/ui/ > > > > ( I use Unifi because their wifi stuff seems ok, and all the > switching/routing/wifi is all integrated into the single gui control > system. Also honestly, I'm not sure I know how to do prefix delegation > stuff on Linux by hand. ) > > > > Network diagram > > > > Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> > [Switches] <----> Access points > > > > "Smart Queue" Configuration > > Ubiquity doesn't have many knobs, you just enable "smart queues" and set > the bandwidth. > > > > > > > > > > "Smart Queue" Implementation > > > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT > cake, but fq_codel. > > > > And cake isn't available :( > > > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > > Outbound eth2 > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver > 3.17 > > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 > requeues 0) <---- OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 > target 5.0ms interval 100.0ms ecn > > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues > 0) <----- DROPS > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > > new_flows_len 1 old_flows_len 1 > > qdisc ingress ffff: parent ffff:fff1 ---------------- > > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues > 0) > > backlog 0b 0p requeues 0 > > • target 5.0ms is the default ( > https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder > if they did much testing on this hardware? > > [SM] Not sure whether playing with target in isolation would be much use, > in codel theory target should be 5-10% of interval ans interval should be > in the order of magnitude of to be handled RTTs (the default is 100ms wich > works reasonably well even across the Atlantic, but you probably knew all > that). > > > • ( I actually have a spare "wan" ethernet port, so I > guess I could hook up a PC and perform a flent test. ) > > • It's unclear to me what the "htb" is doing, because I would have > expected the download/upload rates to be configured here, but they appear > not to be > > [SM] Likely because HTB does not reveal this when asked with the `-s` > option, try `-q` instead and not as qdisc but as class (so maybe `tc -d > class show dev eth2`). > > > • I'm not really sure what "overlimits" means or what that does, > and tried looking this up, but I guess the kernel source is likely the > "best" documentation for this. Maybe this means it's dropping? Or is it > ECN? > > I think this text about TBF explains this reasonably well (HTB is > essentially a hierarchical version of TBF): > > see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html > > 9.2.2. Token Bucket Filter > > The Token Bucket Filter (TBF) is a simple qdisc that only passes packets > arriving at a rate which is not exceeding some administratively set rate, > but with the possibility to allow short bursts in excess of this rate. > > TBF is very precise, network- and processor friendly. It should be your > first choice if you simply want to slow an interface down! > > The TBF implementation consists of a buffer (bucket), constantly filled by > some virtual pieces of information called tokens, at a specific rate (token > rate). The most important parameter of the bucket is its size, that is the > number of tokens it can store. > > Each arriving token collects one incoming data packet from the data queue > and is then deleted from the bucket. Associating this algorithm with the > two flows -- token and data, gives us three possible scenarios: > > • The data arrives in TBF at a rate that's equal to the rate of > incoming tokens. In this case each incoming packet has its matching token > and passes the queue without delay. > > • The data arrives in TBF at a rate that's smaller than the token > rate. Only a part of the tokens are deleted at output of each data packet > that's sent out the queue, so the tokens accumulate, up to the bucket size. > The unused tokens can then be used to send data a a speed that's exceeding > the standard token rate, in case short data bursts occur. > > • The data arrives in TBF at a rate bigger than the token rate. > This means that the bucket will soon be devoid of tokens, which causes the > TBF to throttle itself for a while. This is called an 'overlimit > situation'. If packets keep coming in, packets will start to get dropped. > > The last scenario is very important, because it allows to administratively > shape the bandwidth available to data that's passing the filter. > > The accumulation of tokens allows a short burst of overlimit data to be > still passed without loss, but any lasting overload will cause packets to > be constantly delayed, and then dropped. > > Please note that in the actual implementation, tokens correspond to bytes, > not packets. > > > > > > Inbound eth2 via ifb > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver > 3.17 > > Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 > requeues 0) <---- OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 > target 5.0ms interval 100.0ms ecn > > Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 > requeues 0) <---- WOW. DROPS!! > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0 > > new_flows_len 0 old_flows_len 2 > > > > Apparently rather than applying the tc qdsic on the outbound path on the > LAN side ( eth0 ), they are applying it inbound on the the eth2 via > ifb_eth2. > > [SM] Same approach that sqm-scripts takes, if you attach the ingress > shaper to the LAN port egress all internet traffic not traversing that > interface will not be shaped, e.g. traffic to/from the router itself or > WiFi traffic. If you are sure that such by-pass traffic does not exist, > putting the ingress shaper on lan-egress can save the cost of the ifb > indirection, but for full WiFi routers that is generally not true. > > > > Initially, I was pretty surprised to see so many drops on the inbound > path, but maybe this is actually normal? > > [SM] Depends on your traffic and whether ECN is used or not. In your case > it appears ECN is not used and then DROPS are the only way fq_codel can > tell a flow to step on the brakes.... > > > > > I could imagine the upstream CDNs pushing pretty hard with low RTTs, but > I would probably have expected the bottlenecks to form at the access > points. e.g. It's gigabit all the way until it reaches the air interface of > the access points. .... Or do I have a problem in my LAN network? > > [SM] The idea is to create an artificial bittleneck (using HTB) so the > most relevant queue is under AQM control... > > > > > I wonder if I can log into the access points to look at them too?.... > > > > ( BTW - to get to root on these devices you can SSH in as an "admin" > users, and then just "sudo su" ) > > > > ifconfig > > > > root@USG-Pro-4:~# ifconfig -a > > eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > <---- queue len 0? Maybe this is a driver issue? > > RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB) > > > > eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0 > > inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0 > > inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link > > inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:154930 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB) > > > > eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1 > > inet addr:172.88.[SNIP] Bcast:255.255.255.255 > Mask:255.255.240.0 > > inet6 addr: [SNIP]:d474:3d71/128 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB) > > > > eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:782123 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB) > > > > eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:2695 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB) > > > > ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27 > > inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link > > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 > > RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0 > > TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:32 > <----- queue len 32? Curious > > RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 > GiB) > > > > > > System info > > > > This has a prehistoric kernel, I guess because they have some stuff that > taints the kernel > > > > root@USG-Pro-4:~# uname -a > > Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 > GNU/Linux > > [SM] I remember the time we felt great about using a series 3 kernel > instead of old series 2 gunk, but upstream is at series 6 by now (but it > also started to increase major numbers more aggressively after series 2) > > > > > root@USG-Pro-4:~# cat /var/log/dmesg | grep taint > > ubnt_platform: module license 'Proprietary' taints kernel. > > Disabling lock debugging due to kernel taint > > > > I also notice this module, but I'm not sure it is in use. > > /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko > > > > > > root@USG-Pro-4:~# cat /proc/cpuinfo > > system type : UBNT_E220 > > machine : Unknown > > processor : 0 > > cpu model : Cavium Octeon II V0.1 > > BogoMIPS : 2000.00 > > wait instruction : yes > > microsecond timers : yes > > tlb_entries : 128 > > extra interrupt vector : yes > > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > > ASEs implemented : > > shadow register sets : 1 > > kscratch registers : 3 > > core : 0 > > VCED exceptions : not available > > VCEI exceptions : not available > > > > processor : 1 > > cpu model : Cavium Octeon II V0.1 > > BogoMIPS : 2000.00 > > wait instruction : yes > > microsecond timers : yes > > tlb_entries : 128 > > extra interrupt vector : yes > > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > > ASEs implemented : > > shadow register sets : 1 > > kscratch registers : 3 > > core : 1 > > VCED exceptions : not available > > VCEI exceptions : not available > > > > > > > > root@USG-Pro-4:~# ethtool -i eth2 > > driver: octeon-ethernet > > version: 2.0 > > firmware-version: > > bus-info: Builtin > > supports-statistics: no > > supports-test: no > > supports-eeprom-access: no > > supports-register-dump: no > > supports-priv-flags: no > > > > root@USG-Pro-4:~# ethtool -S eth2 > > no stats available > > > > ( Oh great! Thanks guys! ) > > > > root@USG-Pro-4:~# netstat -ia > > Kernel Interface table > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP > TX-OVR Flg > > eth0 1500 0 11340913 0 0 0 21612063 0 0 > 0 BMRU > > eth1 1500 0 154902 0 0 0 233236 0 0 > 0 BMRU > > eth2 1500 0 60898610 0 0 0 10544414 0 0 > 0 BMRU > > eth3 1500 0 0 0 0 0 0 0 0 > 0 BM > > eth0.20 1500 0 781992 0 0 0 480214 0 0 > 0 BMRU > > eth0.40 1500 0 2695 0 0 0 194260 0 0 > 0 BMRU > > ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 > 0 BORU <---- RX drops? > > imq0 16000 0 0 0 0 0 0 0 0 > 0 ORU > > lo 65536 0 9255 0 0 0 9255 0 0 > 0 LRU > > loop0 1500 0 0 0 0 0 0 0 0 > 0 BM > > loop1 1500 0 0 0 0 0 0 0 0 > 0 BM > > loop2 1500 0 0 0 0 0 0 0 0 > 0 BM > > loop3 1500 0 0 0 0 0 0 0 0 > 0 BM > > npi0 1500 0 0 0 0 0 0 0 0 > 0 BM > > npi1 1500 0 0 0 0 0 0 0 0 > 0 BM > > npi2 1500 0 0 0 0 0 0 0 0 > 0 BM > > npi3 1500 0 0 0 0 0 0 0 0 > 0 BM > > > > root@USG-Pro-4:/opt/vyatta/etc# cat version > > Version: v4.4.57.5578372.230112.0824 > > > > -- > > Regards, > > Dave Seddon > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > -- Regards, Dave Seddon +1 415 857 5102 [-- Attachment #1.2: Type: text/html, Size: 21280 bytes --] [-- Attachment #2: image.png --] [-- Type: image/png, Size: 6574 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-02 21:15 ` dave seddon @ 2024-01-02 21:24 ` Sebastian Moeller 0 siblings, 0 replies; 16+ messages in thread From: Sebastian Moeller @ 2024-01-02 21:24 UTC (permalink / raw) To: dave seddon; +Cc: Cake List Hi Dave, > On Jan 2, 2024, at 22:15, dave seddon <dave.seddon.ca@gmail.com> wrote: > > Thanks Sebastian! > > Now I see the rates!! > > I actually reduced the rates to ensure this device is the bottleneck 80/10 Mb/s > > <image.png> > > root@USG-Pro-4:~# tc -d class show dev eth2 > class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 0 > class fq_codel 100:12c parent 100: > class fq_codel 100:213 parent 100: > class fq_codel 100:22e parent 100: > > root@USG-Pro-4:~# tc -d class show dev ifb_eth2 > class htb 1:10 root leaf 100: prio 0 quantum 200000 rate 76000Kbit ceil 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b level 0 > class fq_codel 100:2c8 parent 100: > class fq_codel 100:3df parent 100: > [SM] They apparently have a rather loose translation from 80/10 GUI rates to shaper settings (derated by 5%), and my personal hobby horse seem to ignore the overhead question more or less entirely**... bad Ubiquity... if you want to be a "good boy" email me ;) **) If that 5% derating is supposed to handle overhead, oh my... per packet overhead stays constant with packet number while payload size scales with packet size so overhead inversely scales with packet size and hence static rerating for per-packet overhead is approximate at best to stay polite. Also what is up with the burst values? why do they differ by 32 octets? Sorry for bothering you with this, these are mostly questions for Ubiquity and the are unlikely to monitor this list ;) Regards Sebastian > On Tue, Jan 2, 2024 at 12:53 PM Sebastian Moeller <moeller0@gmx.de> wrote: > Hi Dave. > > just a few comments from the peanut gallery... > > > > On Jan 2, 2024, at 19:59, dave seddon via Cake <cake@lists.bufferbloat.net> wrote: > > > > G'day, > > > > Happy new year y'all > > +1 > > > > > I thought people might be interested to see what Ubiquity/Unifi is doing with "Smart Queues" on their devices. The documentation on their website is not very informative. > > > > Hopefully, this is vaguely interesting because Ubiquity is widely deployed and apparently they have a market cap of >$8 billion, so you would hope they do a "good job" (... Seems like they might be a target customer for libreqos ) > > > > <image.png> > > https://finance.yahoo.com/quote/ui/ > > > > ( I use Unifi because their wifi stuff seems ok, and all the switching/routing/wifi is all integrated into the single gui control system. Also honestly, I'm not sure I know how to do prefix delegation stuff on Linux by hand. ) > > > > Network diagram > > > > Spectrum Cable Internets <----------> Eth2 [ USG-Pro-4 ] Eth0 <---> [Switches] <----> Access points > > > > "Smart Queue" Configuration > > Ubiquity doesn't have many knobs, you just enable "smart queues" and set the bandwidth. > > > > > > > > > > "Smart Queue" Implementation > > > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is NOT cake, but fq_codel. > > > > And cake isn't available :( > > > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt 20ms > > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > > Outbound eth2 > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > > Sent 1071636465 bytes 5624944 pkt (dropped 0, overlimits 523078 requeues 0) <---- OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn > > Sent 1071636465 bytes 5624944 pkt (dropped 2384, overlimits 0 requeues 0) <----- DROPS > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 1244991 ecn_mark 0 > > new_flows_len 1 old_flows_len 1 > > qdisc ingress ffff: parent ffff:fff1 ---------------- > > Sent 12636045136 bytes 29199533 pkt (dropped 0, overlimits 0 requeues 0) > > backlog 0b 0p requeues 0 > > • target 5.0ms is the default ( https://www.man7.org/linux/man-pages/man8/tc-fq_codel.8.html ). I wonder if they did much testing on this hardware? > > [SM] Not sure whether playing with target in isolation would be much use, in codel theory target should be 5-10% of interval ans interval should be in the order of magnitude of to be handled RTTs (the default is 100ms wich works reasonably well even across the Atlantic, but you probably knew all that). > > > • ( I actually have a spare "wan" ethernet port, so I guess I could hook up a PC and perform a flent test. ) > > • It's unclear to me what the "htb" is doing, because I would have expected the download/upload rates to be configured here, but they appear not to be > > [SM] Likely because HTB does not reveal this when asked with the `-s` option, try `-q` instead and not as qdisc but as class (so maybe `tc -d class show dev eth2`). > > > • I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN? > > I think this text about TBF explains this reasonably well (HTB is essentially a hierarchical version of TBF): > > see: https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.classless.html > > 9.2.2. Token Bucket Filter > > The Token Bucket Filter (TBF) is a simple qdisc that only passes packets arriving at a rate which is not exceeding some administratively set rate, but with the possibility to allow short bursts in excess of this rate. > > TBF is very precise, network- and processor friendly. It should be your first choice if you simply want to slow an interface down! > > The TBF implementation consists of a buffer (bucket), constantly filled by some virtual pieces of information called tokens, at a specific rate (token rate). The most important parameter of the bucket is its size, that is the number of tokens it can store. > > Each arriving token collects one incoming data packet from the data queue and is then deleted from the bucket. Associating this algorithm with the two flows -- token and data, gives us three possible scenarios: > > • The data arrives in TBF at a rate that's equal to the rate of incoming tokens. In this case each incoming packet has its matching token and passes the queue without delay. > > • The data arrives in TBF at a rate that's smaller than the token rate. Only a part of the tokens are deleted at output of each data packet that's sent out the queue, so the tokens accumulate, up to the bucket size. The unused tokens can then be used to send data a a speed that's exceeding the standard token rate, in case short data bursts occur. > > • The data arrives in TBF at a rate bigger than the token rate. This means that the bucket will soon be devoid of tokens, which causes the TBF to throttle itself for a while. This is called an 'overlimit situation'. If packets keep coming in, packets will start to get dropped. > > The last scenario is very important, because it allows to administratively shape the bandwidth available to data that's passing the filter. > > The accumulation of tokens allows a short burst of overlimit data to be still passed without loss, but any lasting overload will cause packets to be constantly delayed, and then dropped. > > Please note that in the actual implementation, tokens correspond to bytes, not packets. > > > > > > Inbound eth2 via ifb > > > > root@USG-Pro-4:~# tc -p -s -d qdisc show dev ifb_eth2 > > qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 > > Sent 13029810569 bytes 29185742 pkt (dropped 0, overlimits 14774339 requeues 0) <---- OVERLIMITS? > > backlog 0b 0p requeues 0 > > qdisc fq_codel 100: parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn > > Sent 13029810569 bytes 29185742 pkt (dropped 10688, overlimits 0 requeues 0) <---- WOW. DROPS!! > > backlog 0b 0p requeues 0 > > maxpacket 1514 drop_overlimit 0 new_flow_count 2256895 ecn_mark 0 > > new_flows_len 0 old_flows_len 2 > > > > Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. > > [SM] Same approach that sqm-scripts takes, if you attach the ingress shaper to the LAN port egress all internet traffic not traversing that interface will not be shaped, e.g. traffic to/from the router itself or WiFi traffic. If you are sure that such by-pass traffic does not exist, putting the ingress shaper on lan-egress can save the cost of the ifb indirection, but for full WiFi routers that is generally not true. > > > > Initially, I was pretty surprised to see so many drops on the inbound path, but maybe this is actually normal? > > [SM] Depends on your traffic and whether ECN is used or not. In your case it appears ECN is not used and then DROPS are the only way fq_codel can tell a flow to step on the brakes.... > > > > > I could imagine the upstream CDNs pushing pretty hard with low RTTs, but I would probably have expected the bottlenecks to form at the access points. e.g. It's gigabit all the way until it reaches the air interface of the access points. .... Or do I have a problem in my LAN network? > > [SM] The idea is to create an artificial bittleneck (using HTB) so the most relevant queue is under AQM control... > > > > > I wonder if I can log into the access points to look at them too?.... > > > > ( BTW - to get to root on these devices you can SSH in as an "admin" users, and then just "sudo su" ) > > > > ifconfig > > > > root@USG-Pro-4:~# ifconfig -a > > eth0 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.50.1 Bcast:172.16.50.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:feec:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:11343139 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:21614272 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 <---- queue len 0? Maybe this is a driver issue? > > RX bytes:2047750597 (1.9 GiB) TX bytes:23484692545 (21.8 GiB) > > > > eth1 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a0 > > inet addr:172.16.51.1 Bcast:172.16.51.255 Mask:255.255.255.0 > > inet6 addr: fe80::feec:daff:fed1:1ba0/64 Scope:Link > > inet6 addr: [SNIP]:daff:fed1:1ba0/64 Scope:Global > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:154930 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:233294 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:32255162 (30.7 MiB) TX bytes:116504400 (111.1 MiB) > > > > eth2 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a1 > > inet addr:172.88.[SNIP] Bcast:255.255.255.255 Mask:255.255.240.0 > > inet6 addr: [SNIP]:d474:3d71/128 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1ba1/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:60912335 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:10546508 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:26087920038 (24.2 GiB) TX bytes:1892854725 (1.7 GiB) > > > > eth3 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:a2 > > BROADCAST MULTICAST MTU:1500 Metric:1 > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > eth0.20 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.60.1 Bcast:172.16.60.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:782123 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:480343 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:60600161 (57.7 MiB) TX bytes:108372413 (103.3 MiB) > > > > eth0.40 Link encap:Ethernet HWaddr fc:ec:da:d1:1b:9f > > inet addr:172.16.40.1 Bcast:172.16.40.255 Mask:255.255.255.0 > > inet6 addr: [SNIP]:daff:fed1:1b9f/64 Scope:Global > > inet6 addr: fe80::feec:daff:fed1:1b9f/64 Scope:Link > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:2695 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:194291 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:123970 (121.0 KiB) TX bytes:42370172 (40.4 MiB) > > > > ifb_eth2 Link encap:Ethernet HWaddr de:ed:87:85:80:27 > > inet6 addr: fe80::dced:87ff:fe85:8027/64 Scope:Link > > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 > > RX packets:29656324 errors:0 dropped:2531 overruns:0 frame:0 > > TX packets:29653793 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:32 <----- queue len 32? Curious > > RX bytes:13086765284 (12.1 GiB) TX bytes:13086264146 (12.1 GiB) > > > > > > System info > > > > This has a prehistoric kernel, I guess because they have some stuff that taints the kernel > > > > root@USG-Pro-4:~# uname -a > > Linux USG-Pro-4 3.10.107-UBNT #1 SMP Thu Jan 12 08:30:03 UTC 2023 mips64 GNU/Linux > > [SM] I remember the time we felt great about using a series 3 kernel instead of old series 2 gunk, but upstream is at series 6 by now (but it also started to increase major numbers more aggressively after series 2) > > > > > root@USG-Pro-4:~# cat /var/log/dmesg | grep taint > > ubnt_platform: module license 'Proprietary' taints kernel. > > Disabling lock debugging due to kernel taint > > > > I also notice this module, but I'm not sure it is in use. > > /lib/modules/3.10.107-UBNT/kernel/net/netfilter/xt_rateest.ko > > > > > > root@USG-Pro-4:~# cat /proc/cpuinfo > > system type : UBNT_E220 > > machine : Unknown > > processor : 0 > > cpu model : Cavium Octeon II V0.1 > > BogoMIPS : 2000.00 > > wait instruction : yes > > microsecond timers : yes > > tlb_entries : 128 > > extra interrupt vector : yes > > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > > ASEs implemented : > > shadow register sets : 1 > > kscratch registers : 3 > > core : 0 > > VCED exceptions : not available > > VCEI exceptions : not available > > > > processor : 1 > > cpu model : Cavium Octeon II V0.1 > > BogoMIPS : 2000.00 > > wait instruction : yes > > microsecond timers : yes > > tlb_entries : 128 > > extra interrupt vector : yes > > hardware watchpoint : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb] > > isa : mips1 mips2 mips3 mips4 mips5 mips64r2 > > ASEs implemented : > > shadow register sets : 1 > > kscratch registers : 3 > > core : 1 > > VCED exceptions : not available > > VCEI exceptions : not available > > > > > > > > root@USG-Pro-4:~# ethtool -i eth2 > > driver: octeon-ethernet > > version: 2.0 > > firmware-version: > > bus-info: Builtin > > supports-statistics: no > > supports-test: no > > supports-eeprom-access: no > > supports-register-dump: no > > supports-priv-flags: no > > > > root@USG-Pro-4:~# ethtool -S eth2 > > no stats available > > > > ( Oh great! Thanks guys! ) > > > > root@USG-Pro-4:~# netstat -ia > > Kernel Interface table > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > > eth0 1500 0 11340913 0 0 0 21612063 0 0 0 BMRU > > eth1 1500 0 154902 0 0 0 233236 0 0 0 BMRU > > eth2 1500 0 60898610 0 0 0 10544414 0 0 0 BMRU > > eth3 1500 0 0 0 0 0 0 0 0 0 BM > > eth0.20 1500 0 781992 0 0 0 480214 0 0 0 BMRU > > eth0.40 1500 0 2695 0 0 0 194260 0 0 0 BMRU > > ifb_eth2 1500 0 29642598 0 2530 0 29640068 0 0 0 BORU <---- RX drops? > > imq0 16000 0 0 0 0 0 0 0 0 0 ORU > > lo 65536 0 9255 0 0 0 9255 0 0 0 LRU > > loop0 1500 0 0 0 0 0 0 0 0 0 BM > > loop1 1500 0 0 0 0 0 0 0 0 0 BM > > loop2 1500 0 0 0 0 0 0 0 0 0 BM > > loop3 1500 0 0 0 0 0 0 0 0 0 BM > > npi0 1500 0 0 0 0 0 0 0 0 0 BM > > npi1 1500 0 0 0 0 0 0 0 0 0 BM > > npi2 1500 0 0 0 0 0 0 0 0 0 BM > > npi3 1500 0 0 0 0 0 0 0 0 0 BM > > > > root@USG-Pro-4:/opt/vyatta/etc# cat version > > Version: v4.4.57.5578372.230112.0824 > > > > -- > > Regards, > > Dave Seddon > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > -- > Regards, > Dave Seddon > +1 415 857 5102 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-02 18:59 [Cake] Ubiquity (Unifi ) Smart Queues dave seddon 2024-01-02 20:52 ` Sebastian Moeller @ 2024-01-02 21:57 ` Jonathan Morton 2024-01-03 13:44 ` Pete Heist 2 siblings, 0 replies; 16+ messages in thread From: Jonathan Morton @ 2024-01-02 21:57 UTC (permalink / raw) To: dave seddon; +Cc: Cake List > On 2 Jan, 2024, at 8:59 pm, dave seddon via Cake <cake@lists.bufferbloat.net> wrote: > > • I'm not really sure what "overlimits" means or what that does, and tried looking this up, but I guess the kernel source is likely the "best" documentation for this. Maybe this means it's dropping? Or is it ECN? Overlimit just means the shaper in HTB is restricting the flow, thus doing something useful. Drop means the AQM is working to provide congestion information to a Not-ECT flow. Those numbers look normal (about 0.2% drop rate, and most packets hitting "overlimit" when the link is saturated). Using fq_codel's default parameters is not a bad thing at all. I'd much rather they did that, thereby using numbers that are tuned for general Internet conditions, than try to change that tuning in ignorance of the end-user's actual network environment. Most end-users have their WAN port facing "general Internet conditions" anyway. > Apparently rather than applying the tc qdsic on the outbound path on the LAN side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. That's the correct place to do it, so that the qdisc applies specifically to traffic coming from the WAN. If you apply it to the LAN egress, it tends to affect traffic coming through the router from the WiFi or its internal servers, which is not desirable. These are all questions we've considered at length in the process of developing and deploying Cake and other SQM solutions. - Jonathan Morton ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-02 18:59 [Cake] Ubiquity (Unifi ) Smart Queues dave seddon 2024-01-02 20:52 ` Sebastian Moeller 2024-01-02 21:57 ` Jonathan Morton @ 2024-01-03 13:44 ` Pete Heist 2024-01-09 15:39 ` Nils Andreas Svee 2 siblings, 1 reply; 16+ messages in thread From: Pete Heist @ 2024-01-03 13:44 UTC (permalink / raw) To: dave seddon, Cake List On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote: > I thought people might be interested to see what Ubiquity/Unifi is > doing with "Smart Queues" on their devices. The documentation on > their website is not very informative. > <snip> > "Smart Queue" Implementation > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is > NOT cake, but fq_codel. > > And cake isn't available :( > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt > 20ms > Unknown qdisc "cake", hence option "bandwidth" is unparsable Hi Dave, there's a community contributed version of Cake for EdgeRouter devices that I've been using for years on production ER-X's: https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 I don't think that works for UniFi/USG devices, however, and one should note the disclaimer and be careful when installing it. Also, it must be re-installed after every upgrade. Cheers, Pete ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-03 13:44 ` Pete Heist @ 2024-01-09 15:39 ` Nils Andreas Svee 2024-01-09 15:59 ` Dave Taht 2024-01-09 16:05 ` Dave Taht 0 siblings, 2 replies; 16+ messages in thread From: Nils Andreas Svee @ 2024-01-09 15:39 UTC (permalink / raw) To: Pete Heist; +Cc: dave seddon, CAKE list [-- Attachment #1: Type: text/plain, Size: 2460 bytes --] You’re unlikely to do any real harm though, but the warning is there cause you can potentially soft brick your router using it. I’ve run into that myself if I remember correctly, where after a firmware upgrade the kernel had slightly changed, so loading the sch_cake module caused it to panic. And I had it start through /config/scripts/post-config.d of course, so it would happen on every restart. Nothing a factory reset won’t solve, but annoying when if you’re messing about remotely :) As for USG, I think I used to have some binaries for those too. I do still have some old kernel sources for them laying around in a repo. It’s been awhile, but I probably stopped building for those as it wasn’t as straightforward to keep up with the versions of the firmware. Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. Best Regards, Nils Andreas Svee > On Jan 3, 2024, at 14:44, Pete Heist via Cake <cake@lists.bufferbloat.net> wrote: > > On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote: >> I thought people might be interested to see what Ubiquity/Unifi is >> doing with "Smart Queues" on their devices. The documentation on >> their website is not very informative. >> <snip> >> "Smart Queue" Implementation >> >> Looks like they only apply tc qdiscs to the Eth2, and sadly this is >> NOT cake, but fq_codel. >> >> And cake isn't available :( >> >> root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt >> 20ms >> Unknown qdisc "cake", hence option "bandwidth" is unparsable > > Hi Dave, there's a community contributed version of Cake for EdgeRouter > devices that I've been using for years on production ER-X's: > > https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 > > I don't think that works for UniFi/USG devices, however, and one should > note the disclaimer and be careful when installing it. Also, it must be > re-installed after every upgrade. > > Cheers, > Pete > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake [-- Attachment #2: Type: text/html, Size: 2970 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 15:39 ` Nils Andreas Svee @ 2024-01-09 15:59 ` Dave Taht 2024-01-09 16:05 ` Dave Taht 1 sibling, 0 replies; 16+ messages in thread From: Dave Taht @ 2024-01-09 15:59 UTC (permalink / raw) To: Nils Andreas Svee; +Cc: Pete Heist, CAKE list It is so nice to see this list come to life again. I just wanted to point out that the inbound drop rate was merely .03% in the first email. Elsewhere we keep hearing of 1-2% drop rates being common, and I just aint seeing it in any of the larger scale data I have been getting from the libreqos deployment. Maybe that´s how fifos collapse in slow start a lot more than we have ever observed. Maybe it is retransmits going wild at 250ms worth of buffering. scale=10 10688/29185742 .0003662062 I still have to sit back and admire you all for this magnificent achievement of cake. I am not really the gloating sort, but seeing juniper go up for sale to hpe today, at a firesale price, is oddly satisfying. https://blog.cerowrt.org/post/juniper/ On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake <cake@lists.bufferbloat.net> wrote: > > You’re unlikely to do any real harm though, but the warning is there cause you can potentially soft brick your router using it. I’ve run into that myself if I remember correctly, where after a firmware upgrade the kernel had slightly changed, so loading the sch_cake module caused it to panic. And I had it start through /config/scripts/post-config.d of course, so it would happen on every restart. > > Nothing a factory reset won’t solve, but annoying when if you’re messing about remotely :) > > As for USG, I think I used to have some binaries for those too. I do still have some old kernel sources for them laying around in a repo. > It’s been awhile, but I probably stopped building for those as it wasn’t as straightforward to keep up with the versions of the firmware. > > Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. > > Best Regards, > Nils Andreas Svee > > On Jan 3, 2024, at 14:44, Pete Heist via Cake <cake@lists.bufferbloat.net> wrote: > > On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote: > > I thought people might be interested to see what Ubiquity/Unifi is > doing with "Smart Queues" on their devices. The documentation on > their website is not very informative. > <snip> > "Smart Queue" Implementation > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is > NOT cake, but fq_codel. > > And cake isn't available :( > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt > 20ms > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > Hi Dave, there's a community contributed version of Cake for EdgeRouter > devices that I've been using for years on production ER-X's: > > https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 > > I don't think that works for UniFi/USG devices, however, and one should > note the disclaimer and be careful when installing it. Also, it must be > re-installed after every upgrade. > > Cheers, > Pete > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- 40 years of net history, a couple songs: https://www.youtube.com/watch?v=D9RGX6QFm5E Dave Täht CSO, LibreQos ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 15:39 ` Nils Andreas Svee 2024-01-09 15:59 ` Dave Taht @ 2024-01-09 16:05 ` Dave Taht 2024-01-09 16:47 ` dave seddon 2024-01-09 16:57 ` Nils Andreas Svee 1 sibling, 2 replies; 16+ messages in thread From: Dave Taht @ 2024-01-09 16:05 UTC (permalink / raw) To: Nils Andreas Svee; +Cc: Pete Heist, CAKE list On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake <cake@lists.bufferbloat.net> wrote: > Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. This irks me enormously. It is the direct outcome of the cambium elevate lawsuit, where both companies lost, the ISPs lost, open source practices long established about publishing sources, lost, and the lawyers went on to other nasty things leaving this trail of awful precedents in their wake. https://www.mtin.net/blog/ubnt-vs-cambium/ I do not know what to do about it. It also irks me that as a contributor to "smart queues" they are not maintaining it well. > > Best Regards, > Nils Andreas Svee > > On Jan 3, 2024, at 14:44, Pete Heist via Cake <cake@lists.bufferbloat.net> wrote: > > On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote: > > I thought people might be interested to see what Ubiquity/Unifi is > doing with "Smart Queues" on their devices. The documentation on > their website is not very informative. > <snip> > "Smart Queue" Implementation > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is > NOT cake, but fq_codel. > > And cake isn't available :( > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt > 20ms > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > Hi Dave, there's a community contributed version of Cake for EdgeRouter > devices that I've been using for years on production ER-X's: > > https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 > > I don't think that works for UniFi/USG devices, however, and one should > note the disclaimer and be careful when installing it. Also, it must be > re-installed after every upgrade. > > Cheers, > Pete > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake -- 40 years of net history, a couple songs: https://www.youtube.com/watch?v=D9RGX6QFm5E Dave Täht CSO, LibreQos ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 16:05 ` Dave Taht @ 2024-01-09 16:47 ` dave seddon 2024-01-09 16:57 ` Nils Andreas Svee 1 sibling, 0 replies; 16+ messages in thread From: dave seddon @ 2024-01-09 16:47 UTC (permalink / raw) To: Dave Taht; +Cc: Nils Andreas Svee, CAKE list [-- Attachment #1: Type: text/plain, Size: 8066 bytes --] Thanks to everyone for the comments. Wow Dave. Thanks for the links to the lawsuit. Fascinating. I didn't know about this. Since the original email, I actually downgraded from Spectrum cable 300 Mb/s to ? maybe it's 200 now ?, anyway it's $20 a month cheaper. And decreased the "smart queue" limits to 80/10 Mb/s. Video streaming multiple TVs at once is all working well. No complaints from the family. Win! Stats are looking pretty good after ~8 days uptime root@USG-Pro-4:/home/daveseddon# tc -p -s -d qdisc show qdisc fq_codel 8001: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 161758598841 bytes 147108740 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 299 ecn_mark 0 new_flows_len 0 old_flows_len 4 qdisc htb 1: dev eth2 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 Sent 32491619852 bytes 93191686 pkt (dropped 0, overlimits 36499993 requeues 0) <---- OVER ( 36499993 / 32491619852 ~= 0.00112) backlog 0b 0p requeues 0 qdisc fq_codel 100: dev eth2 parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 32491619852 bytes 93191686 pkt (dropped 37156, overlimits 0 requeues 0) <------- DROPS ( 37156 / 32491619852 ~= 0.0000011 ) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 15549718 ecn_mark 1572 <--- some ECN new_flows_len 1 old_flows_len 5 qdisc ingress ffff: dev eth2 parent ffff:fff1 ---------------- Sent 169147707173 bytes 346339031 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc pfifo_fast 0: dev imq0 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc htb 1: dev ifb_eth2 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17 Sent 173801196835 bytes 346221051 pkt (dropped 0, overlimits 192974926 requeues 0) <------- OVER ( 192974926 / 173801196835 ~= 0.0011 ) backlog 0b 0p requeues 0 qdisc fq_codel 100: dev ifb_eth2 parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 173801196835 bytes 346221051 pkt (dropped 140361, overlimits 0 requeues 0) <------- DROPS ( 37156 / 32491619852 ~= 0.00000080 ) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 30970803 ecn_mark 1721 <--- some ECN new_flows_len 1 old_flows_len 3 root@USG-Pro-4:/home/daveseddon# tc -d class show dev eth2 class htb 1:10 root leaf 100: prio 0 quantum 118750 rate 9500Kbit ceil 9500Kbit burst 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 0 class fq_codel 100:98 parent 100: class fq_codel 100:c7 parent 100: class fq_codel 100:180 parent 100: class fq_codel 100:238 parent 100: class fq_codel 100:305 parent 100: root@USG-Pro-4:/home/daveseddon# tc -d class show dev ifb_eth2 class htb 1:10 root leaf 100: prio 0 quantum 200000 rate 76000Kbit ceil 76000Kbit burst 1596b/1 mpu 0b overhead 0b cburst 1596b/1 mpu 0b overhead 0b level 0 class fq_codel 100:247 parent 100: class fq_codel 100:2c8 parent 100: root@USG-Pro-4:/home/daveseddon# uptime 08:18:21 up 8 days, 19:02, 1 user, load average: 0.00, 0.01, 0.05 root@USG-Pro-4:/home/daveseddon# netstat -ia Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 101353919 0 0 0 157787555 0 0 0 BMRU eth1 1500 0 663477 0 0 0 1090665 0 0 0 BMRU eth2 1500 0 377699134 0 0 0 98049876 0 0 0 BMRU eth3 1500 0 0 0 0 0 0 0 0 0 BM eth0.20 1500 0 3437713 0 0 0 2260022 0 0 0 BMRU eth0.40 1500 0 12524 0 0 0 1012668 0 0 0 BMRU ifb_eth2 1500 0 346333308 0 22391 0 346310917 0 0 0 BORU <--- i'm still curious about these drops, but it's a tiny amount. 22391 / 346333308 ~= 0.000064 imq0 16000 0 0 0 0 0 0 0 0 0 ORU lo 65536 0 38330 0 0 0 38330 0 0 0 LRU loop0 1500 0 0 0 0 0 0 0 0 0 BM loop1 1500 0 0 0 0 0 0 0 0 0 BM loop2 1500 0 0 0 0 0 0 0 0 0 BM loop3 1500 0 0 0 0 0 0 0 0 0 BM npi0 1500 0 0 0 0 0 0 0 0 0 BM npi1 1500 0 0 0 0 0 0 0 0 0 BM npi2 1500 0 0 0 0 0 0 0 0 0 BM npi3 1500 0 0 0 0 0 0 0 0 0 BM On Tue, Jan 9, 2024 at 8:05 AM Dave Taht via Cake < cake@lists.bufferbloat.net> wrote: > On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake > <cake@lists.bufferbloat.net> wrote: > > > Though frankly, I don’t plan on updating the sch_cake and tc binaries > when new firmwares are released anymore, as they don’t publish the GPL > archives on their webpage after the redesign, and they don’t respond to > requests for them either by the looks of the forums. So if it breaks > there’s not much I can do anymore. > > This irks me enormously. It is the direct outcome of the cambium > elevate lawsuit, where both companies lost, the ISPs lost, open source > practices long established about publishing sources, lost, and the > lawyers went on to other nasty things leaving this trail of awful > precedents in their wake. > > https://www.mtin.net/blog/ubnt-vs-cambium/ > > I do not know what to do about it. It also irks me that as a > contributor to "smart queues" they are not maintaining it well. > > > > > Best Regards, > > Nils Andreas Svee > > > > On Jan 3, 2024, at 14:44, Pete Heist via Cake < > cake@lists.bufferbloat.net> wrote: > > > > On Tue, 2024-01-02 at 10:59 -0800, dave seddon via Cake wrote: > > > > I thought people might be interested to see what Ubiquity/Unifi is > > doing with "Smart Queues" on their devices. The documentation on > > their website is not very informative. > > <snip> > > "Smart Queue" Implementation > > > > Looks like they only apply tc qdiscs to the Eth2, and sadly this is > > NOT cake, but fq_codel. > > > > And cake isn't available :( > > > > root@USG-Pro-4:~# tc qdisc replace dev eth0 cake bandwidth 100m rtt > > 20ms > > Unknown qdisc "cake", hence option "bandwidth" is unparsable > > > > > > Hi Dave, there's a community contributed version of Cake for EdgeRouter > > devices that I've been using for years on production ER-X's: > > > > > https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devices/fc1ff27c-f321-4344-8737-fcc755cae8a2 > > > > I don't think that works for UniFi/USG devices, however, and one should > > note the disclaimer and be careful when installing it. Also, it must be > > re-installed after every upgrade. > > > > Cheers, > > Pete > > > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > -- > 40 years of net history, a couple songs: > https://www.youtube.com/watch?v=D9RGX6QFm5E > Dave Täht CSO, LibreQos > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > -- Regards, Dave Seddon +1 415 857 5102 [-- Attachment #2: Type: text/html, Size: 12194 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 16:05 ` Dave Taht 2024-01-09 16:47 ` dave seddon @ 2024-01-09 16:57 ` Nils Andreas Svee 2024-01-09 17:07 ` dave seddon 1 sibling, 1 reply; 16+ messages in thread From: Nils Andreas Svee @ 2024-01-09 16:57 UTC (permalink / raw) To: Dave Taht; +Cc: Pete Heist, CAKE list [-- Attachment #1: Type: text/plain, Size: 1965 bytes --] > On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com> wrote: > > On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake > <cake@lists.bufferbloat.net <mailto:cake@lists.bufferbloat.net>> wrote: > >> Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. > > This irks me enormously. It is the direct outcome of the cambium > elevate lawsuit, where both companies lost, the ISPs lost, open source > practices long established about publishing sources, lost, and the > lawyers went on to other nasty things leaving this trail of awful > precedents in their wake. > https://www.mtin.net/blog/ubnt-vs-cambium/ Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s software on their hardware? That is crazy, just evil corporate lawyers doing their thing I guess. > I do not know what to do about it. It also irks me that as a > contributor to "smart queues" they are not maintaining it well. It leaves something to be desired yes, and I would’ve hoped to see CAKE included too of course, but even WireGuard is only available in the latest release candidates with the redesigned web UI, so I’m not holding my breath. I still have an EdgeRouter 4 that serves the family farm and one of the 8-port switches under my desk, if only because I don’t wanna spend money on replacing them, and they do serve their purpose. I’ve since moved though, and now live in an area that has FTTH, so I needed something beefier to handle CAKE on a 750/750 subscription, because obviously there’s still bloat even on that ;) One of those Chinese boxes with a N100 in it and OpenWrt on top works wonders :) Best Regards, Nils Andreas Svee [-- Attachment #2: Type: text/html, Size: 10649 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 16:57 ` Nils Andreas Svee @ 2024-01-09 17:07 ` dave seddon 2024-01-09 17:17 ` Dave Taht 2024-01-09 23:17 ` Nils Andreas Svee 0 siblings, 2 replies; 16+ messages in thread From: dave seddon @ 2024-01-09 17:07 UTC (permalink / raw) To: Nils Andreas Svee; +Cc: Dave Taht, CAKE list [-- Attachment #1: Type: text/plain, Size: 2350 bytes --] Nils - I guess you could run LibreQoS on N100? On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake < cake@lists.bufferbloat.net> wrote: > On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com> wrote: > > On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake > <cake@lists.bufferbloat.net> wrote: > > Though frankly, I don’t plan on updating the sch_cake and tc binaries when > new firmwares are released anymore, as they don’t publish the GPL archives > on their webpage after the redesign, and they don’t respond to requests for > them either by the looks of the forums. So if it breaks there’s not much I > can do anymore. > > > This irks me enormously. It is the direct outcome of the cambium > elevate lawsuit, where both companies lost, the ISPs lost, open source > practices long established about publishing sources, lost, and the > lawyers went on to other nasty things leaving this trail of awful > precedents in their wake. > > https://www.mtin.net/blog/ubnt-vs-cambium/ > > Wow, hadn’t read about that. They even sued an ISP just for using > Cambium’s software on their hardware? > That is crazy, just evil corporate lawyers doing their thing I guess. > > I do not know what to do about it. It also irks me that as a > contributor to "smart queues" they are not maintaining it well. > > It leaves something to be desired yes, and I would’ve hoped to see CAKE > included too of course, > but even WireGuard is only available in the latest release candidates with > the redesigned web UI, so I’m not holding my breath. > > I still have an EdgeRouter 4 that serves the family farm and one of the > 8-port switches under my desk, if only because I don’t wanna spend money on > replacing them, and they do serve their purpose. > > I’ve since moved though, and now live in an area that has FTTH, so I > needed something beefier to handle CAKE on a 750/750 subscription, because > obviously there’s still bloat even on that ;) > > One of those Chinese boxes with a N100 in it and OpenWrt on top works > wonders :) > > Best Regards, > Nils Andreas Svee > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > -- Regards, Dave Seddon +1 415 857 5102 [-- Attachment #2: Type: text/html, Size: 9352 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 17:07 ` dave seddon @ 2024-01-09 17:17 ` Dave Taht 2024-01-09 23:28 ` Nils Andreas Svee 2024-01-09 23:17 ` Nils Andreas Svee 1 sibling, 1 reply; 16+ messages in thread From: Dave Taht @ 2024-01-09 17:17 UTC (permalink / raw) To: dave seddon; +Cc: Nils Andreas Svee, CAKE list Principal limitation for libreqos on a small box is has to have multiple hardware queues and support eBPF. Seriously folks, running libreqos at home is *serious overkill*, although I have to admit the traffic graphs are mesmerizing!!! One of our ISPs has been setting them to music: https://www.youtube.com/@trendaltoews7143 Herbert has been working on adding all sorts of other analytics to it also. On Tue, Jan 9, 2024 at 12:07 PM dave seddon <dave.seddon.ca@gmail.com> wrote: > > Nils - I guess you could run LibreQoS on N100? > > On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake <cake@lists.bufferbloat.net> wrote: >> >> On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com> wrote: >> >> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake >> <cake@lists.bufferbloat.net> wrote: >> >> Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. >> >> >> This irks me enormously. It is the direct outcome of the cambium >> elevate lawsuit, where both companies lost, the ISPs lost, open source >> practices long established about publishing sources, lost, and the >> lawyers went on to other nasty things leaving this trail of awful >> precedents in their wake. >> >> https://www.mtin.net/blog/ubnt-vs-cambium/ >> >> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s software on their hardware? >> That is crazy, just evil corporate lawyers doing their thing I guess. >> >> I do not know what to do about it. It also irks me that as a >> contributor to "smart queues" they are not maintaining it well. >> >> It leaves something to be desired yes, and I would’ve hoped to see CAKE included too of course, >> but even WireGuard is only available in the latest release candidates with the redesigned web UI, so I’m not holding my breath. >> >> I still have an EdgeRouter 4 that serves the family farm and one of the 8-port switches under my desk, if only because I don’t wanna spend money on replacing them, and they do serve their purpose. >> >> I’ve since moved though, and now live in an area that has FTTH, so I needed something beefier to handle CAKE on a 750/750 subscription, because obviously there’s still bloat even on that ;) >> >> One of those Chinese boxes with a N100 in it and OpenWrt on top works wonders :) >> >> Best Regards, >> Nils Andreas Svee >> _______________________________________________ >> Cake mailing list >> Cake@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cake > > > > -- > Regards, > Dave Seddon > +1 415 857 5102 -- 40 years of net history, a couple songs: https://www.youtube.com/watch?v=D9RGX6QFm5E Dave Täht CSO, LibreQos ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 17:17 ` Dave Taht @ 2024-01-09 23:28 ` Nils Andreas Svee 2024-04-29 21:55 ` dave seddon 0 siblings, 1 reply; 16+ messages in thread From: Nils Andreas Svee @ 2024-01-09 23:28 UTC (permalink / raw) To: Dave Taht; +Cc: dave seddon, CAKE list [-- Attachment #1: Type: text/plain, Size: 3576 bytes --] Well my NIC has 4 queues as far as I can tell, so it could likely work, but as you say it’s like killing a mosquito with a gatling gun. Those graphs are sweet though, and it’s been in my backlog for awhile to do something with Grafana to get something similar, like this one from a few years ago you’ve seen too: https://forum.openwrt.org/t/sqm-reporting/59960/96 Best Regards, Nils Andreas Svee > On Jan 9, 2024, at 18:17, Dave Taht <dave.taht@gmail.com> wrote: > > Principal limitation for libreqos on a small box is has to have > multiple hardware queues and support eBPF. > > Seriously folks, running libreqos at home is *serious overkill*, > although I have to admit the traffic graphs are mesmerizing!!! One of > our ISPs has been setting them to music: > https://www.youtube.com/@trendaltoews7143 > > Herbert has been working on adding all sorts of other analytics to it also. > > On Tue, Jan 9, 2024 at 12:07 PM dave seddon <dave.seddon.ca@gmail.com> wrote: >> >> Nils - I guess you could run LibreQoS on N100? >> >> On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake <cake@lists.bufferbloat.net> wrote: >>> >>> On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com> wrote: >>> >>> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake >>> <cake@lists.bufferbloat.net> wrote: >>> >>> Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. >>> >>> >>> This irks me enormously. It is the direct outcome of the cambium >>> elevate lawsuit, where both companies lost, the ISPs lost, open source >>> practices long established about publishing sources, lost, and the >>> lawyers went on to other nasty things leaving this trail of awful >>> precedents in their wake. >>> >>> https://www.mtin.net/blog/ubnt-vs-cambium/ >>> >>> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s software on their hardware? >>> That is crazy, just evil corporate lawyers doing their thing I guess. >>> >>> I do not know what to do about it. It also irks me that as a >>> contributor to "smart queues" they are not maintaining it well. >>> >>> It leaves something to be desired yes, and I would’ve hoped to see CAKE included too of course, >>> but even WireGuard is only available in the latest release candidates with the redesigned web UI, so I’m not holding my breath. >>> >>> I still have an EdgeRouter 4 that serves the family farm and one of the 8-port switches under my desk, if only because I don’t wanna spend money on replacing them, and they do serve their purpose. >>> >>> I’ve since moved though, and now live in an area that has FTTH, so I needed something beefier to handle CAKE on a 750/750 subscription, because obviously there’s still bloat even on that ;) >>> >>> One of those Chinese boxes with a N100 in it and OpenWrt on top works wonders :) >>> >>> Best Regards, >>> Nils Andreas Svee >>> _______________________________________________ >>> Cake mailing list >>> Cake@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cake >> >> >> >> -- >> Regards, >> Dave Seddon >> +1 415 857 5102 > > > > -- > 40 years of net history, a couple songs: > https://www.youtube.com/watch?v=D9RGX6QFm5E > Dave Täht CSO, LibreQos [-- Attachment #2: Type: text/html, Size: 4081 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 23:28 ` Nils Andreas Svee @ 2024-04-29 21:55 ` dave seddon 0 siblings, 0 replies; 16+ messages in thread From: dave seddon @ 2024-04-29 21:55 UTC (permalink / raw) To: CAKE list [-- Attachment #1.1: Type: text/plain, Size: 11685 bytes --] G'day, Just a small update on the Unifi security gateway stuff. They have a new range of devices which are a lot more powerful. ( https://store.ui.com/us/en/collections/cloud-gateway-ultra/products/ucg-ultra ) The good news is that the limits set in the GUI now match exactly the "rate" set in the qcdisc. root@UCG-Ultra:~# *tc -p -s -d qdisc show dev eth4* qdisc htb 1: root refcnt 5 r2q 10 default 0x2 direct_packets_stat 0 ver 3.17 direct_qlen 1000 Sent 13112672757 bytes 41407610 pkt (dropped 2863, overlimits 12123381 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 2: parent 1:2 limit 2000p flows 1024 quantum 300 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 Sent 13112672757 bytes 41407610 pkt (dropped 2863, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 27888 drop_overlimit 0 new_flow_count 9175282 ecn_mark 0 new_flows_len 1 old_flows_len 3 qdisc ingress ffff: parent ffff:fff1 ---------------- Sent 104038056896 bytes 143646981 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 root@UCG-Ultra:/etc/systemd# *tc -d class show dev eth4* class htb 1:1 root rate 35Mbit ceil 35Mbit linklayer ethernet burst 1491b/1 mpu 0b cburst 1491b/1 mpu 0b level 7 class htb 1:2 parent 1:1 leaf 2: prio 7 quantum 1514 rate 64bit ceil 35Mbit linklayer ethernet burst 1500b/1 mpu 0b cburst 1491b/1 mpu 0b level 0 class fq_codel 2:1bf parent 2: class fq_codel 2:274 parent 2: class fq_codel 2:296 parent 2: class fq_codel 2:2ca parent 2: class fq_codel 2:34a parent 2: class fq_codel 2:364 parent 2: root@UCG-Ultra:~# *tc -p -s -d qdisc show dev ifbeth4* qdisc htb 1: root refcnt 2 r2q 10 default 0x2 direct_packets_stat 0 ver 3.17 direct_qlen 1000 Sent 108770017013 bytes 143572868 pkt (dropped 24028, overlimits 43487579 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 2: parent 1:2 limit 2000p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 Sent 108770017013 bytes 143572868 pkt (dropped 24028, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 69876 drop_overlimit 10448 new_flow_count 14414347 ecn_mark 0 drop_overmemory 10448 new_flows_len 1 old_flows_len 2 root@UCG-Ultra:/etc/systemd# *tc -d class show dev ifbeth4* class htb 1:1 root rate 800Mbit ceil 800Mbit linklayer ethernet burst 1400b/1 mpu 0b cburst 1400b/1 mpu 0b level 7 class htb 1:2 parent 1:1 leaf 2: prio 7 quantum 1514 rate 64bit ceil 800Mbit linklayer ethernet burst 1500b/1 mpu 0b cburst 1400b/1 mpu 0b level 0 class fq_codel 2:111 parent 2: class fq_codel 2:3cc parent 2: So 35Mb/s and 800Mb/s match what is configured in the GUI. [image: image.png] The bad news is still no cake. The bottleneck in my house is now the air interfaces. I'll run some flent tests soon. Thanks, Dave Seddon Other device details.... root@UCG-Ultra:~# uname -a Linux UCG-Ultra 5.4.213-ui-ipq5322 #5.4.213 SMP PREEMPT Fri Jan 26 01:53:55 CST 2024 aarch64 GNU/Linux root@UCG-Ultra:~# cat /proc/cpuinfo processor : 0 BogoMIPS : 48.00 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid CPU implementer : 0x51 CPU architecture: 8 CPU variant : 0xa CPU part : 0x801 CPU revision : 4 processor : 1 BogoMIPS : 48.00 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid CPU implementer : 0x51 CPU architecture: 8 CPU variant : 0xa CPU part : 0x801 CPU revision : 4 processor : 2 BogoMIPS : 48.00 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid CPU implementer : 0x51 CPU architecture: 8 CPU variant : 0xa CPU part : 0x801 CPU revision : 4 processor : 3 BogoMIPS : 48.00 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid CPU implementer : 0x51 CPU architecture: 8 CPU variant : 0xa CPU part : 0x801 CPU revision : 4 root@UCG-Ultra:~# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 4: 49385470 71684295 74561605 77496134 GIC-0 20 Level arch_timer 6: 0 0 0 0 GIC-0 39 Level arch_mem_timer 8: 0 0 0 0 GIC-0 195 Level edma_txcmpl_4 9: 0 0 0 0 GIC-0 196 Level edma_txcmpl_5 10: 0 0 0 0 GIC-0 197 Level edma_txcmpl_6 11: 0 0 0 0 GIC-0 198 Level edma_txcmpl_7 12: 1301701 0 0 0 GIC-0 199 Level edma_txcmpl_8 13: 16537922 0 0 0 GIC-0 200 Level edma_txcmpl_9 14: 16902391 0 0 0 GIC-0 201 Level edma_txcmpl_10 15: 19093638 0 0 0 GIC-0 202 Level edma_txcmpl_11 16: 218358 0 0 0 GIC-0 203 Level edma_txcmpl_12 17: 14172534 0 0 0 GIC-0 204 Level edma_txcmpl_13 18: 12228644 0 0 0 GIC-0 205 Level edma_txcmpl_14 19: 14848643 0 0 0 GIC-0 206 Level edma_txcmpl_15 20: 141424886 0 0 0 GIC-0 171 Level edma_rxdesc_12 21: 10923594 0 0 0 GIC-0 172 Level edma_rxdesc_13 22: 10953671 0 0 0 GIC-0 173 Level edma_rxdesc_14 23: 13031550 0 0 0 GIC-0 174 Level edma_rxdesc_15 24: 0 0 0 0 GIC-0 223 Level edma_misc 33: 5199506 0 0 0 GIC-0 321 Level bam_dma 34: 929 0 0 0 GIC-0 322 Level msm_serial0 35: 10465714 0 0 0 GIC-0 345 Level mmc0 36: 3 0 0 0 GIC-0 348 Level 7804000.sdhci 37: 40125 0 0 0 GIC-0 324 Level 78b5000.spi 38: 5187812 0 0 0 GIC-0 326 Level 78b7000.spi 39: 9865790 0 0 0 GIC-0 325 Level i2c_qup 45: 0 0 0 0 GIC-0 450 Edge smp2p 46: 0 0 0 0 GIC-0 453 Edge q6v5 wdog 47: 0 0 0 0 GIC-0 352 Edge tsens_interrupt 48: 0 0 0 0 GIC-0 23 Level arm-pmu 49: 0 0 0 0 GIC-0 299 Edge tzerror 50: 360 0 0 0 GIC-0 96 Level xhci-hcd:usb1 51: 0 0 0 0 smp2p 0 Edge q6v5 fatal 52: 0 0 0 0 smp2p 1 Edge q6v5 ready 53: 0 0 0 0 smp2p 2 Edge q6v5 handover 54: 0 0 0 0 smp2p 3 Edge q6v5 stop 55: 0 0 0 0 smp2p 8 Edge q6v5_wcss_userpd1_fatal 56: 0 0 0 0 smp2p 9 Edge q6v5_wcss_userpd1_ready 57: 0 0 0 0 smp2p 12 Edge q6v5_wcss_userpd1_spawn_ack 58: 0 0 0 0 smp2p 11 Edge q6v5_wcss_userpd1_stop_ack 59: 0 0 0 0 msmgpio 52 Edge Reset Button IPI0: 11014796 16872295 17455146 17378253 Rescheduling interrupts IPI1: 54693 31818397 30245960 124867604 Function call interrupts IPI2: 0 0 0 0 CPU stop interrupts IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts IPI4: 0 0 0 0 Timer broadcast interrupts IPI5: 914 0 0 0 IRQ work interrupts IPI6: 0 0 0 0 CPU wake-up interrupts Err: 0 On Tue, Jan 9, 2024 at 3:28 PM Nils Andreas Svee <me@lochnair.net> wrote: > Well my NIC has 4 queues as far as I can tell, so it could likely work, > but as you say it’s like killing a mosquito with a gatling gun. > > Those graphs are sweet though, and it’s been in my backlog for awhile to > do something with Grafana to get something similar, like this one from a > few years ago you’ve seen too: > https://forum.openwrt.org/t/sqm-reporting/59960/96 > > Best Regards, > Nils Andreas Svee > > On Jan 9, 2024, at 18:17, Dave Taht <dave.taht@gmail.com> wrote: > > Principal limitation for libreqos on a small box is has to have > multiple hardware queues and support eBPF. > > Seriously folks, running libreqos at home is *serious overkill*, > although I have to admit the traffic graphs are mesmerizing!!! One of > our ISPs has been setting them to music: > https://www.youtube.com/@trendaltoews7143 > > Herbert has been working on adding all sorts of other analytics to it also. > > On Tue, Jan 9, 2024 at 12:07 PM dave seddon <dave.seddon.ca@gmail.com> > wrote: > > > Nils - I guess you could run LibreQoS on N100? > > On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake < > cake@lists.bufferbloat.net> wrote: > > > On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com> wrote: > > On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake > <cake@lists.bufferbloat.net> wrote: > > Though frankly, I don’t plan on updating the sch_cake and tc binaries when > new firmwares are released anymore, as they don’t publish the GPL archives > on their webpage after the redesign, and they don’t respond to requests for > them either by the looks of the forums. So if it breaks there’s not much I > can do anymore. > > > This irks me enormously. It is the direct outcome of the cambium > elevate lawsuit, where both companies lost, the ISPs lost, open source > practices long established about publishing sources, lost, and the > lawyers went on to other nasty things leaving this trail of awful > precedents in their wake. > > https://www.mtin.net/blog/ubnt-vs-cambium/ > > Wow, hadn’t read about that. They even sued an ISP just for using > Cambium’s software on their hardware? > That is crazy, just evil corporate lawyers doing their thing I guess. > > I do not know what to do about it. It also irks me that as a > contributor to "smart queues" they are not maintaining it well. > > It leaves something to be desired yes, and I would’ve hoped to see CAKE > included too of course, > but even WireGuard is only available in the latest release candidates with > the redesigned web UI, so I’m not holding my breath. > > I still have an EdgeRouter 4 that serves the family farm and one of the > 8-port switches under my desk, if only because I don’t wanna spend money on > replacing them, and they do serve their purpose. > > I’ve since moved though, and now live in an area that has FTTH, so I > needed something beefier to handle CAKE on a 750/750 subscription, because > obviously there’s still bloat even on that ;) > > One of those Chinese boxes with a N100 in it and OpenWrt on top works > wonders :) > > Best Regards, > Nils Andreas Svee > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > > > > > -- > Regards, > Dave Seddon > +1 415 857 5102 > > > > > -- > 40 years of net history, a couple songs: > https://www.youtube.com/watch?v=D9RGX6QFm5E > Dave Täht CSO, LibreQos > > > -- Regards, Dave Seddon +1 415 857 5102 [-- Attachment #1.2: Type: text/html, Size: 15637 bytes --] [-- Attachment #2: image.png --] [-- Type: image/png, Size: 32758 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Cake] Ubiquity (Unifi ) Smart Queues 2024-01-09 17:07 ` dave seddon 2024-01-09 17:17 ` Dave Taht @ 2024-01-09 23:17 ` Nils Andreas Svee 1 sibling, 0 replies; 16+ messages in thread From: Nils Andreas Svee @ 2024-01-09 23:17 UTC (permalink / raw) To: dave seddon; +Cc: Dave Taht, CAKE list [-- Attachment #1: Type: text/plain, Size: 2722 bytes --] I probably could, but it seems *a bit* more complex than I need more my little home network? ;) Best Regards, Nils Andreas Svee > On Jan 9, 2024, at 18:07, dave seddon <dave.seddon.ca@gmail.com> wrote: > > Nils - I guess you could run LibreQoS on N100? > > On Tue, Jan 9, 2024 at 8:57 AM Nils Andreas Svee via Cake <cake@lists.bufferbloat.net <mailto:cake@lists.bufferbloat.net>> wrote: >>> On Jan 9, 2024, at 17:05, Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote: >>> >>> On Tue, Jan 9, 2024 at 10:40 AM Nils Andreas Svee via Cake >>> <cake@lists.bufferbloat.net <mailto:cake@lists.bufferbloat.net>> wrote: >>> >>>> Though frankly, I don’t plan on updating the sch_cake and tc binaries when new firmwares are released anymore, as they don’t publish the GPL archives on their webpage after the redesign, and they don’t respond to requests for them either by the looks of the forums. So if it breaks there’s not much I can do anymore. >>> >>> This irks me enormously. It is the direct outcome of the cambium >>> elevate lawsuit, where both companies lost, the ISPs lost, open source >>> practices long established about publishing sources, lost, and the >>> lawyers went on to other nasty things leaving this trail of awful >>> precedents in their wake. >>> https://www.mtin.net/blog/ubnt-vs-cambium/ >> Wow, hadn’t read about that. They even sued an ISP just for using Cambium’s software on their hardware? >> That is crazy, just evil corporate lawyers doing their thing I guess. >> >>> I do not know what to do about it. It also irks me that as a >>> contributor to "smart queues" they are not maintaining it well. >> It leaves something to be desired yes, and I would’ve hoped to see CAKE included too of course, >> but even WireGuard is only available in the latest release candidates with the redesigned web UI, so I’m not holding my breath. >> >> I still have an EdgeRouter 4 that serves the family farm and one of the 8-port switches under my desk, if only because I don’t wanna spend money on replacing them, and they do serve their purpose. >> >> I’ve since moved though, and now live in an area that has FTTH, so I needed something beefier to handle CAKE on a 750/750 subscription, because obviously there’s still bloat even on that ;) >> >> One of those Chinese boxes with a N100 in it and OpenWrt on top works wonders :) >> >> Best Regards, >> Nils Andreas Svee >> _______________________________________________ >> Cake mailing list >> Cake@lists.bufferbloat.net <mailto:Cake@lists.bufferbloat.net> >> https://lists.bufferbloat.net/listinfo/cake > > > -- > Regards, > Dave Seddon > +1 415 857 5102 [-- Attachment #2: Type: text/html, Size: 9900 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2024-04-29 21:55 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-01-02 18:59 [Cake] Ubiquity (Unifi ) Smart Queues dave seddon 2024-01-02 20:52 ` Sebastian Moeller 2024-01-02 21:15 ` dave seddon 2024-01-02 21:24 ` Sebastian Moeller 2024-01-02 21:57 ` Jonathan Morton 2024-01-03 13:44 ` Pete Heist 2024-01-09 15:39 ` Nils Andreas Svee 2024-01-09 15:59 ` Dave Taht 2024-01-09 16:05 ` Dave Taht 2024-01-09 16:47 ` dave seddon 2024-01-09 16:57 ` Nils Andreas Svee 2024-01-09 17:07 ` dave seddon 2024-01-09 17:17 ` Dave Taht 2024-01-09 23:28 ` Nils Andreas Svee 2024-04-29 21:55 ` dave seddon 2024-01-09 23:17 ` Nils Andreas Svee
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
![[-- Attachment #2: image.png --] [-- Type: image/png, Size: 6574 bytes --]](../../CANypexQ6NpwV0onwFeOugogc+H7yCa03SNOJcC3OyZQGc8KRDQ@mail.gmail.com/2-image.png){kind=link}
![[-- Attachment #2: image.png --] [-- Type: image/png, Size: 32758 bytes --]](../../CANypexTLrLcVSN8sM_bTdsXRyxcspsCSN7h2x--6MxxAsgijBg@mail.gmail.com/2-image.png){kind=link}