[Bloat] UniFi Dream Machine Pro

[Bloat] UniFi Dream Machine Pro

[Dave Taht]

This review, highly recommending this router on the high end

https://www.increasebroadbandspeed.co.uk/best-router-2020

also states that the sqm implementation has been dumbed down
significantly and can only shape 800Mbit inbound. Long ago we did a
backport of cake to the other ubnt routers mentioned in the review,
has anyone tackled this one?


-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] UniFi Dream Machine Pro

[Stuart Cheshire]

On 20 Jan 2021, at 07:55, Dave Taht <dave.taht@gmail.com> wrote:

> This review, highly recommending this router on the high end
> 
> https://www.increasebroadbandspeed.co.uk/best-router-2020
> 
> also states that the sqm implementation has been dumbed down significantly and can only shape 800Mbit inbound. Long ago we did a backport of cake to the other ubnt routers mentioned in the review, has anyone tackled this one?

According to the UniFi Dream Machine Pro data sheet, it has a 1.7 GHz quad-core ARM Cortex-A57 processor and achieves the following throughput numbers (downlink direction):

8.0 Gb/s with Deep Packet Inspection
3.5 Gb/s with DPI + Intrusion Detection
0.8 Gb/s with IPsec VPN

<https://dl.ubnt.com/ds/udm-pro>

Is implementing CoDel queueing really 10x more burden than running “Ubiquiti’s proprietary Deep Packet Inspection (DPI) engine”? Is CoDel 4x more burden than Ubiquiti’s IDS (Intrusion Detection System) and IPS (Intrusion Prevention System)?

Is CoDel really the same per-packet cost as doing full IPsec VPN decryption on every packet? I realize the IPsec VPN decryption probably has some assist from crypto-specific ARM instructions or hardware, but even so, crypto operations are generally considered relatively expensive. If this device can do 800 Mb/s throughput doing IPsec VPN decryption for every packet, it feels like it ought to be able to do a lot better than that just doing CoDel queueing calculations for every packet.

Is this just a software polish issue, that could be remedied by doing some performance optimization on the CoDel code?

It’s also possible that the information in the review might simply be wrong -- it’s hard to measure throughput numbers in excess of 1 Gb/s unless you have both a client and a server connected faster than that in order to run the test. In other words, gigabit Ethernet is out, so both client and server would have to be connected via the 10 Gb/s SFP+ ports (of which the UDM-PRO has just two -- one in the upstream direction, and one in the downstream direction). Speaking for myself personally, I don’t have any devices with 10 Gb/s capability, and my Internet connection isn’t above 1 Gb/s either, so as long as it can get reasonably close to 1 Gb/s that’s more than I need (or could use) right now.

Stuart Cheshire

Re: [Bloat] UniFi Dream Machine Pro

[Sebastian Moeller]

Maybe I can help out here.

I believe you are talking about two different issues here. Dave notes that "traffic shaping" performance seems to top out at ~800 Mbps and you interpret that to mean CoDel being the rate limiting step here. The number seems a bit low though, I have seen unidirectional traffic shaping up to ~940 Mbps on a 1.6 GHz dual core arm a9 Marvell Armada 380/385 SoC (under bidirectional stress however it topped out at ~500/500 Mbps with cake as shaper). Raspberry pi 4Bs are reported (on the openwrt forum) to allow bidirectional traffic shaping at > 900Mbps.
	I am confident that this device will happily run CoDel and even fq_codel close to line-rate, as codel/fq_codel have a relative modest processing cost. Traffic shaping however, at least as implemented in Linux' TBF, HTB, HFSC, and cake qdiscs is a different beast and quite CPU intensive. That might actually top out at ~800 Mbps on that device. (In my understanding the issue with shaping is more of a latency than throughput issue, if the traffic shaper is not to push large bursts of data into the next layer, it basically wants to yield CPU only until shortly before the lower level drivers run dry to then only admit a small batch of packets, and at increasingly faster links the serialization time of an individual packet becomes shorter and shorter increasing the CPU "neediness" of traffic shapers).  

	I also guess that the Ubiquity numbers assume the use of some offload engines that might or might not be supported by main line linux, but that is not backed by factual knowledge.

Hope that helps
	Sebastian


> On Jan 22, 2021, at 20:42, Stuart Cheshire via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> On 20 Jan 2021, at 07:55, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> This review, highly recommending this router on the high end
>> 
>> https://www.increasebroadbandspeed.co.uk/best-router-2020
>> 
>> also states that the sqm implementation has been dumbed down significantly and can only shape 800Mbit inbound. Long ago we did a backport of cake to the other ubnt routers mentioned in the review, has anyone tackled this one?
> 
> According to the UniFi Dream Machine Pro data sheet, it has a 1.7 GHz quad-core ARM Cortex-A57 processor and achieves the following throughput numbers (downlink direction):
> 
> 8.0 Gb/s with Deep Packet Inspection
> 3.5 Gb/s with DPI + Intrusion Detection
> 0.8 Gb/s with IPsec VPN
> 
> <https://dl.ubnt.com/ds/udm-pro>
> 
> Is implementing CoDel queueing really 10x more burden than running “Ubiquiti’s proprietary Deep Packet Inspection (DPI) engine”? Is CoDel 4x more burden than Ubiquiti’s IDS (Intrusion Detection System) and IPS (Intrusion Prevention System)?
> 
> Is CoDel really the same per-packet cost as doing full IPsec VPN decryption on every packet? I realize the IPsec VPN decryption probably has some assist from crypto-specific ARM instructions or hardware, but even so, crypto operations are generally considered relatively expensive. If this device can do 800 Mb/s throughput doing IPsec VPN decryption for every packet, it feels like it ought to be able to do a lot better than that just doing CoDel queueing calculations for every packet.
> 
> Is this just a software polish issue, that could be remedied by doing some performance optimization on the CoDel code?
> 
> It’s also possible that the information in the review might simply be wrong -- it’s hard to measure throughput numbers in excess of 1 Gb/s unless you have both a client and a server connected faster than that in order to run the test. In other words, gigabit Ethernet is out, so both client and server would have to be connected via the 10 Gb/s SFP+ ports (of which the UDM-PRO has just two -- one in the upstream direction, and one in the downstream direction). Speaking for myself personally, I don’t have any devices with 10 Gb/s capability, and my Internet connection isn’t above 1 Gb/s either, so as long as it can get reasonably close to 1 Gb/s that’s more than I need (or could use) right now.
> 
> Stuart Cheshire
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] UniFi Dream Machine Pro

[Toke Høiland-Jørgensen]

Sebastian Moeller <moeller0@gmx.de> writes:

> I am confident that this device will happily run CoDel and even
> fq_codel close to line-rate, as codel/fq_codel have a relative modest
> processing cost.

Yeah, the overhead of CoDel itself (and even FQ-CoDel) is basically nil
(as in, we have not been able to measure it), when otherwise doing
forwarding using the regular Linux stack.

As Sebastian says, the source of lower performance when using SQM on
some boxes is the traffic shaper, and sometimes the lack of hardware
offloads.

-Toke

Re: [Bloat] UniFi Dream Machine Pro

[Jonathan Morton]

> On 22 Jan, 2021, at 11:09 pm, Toke Høiland-Jørgensen via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> As Sebastian says, the source of lower performance when using SQM on
> some boxes is the traffic shaper, and sometimes the lack of hardware
> offloads.

I have a strong suspicion that on some hardware, the offload engine & switch is connected to the SoC through a link that is much slower than the Ethernet ports exposed to the outside.  As long as traffic stays within the engine, it can run at line rate, but engaging rich software measures requires stuffing it all through the narrower link.  This is independent of the CPU's capabilities and is purely an I/O bottleneck.

In this particular case, I believe the router portion of the Dream Machine is natively a Gigabit Ethernet device, for which good IPsec and SQM performance at 800Mbps is reasonably expected.  The pair of 10G ports are part of the switch portion, and thus intended to support LAN rather than WAN traffic.  Think of it as equivalent to attaching a Raspberry Pi 4 (which has native GigE) to a switch with a pair of 10G "uplink" ports for daisy-chaining to other switches.

 - Jonathan Morton

Re: [Bloat] UniFi Dream Machine Pro

[Dave Taht]

On Fri, Jan 22, 2021 at 11:43 AM Stuart Cheshire <cheshire@apple.com> wrote:
>
> On 20 Jan 2021, at 07:55, Dave Taht <dave.taht@gmail.com> wrote:
>
> > This review, highly recommending this router on the high end
> >
> > https://www.increasebroadbandspeed.co.uk/best-router-2020
> >
> > also states that the sqm implementation has been dumbed down significantly and can only shape 800Mbit inbound. Long ago we did a backport of cake to the other ubnt routers mentioned in the review, has anyone tackled this one?

It's nice to see the "godfadder" of our effort back again here. I do
re-read periodically http://www.stuartcheshire.org/rants/latency.html

At the price of perhaps over-lecturing for a wider audience.

> According to the UniFi Dream Machine Pro data sheet, it has a 1.7 GHz quad-core ARM Cortex-A57 processor and achieves the following throughput numbers (downlink direction):

>
> 8.0 Gb/s with Deep Packet Inspection
I'm always very dubious of these kind of numbers against anything but
single large, bulk flows. Also if the
fast path is not entirely offloaded, performance goes to hell.

> 3.5 Gb/s with DPI + Intrusion Detection
> 0.8 Gb/s with IPsec VPN

Especially here, also. I should also note that the rapidly deploying
wireguard vpn outperforms ipsec
in just about every way... in software.

>
> <https://dl.ubnt.com/ds/udm-pro>
>
> Is implementing CoDel queueing really 10x more burden than running “Ubiquiti’s proprietary Deep Packet Inspection (DPI) engine”? Is CoDel 4x more burden than Ubiquiti’s IDS (Intrusion Detection System) and IPS (Intrusion Prevention System)?

These questions, given that the actual fq-codel overhead is nearly
immeasurable, and the code complexity much less than these, are the
makings of a very good rant targetted at a hw offload maker. :)

hashing is generally "free" and in hw, selecting a different queue can
be done with single indirect

Cake has a lot of ideas that would benefit from actual hw offloads. a
4 or 8 way associative cache is a common IP hw block....

> Is CoDel really the same per-packet cost as doing full IPsec VPN decryption on every packet? I realize the IPsec VPN decryption probably has

No.

>some assist from crypto-specific ARM instructions or hardware, but even so, crypto operations are generally considered relatively expensive. If this device can do 800 Mb/s throughput doing IPsec VPN decryption for every packet, it feels like it ought to be able to do a lot better than that just doing CoDel queueing calculations for every packet.

yep.

the only even semi-costly codel function is an invsqrt which can be
implemented in 3k gates or so in hw. In software the
newton approximation is nearly immeasurable, and accurate enough. (we
went to great lengths to make it
accurate in cake to no observable effect)

codel is not O(1) A nice thing about fq is that you can be codeling in
parallel, or if you are acting
on a single queue at a time, short circuit the overload section of
codel to give up and deliver a packet
if you cannot meet the deadline... or... using a very small fifo queue
(say 3k bytes at a gbit), the odds are extremely
good (millions? ... A lot. I worked it out once with various
assumptions...) that no matter how many packets you need to drop
at once, you can still run at line rate at a reasonable clock. bql
manages this short fifo in linux,
but it tends to be much larger and inflated by tso offloads.

You really don't need to drop or mark a lot of packets to achieve good
congestion control at high rates. But you know that. :)

Most "hw" offloads are actually offloads to a specialized cpu and thus
O(1) or not isn't much of a problem there.

> Is this just a software polish issue, that could be remedied by doing some performance optimization on the CoDel code?

Don't know how to make it faster. The linux version is about as
optimized as we know how. A p4 implementation exists.

As everyone points out later on this thread, it's the software
*shaper* (on inbound especially) that is the real burden. TB has been
offloaded to hw. The QCA offloaded version has both the tb and
fq_codel in there.

also hw shaping outbound is vastly cheaper with a programmable
completion interrrupt. tell 1Gbit hardware to
interrupt at half the rate, bang, it's 500Mbit. (this is implemented
in several intel ethernet cards)

inbound shaping in sw is another one of the it's the latency stupid
things. It's not so much the clock rate, but how fast
the cpu can reschedule the thread, a number that doesn't scale much
with clock, but with cache and pipeline depth.

One reason why I adore the mill cpu design is that it can context
switch in 5 clocks, where x86 takes 1000....


> It’s also possible that the information in the review might simply be wrong -- it’s hard to measure throughput numbers in excess of 1 Gb/s unless you have both a client and a server connected faster than that in order to run the test. In other words, gigabit Ethernet is out, so both client and server would have to be connected via the 10 Gb/s SFP+ ports (of which the UDM-PRO has just two -- one in the upstream direction, and one in the downstream direction). Speaking for myself personally, I don’t have any devices with 10 Gb/s capability, and my Internet connection isn’t above 1 Gb/s either, so as long as it can get reasonably close to 1 Gb/s that’s more than I need (or could use) right now.

As most 1Gbit ISP links are still quite overbuffered (over 120ms was
what I'd measured with comcast, 60ms on sonic fiber, both a few years
back), vs a total induced latency of *0-5ms* with sqm at 800mbit, it
generally seems to me that inbound shaping to something close to a
gbit is a win for videoconferencing, gaming, vr, jacktrip and other
latency sensitive traffic.

On a 35Mbit upload, fq_codel or cake are *loafing*. If we were to get
around to doing a backport of cake to this device, I'd probably
go with htb+fq_codel on the download and cake on the upload, where the
ack-filtering and per host/per flow fq of cake would be ideal.

(this, btw, is what I do presently)

ack-filtering at these asymmetries is a pretty big win for retaining a
high download speed with competing upload traffic.
https://blog.cerowrt.org/post/ack_filtering/ you cannot do anything
even close to a steady gbit down with competing uplink traffic on the
cable modems I've tested to date.

> Stuart Cheshire
>


-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] UniFi Dream Machine Pro

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Fri, 22 Jan 2021, Stuart Cheshire via Bloat wrote:

> Is implementing CoDel queueing really 10x more burden than running 
> “Ubiquiti’s proprietary Deep Packet Inspection (DPI) engine”? Is CoDel 
> 4x more burden than Ubiquiti’s IDS (Intrusion Detection System) and IPS 
> (Intrusion Prevention System)?

No, it isn't but all the other functions have hw offloads and the CoDel 
numbers shown are when you turn hw offloads off, basically only running it 
in CPU forwarding mode. That's when you get those kinds of numbers (~800 
megabit/s).

When enabling SQM on their USG3 you get ~100 megabit/s of throughput, 
because it has a very slow CPU (but has plenty of offloads, so full gig 
with offloads enabled works well, but then you don't get any SQM/DPI).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] UniFi Dream Machine Pro

[Dave Taht]

On Wed, Jan 20, 2021 at 7:55 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> This review, highly recommending this router on the high end
>
> https://www.increasebroadbandspeed.co.uk/best-router-2020
>
> also states that the sqm implementation has been dumbed down
> significantly and can only shape 800Mbit inbound. Long ago we did a
> backport of cake to the other ubnt routers mentioned in the review,
> has anyone tackled this one?

I take it the answer to this question is... no?

For the record I helped do the original fq_codel port to the
edgerouter over a weekend back in 2013. the ubnt
community jumped all over it, polished up the configuration language
to use it, leveraged the sqm
spec and demanded the result in their next formal release... which
ubnt did. It's been an ever
increasing "selling feature" for them ever since.

There were like 800 messages on that wonderful 2013 thread, can't find it now.

I don't know how closed this hw is, compared to the older edgerouters.
I can go hit the ubnt
forums to find out. Back then the devkit was freely available.

I wouldn't mind getting in there to finally fix frame compensation or
integrate cake.


>
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] UniFi Dream Machine Pro

[Nils Andreas Svee]

Was asked to chip in here, so I'll provide what little I know.

I never looked too hard at the new ARM-based units, because Ubiquiti's been refusing to release the GPL archives for them [1].

But AFAIK they're based on Buildroot (there's some info about that in the same thread [2]), and people have been cross-compiling userspace stuff for them, tusc from the WireGuard thread did just that with Boringtun to at least have WireGuard running in userspace [3].

It's running kernel 4.1.37 (heavily modified I assume, as they always are). And in [2] there's linked a text file with a information dump from on of those boxes, kernel config included. I assume though, that they've backported so much it's not possible to use the upstream kernel tree to compile against.

[1]: https://github.com/WireGuard/wireguard-vyatta-ubnt/issues/32#issuecomment-652755919
[2]: https://github.com/WireGuard/wireguard-vyatta-ubnt/issues/32#issuecomment-643757919
[3]: https://github.com/tusc/wireguard/blob/master/compiling.md

Best Regards
Nils

Re: [Bloat] UniFi Dream Machine Pro

[Nils Andreas Svee]

> For the record I helped do the original fq_codel port to the
> edgerouter over a weekend back in 2013. the ubnt
> community jumped all over it, polished up the configuration language
> to use it, leveraged the sqm
> spec and demanded the result in their next formal release... which
> ubnt did. It's been an ever
> increasing "selling feature" for them ever since.
>
> There were like 800 messages on that wonderful 2013 thread, can't find it now.
>
> I don't know how closed this hw is, compared to the older edgerouters.
> I can go hit the ubnt
> forums to find out. Back then the devkit was freely available.
>
> I wouldn't mind getting in there to finally fix frame compensation or
> integrate cake.

Shame they haven't been as interested in implementing CAKE in their firmware. We haven't reached fq_codel levels of activity for CAKE, but we've still got well over 500 replies to that thread. Personally after getting CAKE to work on those boxes, I have never looked back. It's just that good.

Last time I saw someone ask about getting CAKE implemented in the stock FW, UBNT-staff just sent them my way [1]. 

[1]: https://community.ui.com/questions/Cake-SQM-AQM/8da70eda-de9c-46e1-b439-056f1d035350#answer/92c0a3b6-23b8-4140-92f1-4410d5daeb34

Best Regards
Nils

Re: [Bloat] UniFi Dream Machine Pro

[Dave Taht]

On Sat, Jan 23, 2021 at 5:08 PM Nils Andreas Svee <me@lochnair.net> wrote:
>
> > For the record I helped do the original fq_codel port to the
> > edgerouter over a weekend back in 2013. the ubnt
> > community jumped all over it, polished up the configuration language
> > to use it, leveraged the sqm
> > spec and demanded the result in their next formal release... which
> > ubnt did. It's been an ever
> > increasing "selling feature" for them ever since.
> >
> > There were like 800 messages on that wonderful 2013 thread, can't find it now.
> >
> > I don't know how closed this hw is, compared to the older edgerouters.
> > I can go hit the ubnt
> > forums to find out. Back then the devkit was freely available.
> >
> > I wouldn't mind getting in there to finally fix frame compensation or
> > integrate cake.
>
> Shame they haven't been as interested in implementing CAKE in their firmware. We haven't reached fq_codel levels of activity for CAKE, but we've still got well over 500 replies to that thread. Personally after getting CAKE to work on those boxes, I have never looked back. It's just that good.
>
> Last time I saw someone ask about getting CAKE implemented in the stock FW, UBNT-staff just sent them my way [1].
>
> [1]: https://community.ui.com/questions/Cake-SQM-AQM/8da70eda-de9c-46e1-b439-056f1d035350#answer/92c0a3b6-23b8-4140-92f1-4410d5daeb34

The contacts that I had that supported the original effort (notably
stig) have long since left ubnt. But we could make some noise on the
forums.

you are a hero, btw, for getting the cake port going on that hw. thx
so much for that!

>
> Best Regards
> Nils



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] UniFi Dream Machine Pro

[Nils Andreas Svee]

Didn't really feel the same after he and the other guys I don't recall the names of left.

I thought I'd get something *not* Ubiquiti when the time to replace my Edgerouter Lite came, but now I have an Edgerouter 4 on the way, so I guess it's time to make noise on the forums again. Doesn't hurt to be able to use the old hardware to test things on either.

I'm tempted to say you exaggerating, but hey, always nice when your efforts are appreciated.
Glad something good came out of my being unemployed at the time.

Best Regards
Nils