[Ecn-sane] is FQ actually widely deployed?

[Ecn-sane] is FQ actually widely deployed?

[Mikael Abrahamsson]

On Fri, 19 Jul 2019, Dave Taht wrote:

> In particular conflating "low latency" really confounds the subject
> matter, and has for years. FQ gives "low latency" for the vast
> majority of flows running below their fair share. L4S promises "low

My observations from the business is that FQ just isn't a thing, in 
reality.

I run CAKE myself here on my OpenWrt box and it works great, but I have 
yet to find a commercially available box used by ISPs or something that is 
a big-seller in electronics stores that use FQ_anything, or is even flow 
aware when it comes to forwarding.

I have heard nothing about FQ being implemented in packet accelerators. I 
do hear about people wanting to turn on things that control delay/buffer 
fill, but this is still single queue with no flow-aware anything.

Do we have numbers on how much FQ is actually out there? If we don't, can 
we measure it? Anyone know of devices shipping or being designed that does 
FQ of some kind?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] is FQ actually widely deployed?

[Jonathan Morton]

> On 22 Jul, 2019, at 10:14 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> Do we have numbers on how much FQ is actually out there? If we don't, can we measure it? Anyone know of devices shipping or being designed that does FQ of some kind?

Linux and OSX end hosts now routinely run fq_codel by default.  That's an awfully large installed base, both of FQ and of Codel.  It is perhaps worth observing that these are rarely bottlenecks for Internet paths, though they might be for LAN paths.

The IQrouter is probably the best example of a commercial middlebox that does FQ, in this case using Cake.  I hear that it is now being sold in re-branded form to certain large ISPs, but I could be wrong about that.  I've CC'd Jonathan Foulkes for comment.

There's also a large French ISP which has done fq_codel on its last mile for quite some time.

 - Jonathan Morton

Re: [Ecn-sane] is FQ actually widely deployed?

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 22 Jul, 2019, at 10:14 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> Do we have numbers on how much FQ is actually out there? If we don't,
>> can we measure it? Anyone know of devices shipping or being designed
>> that does FQ of some kind?
>
> Linux and OSX end hosts now routinely run fq_codel by default. That's
> an awfully large installed base, both of FQ and of Codel. It is
> perhaps worth observing that these are rarely bottlenecks for Internet
> paths, though they might be for LAN paths.
>
> The IQrouter is probably the best example of a commercial middlebox
> that does FQ, in this case using Cake. I hear that it is now being
> sold in re-branded form to certain large ISPs, but I could be wrong
> about that. I've CC'd Jonathan Foulkes for comment.
>
> There's also a large French ISP which has done fq_codel on its last
> mile for quite some time.

I am aware of at least two Danish and one Norwegian ISP that have
deployed FQ-CoDel-based shapers either as their customer shapers in the
backend or on CPEs.

Anything derived from OpenWrt will also use FQ-CoDel on both WiFi and
wired links. Not sure how common those are, though...

-Toke

Re: [Ecn-sane] is FQ actually widely deployed?

[Pete Heist]

> On Jul 22, 2019, at 1:05 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Jonathan Morton <chromatix99@gmail.com> writes:
> 
>>> On 22 Jul, 2019, at 10:14 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>> 
>>> Do we have numbers on how much FQ is actually out there? If we don't,
>>> can we measure it? Anyone know of devices shipping or being designed
>>> that does FQ of some kind?
>> 
>> There's also a large French ISP which has done fq_codel on its last
>> mile for quite some time.
> 
> I am aware of at least two Danish and one Norwegian ISP that have
> deployed FQ-CoDel-based shapers either as their customer shapers in the
> backend or on CPEs.

In Czech, I’m aware at least of two WISPs that use it. It’s a good bet there
are more, as a company has popped up specifically targeting WISPs which
uses passive monitoring and fq_codel to measure and improve customer
qoe. It doesn’t take too much searching to find them.

Re: [Ecn-sane] is FQ actually widely deployed?

[Mikael Abrahamsson]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

On Mon, 22 Jul 2019, Toke Høiland-Jørgensen wrote:

> Anything derived from OpenWrt will also use FQ-CoDel on both WiFi and 
> wired links. Not sure how common those are, though...

I have OpenWrt derived devices that use hw acceleration and they are not 
using FQ_CODEL. So OpenWrt in itself doesn't mean FQ_CODEL is used.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] is FQ actually widely deployed?

[Sebastian Moeller]

Hi Mikael,


> On Jul 23, 2019, at 07:32, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Mon, 22 Jul 2019, Toke Høiland-Jørgensen wrote:
> 
>> Anything derived from OpenWrt will also use FQ-CoDel on both WiFi and wired links. Not sure how common those are, though...
> 
> I have OpenWrt derived devices that use hw acceleration

	This is my min gripe with accelerators and off-load features, typically they side-step the linux kernel network stack and offer far less features than that stack (this is how they speed up things). IMHO these are crutches and at least up to 1Gbps links reasonable priced x86/x64 hardware can do routing with all bells and whistles (heck mvebu plattforms also come close). 

> and they are not using FQ_CODEL. So OpenWrt in itself doesn't mean FQ_CODEL is used.

	By default OpenWrt does not use hardware offloads (and IIRC it only supports hardware acceleration on mediatek SoCs) but defaults to fq_codel, so usage of OpenWrt should still be a strong indicator of fq_codel being used....

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se_______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane

Re: [Ecn-sane] is FQ actually widely deployed?

[Jonathan Morton]

> On 23 Jul, 2019, at 1:57 am, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> This is my min gripe with accelerators and off-load features, typically they side-step the linux kernel network stack and offer far less features than that stack (this is how they speed up things).

May I introduce you to https://tools.ietf.org/html/draft-morton-tsvwg-cheap-nasty-queueing-00 ?  I'm also in the process of expanding the LFQ draft.  I think they show that HW acceleration *shouldn't* mean just dumb FIFOs.

> IMHO these are crutches and at least up to 1Gbps links reasonable priced x86/x64 hardware can do routing with all bells and whistles

I'd actually be rather interested to see what a Raspberry Pi 4 with a GigE USB3 dongle tacked on can cope with, now that it has native GigE through a direct PHY connection instead of via a USB2 bottleneck.  That would be in the rough price range of a basic wifi router, and I've already got a Pi 2B running as an SCE endpoint.

Any idea which shops I should look in while I'm in Montreal?

 - Jonathan Morton

Re: [Ecn-sane] is FQ actually widely deployed?

[Jonathan Foulkes]

> The IQrouter is probably the best example of a commercial middlebox that does FQ, in this case using Cake.  I hear that it is now being sold in re-branded form to certain large ISPs, but I could be wrong about that.  I've CC'd Jonathan Foulkes for comment.

I wish I could get some ‘large ISPs’ to pay attention, as they have some of the worst issues. But our ISP base is mostly smaller, rural ISPs with the usual challenges using legacy copper or older DOCSIS deployments. But I have also seen benefits on many a higher-capacity, modern infrastructure. Not sure how they allow bloat on some of the setups, but they do, and a CPE with a decent AQM (like Cake) makes a measurable difference.

Since this is also going to the ecn-sane list, let me say that having an AQM that is ECN-aware (like Cake with the ECN flags enabled) is major win for user-experience, as when combined with modern OSs that respect ECN, the immediate reaction to the congestion signaling is wonderful. Also nice that tests like the DSLreports.com speedtest, who grade packet-loss (the usual congestion signaling) as ‘Quality’ now consistently grade lines as an A in that metric with ECN signaling. 

Regards,

Jonathan Foulkes
CEO - Evenroute.com

> On Jul 22, 2019, at 11:20 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 22 Jul, 2019, at 10:14 am, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> Do we have numbers on how much FQ is actually out there? If we don't, can we measure it? Anyone know of devices shipping or being designed that does FQ of some kind?
> 
> Linux and OSX end hosts now routinely run fq_codel by default.  That's an awfully large installed base, both of FQ and of Codel.  It is perhaps worth observing that these are rarely bottlenecks for Internet paths, though they might be for LAN paths.
> 
> The IQrouter is probably the best example of a commercial middlebox that does FQ, in this case using Cake.  I hear that it is now being sold in re-branded form to certain large ISPs, but I could be wrong about that.  I've CC'd Jonathan Foulkes for comment.
> 
> There's also a large French ISP which has done fq_codel on its last mile for quite some time.
> 
> - Jonathan Morton

Re: [Ecn-sane] is FQ actually widely deployed?

[Jonathan Morton]

> On 22 Jul, 2019, at 6:14 pm, Jonathan Foulkes <jfoulkes@evenroute.com> wrote:
> 
>> The IQrouter is probably the best example of a commercial middlebox that does FQ, in this case using Cake.  I hear that it is now being sold in re-branded form to certain large ISPs, but I could be wrong about that.  I've CC'd Jonathan Foulkes for comment.
> 
> I wish I could get some ‘large ISPs’ to pay attention, as they have some of the worst issues. But our ISP base is mostly smaller, rural ISPs with the usual challenges using legacy copper or older DOCSIS deployments. But I have also seen benefits on many a higher-capacity, modern infrastructure. Not sure how they allow bloat on some of the setups, but they do, and a CPE with a decent AQM (like Cake) makes a measurable difference.
> 
> Since this is also going to the ecn-sane list, let me say that having an AQM that is ECN-aware (like Cake with the ECN flags enabled) is major win for user-experience, as when combined with modern OSs that respect ECN, the immediate reaction to the congestion signaling is wonderful. Also nice that tests like the DSLreports.com speedtest, who grade packet-loss (the usual congestion signaling) as ‘Quality’ now consistently grade lines as an A in that metric with ECN signaling. 

Indeed, I brought my IQrouter along to Montreal, so Pete, Rodney and I are using it in our AirBNB as a substitute for the rather decrepit D-Link router we found already installed.  The latter was using 2.4GHz only with 40MHz channels, in a dense urban area, with predictable effect.  The ISP appears to be Comcast…

…and the IQrouter works very well, even though I haven't bothered changing it from the 10M/4M settings I use at home.

 - Jonathan Morton

Re: [Ecn-sane] is FQ actually widely deployed?

[Dave Taht]

On Mon, Jul 22, 2019 at 7:14 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Fri, 19 Jul 2019, Dave Taht wrote:
>
> > In particular conflating "low latency" really confounds the subject
> > matter, and has for years. FQ gives "low latency" for the vast
> > majority of flows running below their fair share. L4S promises "low
>
> My observations from the business is that FQ just isn't a thing, in
> reality.

It's not an easily discussed "thing". As one example of many, if there is
anybody out there in https://common.net/ 's service range that would
like to do some benchmarking...

FQ has always been part of meraki's products and ubnt's airmax. Meraki
does it a bit weirdly, doing sfq at a low level and codel via click.

> I run CAKE myself here on my OpenWrt box and it works great, but I have
> yet to find a commercially available box used by ISPs or something that is

It is certainly a sadness that more ISPs are not shipping gear that
does this at this point.
There are several I know of in addition to common and free.fr but
can't talk about it.

The biggest penetration is on QCA wifi'd devices where no
configuration is required.

> a big-seller in electronics stores that use FQ_anything, or is even flow
> aware when it comes to forwarding.

Hmm? Nearly everything derived from openwrt commercially has it. More
generally, debian linux derived
Google wifi, chromebooks, eero have publically disclosed their usage.

I have not purchased any newer routers in years, but I recently
configged a new synology
router for a friend, and it had "something" that did the right things,
packet captures were interleaved...


>
> I have heard nothing about FQ being implemented in packet accelerators. I
> do hear about people wanting to turn on things that control delay/buffer
> fill, but this is still single queue with no flow-aware anything.

This seems true.

> Do we have numbers on how much FQ is actually out there? If we don't, can
> we measure it? Anyone know of devices shipping or being designed that does
> FQ of some kind?

see above?

> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] is FQ actually widely deployed?

[Mikael Abrahamsson]

On Mon, 22 Jul 2019, Dave Taht wrote:

> Hmm? Nearly everything derived from openwrt commercially has it. More 
> generally, debian linux derived Google wifi, chromebooks, eero have 
> publically disclosed their usage.

For CPU based forwarding platforms, perhaps (and these are not the common 
ones). But one of the few OpenWrt hw acceleration platforms (MT7621) then 
FQ is not done on the hw accelerated flows afaik.

HW acceleration typically means FQ goes out the window.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] is FQ actually widely deployed?

[Sebastian Moeller]

> On Jul 23, 2019, at 07:34, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Mon, 22 Jul 2019, Dave Taht wrote:
> 
>> Hmm? Nearly everything derived from openwrt commercially has it. More generally, debian linux derived Google wifi, chromebooks, eero have publically disclosed their usage.
> 
> For CPU based forwarding platforms, perhaps (and these are not the common ones). But one of the few OpenWrt hw acceleration platforms (MT7621) then FQ is not done on the hw accelerated flows afaik.
> 
> HW acceleration typically means FQ goes out the window.

	That, and most other features the linux network stack offers (it is easier ot be fast if you do less ;) ) I believe users need to actively enable HW offloading, so shpould be in control whether they want to trade in a ship-load of features for alloing their router to punch well above its weight...

Best Regards
	Sebastian


> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane

Re: [Ecn-sane] is FQ actually widely deployed?

[Mikael Abrahamsson]

On Tue, 23 Jul 2019, Sebastian Moeller wrote:

> 	That, and most other features the linux network stack offers (it 
> is easier ot be fast if you do less ;) ) I believe users need to 
> actively enable HW offloading, so shpould be in control whether they 
> want to trade in a ship-load of features for alloing their router to 
> punch well above its weight...

On most HGW SOCs today the choice between HW offload on/off is 200 
megabit/s unidirectional with large packets, or very close to wirespeed.

Anyhow, my statement is that of the users today on the Internet, their 
congestion point will not have FQ enabled, and this for 99% of users. I 
don't see any trend that this is on the verge of drastically increasing 
either.

The trend I am seeing is that delay control is being deployed by means of 
RED, buffer size caps (basically implementing adaptive buffers to only 
provide 10ms buffering until tail drop), PIE/CODEL or similar.

I'd be happy to be proven wrong.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] is FQ actually widely deployed?

[Dave Taht]

On Mon, Jul 22, 2019 at 11:21 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Tue, 23 Jul 2019, Sebastian Moeller wrote:
>
> >       That, and most other features the linux network stack offers (it
> > is easier ot be fast if you do less ;) ) I believe users need to
> > actively enable HW offloading, so shpould be in control whether they
> > want to trade in a ship-load of features for alloing their router to
> > punch well above its weight...
>
> On most HGW SOCs today the choice between HW offload on/off is 200
> megabit/s unidirectional with large packets, or very close to wirespeed.
>
> Anyhow, my statement is that of the users today on the Internet, their
> congestion point will not have FQ enabled, and this for 99% of users. I
> don't see any trend that this is on the verge of drastically increasing
> either.
>
> The trend I am seeing is that delay control is being deployed by means of
> RED, buffer size caps (basically implementing adaptive buffers to only
> provide 10ms buffering until tail drop), PIE/CODEL or similar.

I don't see a trend on "RED" - I see existing implementations.
Certainly most hardware offloads have fairly short buffer sizes. I
have been hoping for a pie implementation in non-cable-hw for years
now.

The title of this mail could just as well been, "have any hw offload
makers been paying the
slightest bit of attention to fixing bufferbloat with *any* aqm or fq
tech?" - with an answer of no.

On 10gige+ ethernet cards, 64 hw queues is the norm. That's fq.
There's some support for aqm
on some. I think the work on timerwheel paced stuff has high potential.

In the switch chip market we see big buffered switches and small
buffered ones, targetted at different markets. There's work in p4
that's promising on the switch front.

We've hit it out of the park on wifi, with atf and fq_codel. Very
visible "trend" there.

What happens in the offloaded word on small routers like the
edgerouter series is to do "qos" at a lower
than line rate, offloads are disabled and it's done in software.



The tit
> I'd be happy to be proven wrong.
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740