[Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote:

Hello,

> > Free has been using SFQ since 2005 (if I remember well).
> > They announced the wide deployment of SFQ in the free.fr newsgroup.
> > Wi-Fi in the free.fr router was not as good though.
> 
> They're working on it. :)

yes indeed.

Switching to softmac approach, so now mac80211 will do rate control
and scheduling (using wake_tx_queue model).

for 5ghz, we use ath10k

> I am very, very happy for y'all. Fiber has always been the sanest
> thing. Is there a SPF+ gpon card yet I can plug into a convention
> open source router yet?

FYI Free.fr uses 10G-EPON, not GPON.

Also most deployments are using an additionnal terminal equipement at
called "ONT" or "ONU" that handle the PON part and exposes an ethernet
port where the operator CPE is plugged. So we are back to the early
days of DSL, where the hardest part (scheduling) is done inside a
black box. That makes it easier to replace the operator CPE with your
own standard ethernet router though.

At least SOCs with integrated PON (supporting all flavours
GPON/EPON/..)  are starting to be deployed. Nothing available in
opensource.

Also note that it's not just kernel drivers, you also need some higher
OAM stack to make that work, and there are a lot of existing
standards, DPOE (EPON), OMCI (GPON)... all with interop challenges.

> > The challenge becomes to keep up with these link rates in software
> > as there is a lot of hardware offloading.

Yes that's our pain point, because that's what the SOCs vendors
deliver and you need to use that because there is no alternative.

It's not entierely the SOCs vendors fault though.

15 years ago, your average SOC's CPU would be something like 200Mhz
MIPS, Linux standard forwarding path (softirq => routing+netfilter =>
qdisc) was too slow for this, too much cache footprint/overhead. So
every vendor started building alternatives forwarding path in their
hardware and never looked back.

Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which
with a non crappy network driver and internal fabric should do be able
to route 1Gbit/s with out-of-the-box kernel forwarding.

But that's too late. SOC vendors compete against each others, and the
big telcos need a way to tell which SOC is better to make a buying
decision. So synthetic benchmarks have become the norm, and since
everybody was able to do fill their pipe with 1500 bytes packets,
benchmarks have moved to unrealistic 64 bytes packets (so called
wirespeed)

If you don't have hardware acceleration for forwarding, you don't
exist in those benchmarks and will not sell your chipset. Also they
invested so much in their alternative network stack that it's
difficult to stop (huge R&D teams). That being said, they do have a
point, when speed go above 1Gbit/s, the kernel becomes the bottleneck.

For Free.fr 10Gbit/s offer, we had to develop an alternative
(software) forwarding path using polling mode model (DPDK style),
otherwise our albeit powerful ARM Cortex A72 at 2Ghz could not forward
more than 2Gbit/s.

And going multicore/RSS does not fly when the test case is single
stream TCP session, which is what most speedtest application do (ookla
only recently added multi-connections test).

-- 
Maxime