From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ns.iliad.fr (ns.iliad.fr [212.27.33.1]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7F35E3B29E for ; Thu, 23 Apr 2020 05:29:11 -0400 (EDT) Received: from ns.iliad.fr (localhost [127.0.0.1]) by ns.iliad.fr (Postfix) with ESMTP id 73052202C7; Thu, 23 Apr 2020 11:29:10 +0200 (CEST) Received: from sakura (freebox.vlq16.iliad.fr [213.36.7.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ns.iliad.fr (Postfix) with ESMTPS id 64E592022C; Thu, 23 Apr 2020 11:29:10 +0200 (CEST) Date: Thu, 23 Apr 2020 11:29:09 +0200 From: Maxime Bizon To: Dave Taht Cc: Luca Muscariello , Jonathan Morton , Cake List Message-ID: <20200423092909.GC28541@sakura> References: <75FEC2D9-BFC8-4FA2-A972-D11A823C5528@gmail.com> <603DFF79-D0C0-41BD-A2FB-E40B95A9CBB0@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-Virus-Scanned: ClamAV using ClamSMTP ; ns.iliad.fr ; Thu Apr 23 11:29:10 2020 +0200 (CEST) X-Mailman-Approved-At: Thu, 23 Apr 2020 06:47:52 -0400 Subject: Re: [Cake] Advantages to tightly tuning latency X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2020 09:29:11 -0000 On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote: Hello, > > Free has been using SFQ since 2005 (if I remember well). > > They announced the wide deployment of SFQ in the free.fr newsgroup. > > Wi-Fi in the free.fr router was not as good though. > > They're working on it. :) yes indeed. Switching to softmac approach, so now mac80211 will do rate control and scheduling (using wake_tx_queue model). for 5ghz, we use ath10k > I am very, very happy for y'all. Fiber has always been the sanest > thing. Is there a SPF+ gpon card yet I can plug into a convention > open source router yet? FYI Free.fr uses 10G-EPON, not GPON. Also most deployments are using an additionnal terminal equipement at called "ONT" or "ONU" that handle the PON part and exposes an ethernet port where the operator CPE is plugged. So we are back to the early days of DSL, where the hardest part (scheduling) is done inside a black box. That makes it easier to replace the operator CPE with your own standard ethernet router though. At least SOCs with integrated PON (supporting all flavours GPON/EPON/..) are starting to be deployed. Nothing available in opensource. Also note that it's not just kernel drivers, you also need some higher OAM stack to make that work, and there are a lot of existing standards, DPOE (EPON), OMCI (GPON)... all with interop challenges. > > The challenge becomes to keep up with these link rates in software > > as there is a lot of hardware offloading. Yes that's our pain point, because that's what the SOCs vendors deliver and you need to use that because there is no alternative. It's not entierely the SOCs vendors fault though. 15 years ago, your average SOC's CPU would be something like 200Mhz MIPS, Linux standard forwarding path (softirq => routing+netfilter => qdisc) was too slow for this, too much cache footprint/overhead. So every vendor started building alternatives forwarding path in their hardware and never looked back. Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which with a non crappy network driver and internal fabric should do be able to route 1Gbit/s with out-of-the-box kernel forwarding. But that's too late. SOC vendors compete against each others, and the big telcos need a way to tell which SOC is better to make a buying decision. So synthetic benchmarks have become the norm, and since everybody was able to do fill their pipe with 1500 bytes packets, benchmarks have moved to unrealistic 64 bytes packets (so called wirespeed) If you don't have hardware acceleration for forwarding, you don't exist in those benchmarks and will not sell your chipset. Also they invested so much in their alternative network stack that it's difficult to stop (huge R&D teams). That being said, they do have a point, when speed go above 1Gbit/s, the kernel becomes the bottleneck. For Free.fr 10Gbit/s offer, we had to develop an alternative (software) forwarding path using polling mode model (DPDK style), otherwise our albeit powerful ARM Cortex A72@2Ghz could not forward more than 2Gbit/s. And going multicore/RSS does not fly when the test case is single stream TCP session, which is what most speedtest application do (ookla only recently added multi-connections test). -- Maxime