From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C760A3B2A4 for ; Thu, 23 Apr 2020 07:57:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587643053; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vowYzBecrQ5pb5zKC6nvwC6lE531WAqQX/W+qP0zz4w=; b=aNuGlP8cpkLKU3GnmSfM9FyQJ3QMLAZlbmq83RC/Vc14aUs2Edzij+ta6phTXQzakPClQU aCkBW7Zl93T1n0oL7ZwsXsKuq78q8RY14eMpeMxVbMHLjufct7EIr4O/51QKuijyTOH2Wk 5HBlPCGJ7JTIYQjJuqArSilMZaEYrDA= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-276-Ry27zsmDPeeIUnqfK6nM1A-1; Thu, 23 Apr 2020 07:57:29 -0400 X-MC-Unique: Ry27zsmDPeeIUnqfK6nM1A-1 Received: by mail-lj1-f197.google.com with SMTP id e19so849618ljb.3 for ; Thu, 23 Apr 2020 04:57:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=vowYzBecrQ5pb5zKC6nvwC6lE531WAqQX/W+qP0zz4w=; b=N7FCXIBDIWHFRNvkJ2IvkTqESMZmyWP6V5XnULgGDcva51RJQjB713tCHKcGFPse5y d7+FznqLCFML8dQmyLwJYNZ+jDad79LbkK1JrzC3OIkVpYULNxW/cOpO4fS2zW28w077 zgTYoDOMlE8XeWKZyo13NOjtlwwx6Pp0BWIabU+Ev/fAmsnmqN/agM1NSIoqx9Opp3Ge lw3XX78EX8KR39jRS0RFZ5fa/b64s44se39pcx0BRCtPLA6ZZbmu/o3ZdySt8pX5oL7f el/0iTAsm4j3hG1wRkz8O3C2XEGYPfJ+PMf2xVNrIdiNdX3sou0ZWEe140sMpLHwUnwF VxfA== X-Gm-Message-State: AGi0PualuIWmypvqVkavDox2QMpwY329cNKw66eRiCCBTkmv08tj8phh y0YHLGxUK+a9gpNaV2/Ehq1pGLhwyj0wYTnRI1BMXlVognot+REFMB+4wJbRcD7mBHwD9LwQcf8 hoQdclKH4PLtvzqB83p3GtA== X-Received: by 2002:a2e:99c2:: with SMTP id l2mr2233976ljj.92.1587643048295; Thu, 23 Apr 2020 04:57:28 -0700 (PDT) X-Google-Smtp-Source: APiQypL03yxM8e8tj9iGvHY/qhYK0wcxsZvd6KgtQ9isWAPgfX3Ap9eaLOfrz5f5+iMNRyvnSFU4eQ== X-Received: by 2002:a2e:99c2:: with SMTP id l2mr2233956ljj.92.1587643047901; Thu, 23 Apr 2020 04:57:27 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id b16sm1635461lfj.2.2020.04.23.04.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 04:57:27 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 6E8521814FF; Thu, 23 Apr 2020 13:57:25 +0200 (CEST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= To: Maxime Bizon , Dave Taht Cc: Cake List In-Reply-To: <20200423092909.GC28541@sakura> References: <75FEC2D9-BFC8-4FA2-A972-D11A823C5528@gmail.com> <603DFF79-D0C0-41BD-A2FB-E40B95A9CBB0@gmail.com> <20200423092909.GC28541@sakura> X-Clacks-Overhead: GNU Terry Pratchett Date: Thu, 23 Apr 2020 13:57:25 +0200 Message-ID: <87o8ri76u2.fsf@toke.dk> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] Advantages to tightly tuning latency X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2020 11:57:33 -0000 Maxime Bizon writes: > On Wednesday 22 Apr 2020 =C3=A0 07:48:43 (-0700), Dave Taht wrote: > > Hello, > >> > Free has been using SFQ since 2005 (if I remember well). >> > They announced the wide deployment of SFQ in the free.fr newsgroup. >> > Wi-Fi in the free.fr router was not as good though. >>=20 >> They're working on it. :) > > yes indeed. > > Switching to softmac approach, so now mac80211 will do rate control > and scheduling (using wake_tx_queue model). > > for 5ghz, we use ath10k That is awesome! Please make sure you include the AQL patch for ath10k, it really works wonders, as Dave demonstrated: https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.ht= ml >> I am very, very happy for y'all. Fiber has always been the sanest >> thing. Is there a SPF+ gpon card yet I can plug into a convention >> open source router yet? > > FYI Free.fr uses 10G-EPON, not GPON. > > Also most deployments are using an additionnal terminal equipement at > called "ONT" or "ONU" that handle the PON part and exposes an ethernet > port where the operator CPE is plugged. So we are back to the early > days of DSL, where the hardest part (scheduling) is done inside a > black box. That makes it easier to replace the operator CPE with your > own standard ethernet router though. > > At least SOCs with integrated PON (supporting all flavours > GPON/EPON/..) are starting to be deployed. Nothing available in > opensource. > > Also note that it's not just kernel drivers, you also need some higher > OAM stack to make that work, and there are a lot of existing > standards, DPOE (EPON), OMCI (GPON)... all with interop challenges. It always bugged me that there was no open source support for these esoteric protocols and standards. It would seem like an obvious place to pool resources, but I guess proprietary vendors are going to keep doing their thing :/ >> > The challenge becomes to keep up with these link rates in software >> > as there is a lot of hardware offloading. > > Yes that's our pain point, because that's what the SOCs vendors > deliver and you need to use that because there is no alternative. > > It's not entierely the SOCs vendors fault though. > > 15 years ago, your average SOC's CPU would be something like 200Mhz > MIPS, Linux standard forwarding path (softirq =3D> routing+netfilter =3D> > qdisc) was too slow for this, too much cache footprint/overhead. So > every vendor started building alternatives forwarding path in their > hardware and never looked back. > > Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which > with a non crappy network driver and internal fabric should do be able > to route 1Gbit/s with out-of-the-box kernel forwarding. > > But that's too late. SOC vendors compete against each others, and the > big telcos need a way to tell which SOC is better to make a buying > decision. So synthetic benchmarks have become the norm, and since > everybody was able to do fill their pipe with 1500 bytes packets, > benchmarks have moved to unrealistic 64 bytes packets (so called > wirespeed) > > If you don't have hardware acceleration for forwarding, you don't > exist in those benchmarks and will not sell your chipset. Also they > invested so much in their alternative network stack that it's > difficult to stop (huge R&D teams). That being said, they do have a > point, when speed go above 1Gbit/s, the kernel becomes the bottleneck. > > For Free.fr 10Gbit/s offer, we had to develop an alternative > (software) forwarding path using polling mode model (DPDK style), > otherwise our albeit powerful ARM Cortex A72@2Ghz could not forward > more than 2Gbit/s. We're working on that in kernel land - ever heard of XDP? On big-iron servers we have no issues pushing 10s and 100s of Gbps in software (well, the latter only given enough cores to throw at the problem :)). There's not a lot of embedded platforms support as of yet, but we do have some people in the ARM world working on that. Personally, I do see embedded platforms as an important (future) use case for XDP, though, in particular for CPEs. So I would be very interested in hearing details about your particular platform, and your DPDK solution, so we can think about what it will take to achieve the same with XDP. If you're interested in this, please feel free to reach out :) > And going multicore/RSS does not fly when the test case is single > stream TCP session, which is what most speedtest application do (ookla > only recently added multi-connections test). Setting aside the fact that those single-stream tests ought to die a horrible death, I do wonder if it would be feasible to do a bit of 'optimising for the test'? With XDP we do have the ability to steer packets between CPUs based on arbitrary criteria, and while it is not as efficient as hardware-based RSS it may be enough to achieve line rate for a single TCP flow? -Toke