From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2a0c:4d80:42:2001::664]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A61403B2A4 for ; Tue, 1 Sep 2020 17:09:13 -0400 (EDT) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1598994551; bh=YeBPaPJ/tSar+EMr0fptREXMIZ5iA0YKWLgPj9wKShE=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=fvOnS/x4fsOoP8QyjKTxE5YteUrLPIkwEm4+7rfMzDpI/Qk/ytI8DCrU++75uDNly kXtItOr9dkBTbTHRlSJmKcQaRs4eb/RsTtzS8HH3AteYaBFYNRC6P9VX4Wu5F8Jalf yin7bUQKG0cnjYwWcUxupJIT0u3gbY5UKYJbleiuVKkW3CJpiobMViqA3Vx36ZfELe 4fCvHwgMC881tU+EzOAgiBj5FYbKZ4ZbdonwzzJKTdqRuMYsEtGRJ0sKCTmkeJiwpc MAxJdwCUQw1HIXnbRjUqrb2cYzVZYq/kUtnGNZCE2S5Qq7KAXVvTSB3gCc7Yu5vVxG ipbAiNcNkS3+Q== To: Jonathan Morton Cc: Sebastian Moeller , bloat@lists.bufferbloat.net In-Reply-To: <48FC5B6D-21B6-4A5E-88DC-18FF03BE47C0@gmail.com> References: <87mu2bjbf8.fsf@toke.dk> <5DBFB383-13E8-4587-BE49-1767471D7D59@jonathanfoulkes.com> <87r1rliiiw.fsf@toke.dk> <87o8mpibew.fsf@toke.dk> <48FC5B6D-21B6-4A5E-88DC-18FF03BE47C0@gmail.com> Date: Tue, 01 Sep 2020 23:09:10 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87lfhti4qx.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] CAKE in openwrt high CPU X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Sep 2020 21:09:13 -0000 Jonathan Morton writes: >> On 1 Sep, 2020, at 9:45 pm, Toke H=C3=B8iland-J=C3=B8rgensen via Bloat <= bloat@lists.bufferbloat.net> wrote: >>=20 >> CAKE takes the global qdisc lock. > > Presumably this is a default mechanism because CAKE doesn't handle any > locking itself. > > Obviously it would need to be replaced with at least a lock over > CAKE's complete data structures, taking the lock on each entry point > and releasing it at each return point, and I assume there is a flag we > can set to indicate we do so. Finer-grained locking might be possible, > but CAKE is fairly complex so that might be hard to implement. Locking > per CAKE instance would at least allow running ingress and egress on > different CPUs. What you're describing here is basically the existing qdisc root lock. It is per instance of the qdisc, and it is held only while enqueueing and dequeueing packets from that qdisc. So it is possible today to run the ingress and egress instances of CAKE on different CPUs. All you have to do is schedule the packets to be processed on different CPUs in the different directions - which usually means messing with RPS settings for the NIC, and as I remarked to Sebastian, for many OpenWrt SOCs this is not really supported... To make CAKE truly take advantage of multiple CPUs, there are to options: 1. Make it aware of multiple hardware queues. To do this, we to implement the 'attach()' method in the Qdisc_ops struct (see sch_mq for an example). The idea here would be to create stub child qdiscs with a separate struct Qdisc_ops implementing enqueue() and dequeue(). These would be called separately for each hardware queue, with their separate locks held at the time; and with proper XPS steering, each hardware queue can be serviced by a separate CPU. 2. Set the TCQ_F_NOLOCK in the qdisc flags; this will cause the existing enqueue() and dequeue() functions to be called without the root lock being held, and the qdisc is responsible for dealing with that itself. Of course in either case, the trick is to get the CAKE data structures to play nice with concurrent access from multiple CPUs. For option 1. above, we could just duplicate all the flow queues for each netdev queue and take the hit in wasted space - or we could partition the data structure, either statically at init, or dynamically as each flow becomes active. But at a minimum there would need to be some way for the shaper to enforce the maximum rate. Maybe a granular lock or an atomic is good enough for this, though? Note also that for 2. there's an ongoing issue[0] with packets getting stuck which is still unresolved, as far as I can tell - so not sure if this is the right way to go. However, apart from this, the benefit of 2. is that CAKE could *potentially* process packets on multiple CPUs without relying on hardware multi-Q. I'm not quite sure if the stack will actually process packets on more than one CPU without them, though. Either way, I suppose some experimentation would be needed to find the best solution. -Toke [0] https://lore.kernel.org/netdev/CACS=3Dqq+a0H=3De8yLFu95aE7Hr0bQ9ytCBBn2= rFx82oJnPpkBpvg@mail.gmail.com/