From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 141793B29D; Fri, 2 Dec 2022 11:53:57 -0500 (EST) Received: by mail-wm1-x335.google.com with SMTP id c65-20020a1c3544000000b003cfffd00fc0so6917447wma.1; Fri, 02 Dec 2022 08:53:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=4eDYB7izO2+mU8VAsQMFHb79zLnx9fUUMl79PfQKaVk=; b=HhrLyVX9SC0+SYfsKCTbyr7/lmb4mZGh6sjt7DqEcwRmTZOj89xYrdW46+q4I3UCgt L3JmHJUA8nEhTVteMFqfyOOMmjptbKmKSwjGA9tDtRrx8eFtWOPGdK4FIgeD9kzXuuEW vveB68FNbvrBk5BkiLYj1V9K4RwDSkToXCl3ZUtwT7Thg6GkX0vn9hW0jlw27v0yKj3f 5bFsfMrAM8Ahzxyow1d9X2eWJNK2CPKfat2cjZzwn41WgxidBXGXGxh66erLxtrWhu0m e1UjKJ7uDYjmwmXm9HvsDbkRERixSKZrGZueDbD1tXmU22Uwd02JCcIEdRClzIeTju8V 6TAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4eDYB7izO2+mU8VAsQMFHb79zLnx9fUUMl79PfQKaVk=; b=4VP6JLlVU+PH8v9oL89wYJD//sSB8zj3xbv4SKCMTCydM9WiToxnHCnpbGLtoEJ/7N EmE9Yqm37VWMLke7UnYkFPvRc1I0UJbSRZh6+d2KuYEx2+sHfbrfguuFGHEs+boYB+6T fUWQapoAvHWhcZljlev1qwAc8ZF4IZ6mxtUQvlEbSx8en2VPcSJooZLW3EBG4ml+epyB zcLmkvW6oqtCm0wsrtvIKeMupsC2+Ja7QWQJ52FbCePxxpeKHRxH2nIIddg88c5q2vLj LbVlSM1noN61a5jxBmsm9QQ3rg6WkHhyyh4YrgXgLAPf9jJp4P395WEsNAQl+qi7d/Av Kx2g== X-Gm-Message-State: ANoB5pnB3vBlg5Vm0H2L00HOctCbvsb/cnVN9+JQtHYGf6PWJARk9B2J eWEarBt2+965KR2YPgOCtCBlVrbzL9Il2jQ6UhXNt3zvGcmiUQ== X-Google-Smtp-Source: AA0mqf4UQ6oK7uRWtkwhBGChCWoTYJIad/Ihm1x7NfkVxPZgGtcbMBwnUZSt+XLdWrdZqzWG2xb8xJCTGXkgvYrEecQ= X-Received: by 2002:a05:600c:4e46:b0:3d0:57ea:3188 with SMTP id e6-20020a05600c4e4600b003d057ea3188mr18818929wmq.28.1670000035865; Fri, 02 Dec 2022 08:53:55 -0800 (PST) MIME-Version: 1.0 From: Dave Taht Date: Fri, 2 Dec 2022 08:53:43 -0800 Message-ID: To: Linux Kernel Network Developers , bpf , xdp-newbies@vger.kernel.org Cc: libreqos , Cake List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [LibreQoS] questions: LPM -> irq or queue mapping on the card for cake? X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2022 16:53:57 -0000 all: After a pretty long layoff from kernel development of any sort of mine... we're pushing the limits again trying to get cake to support 10s of thousands of ISP subscribers, leveraging xdp for reads, and ebpf "pping" for inline monitoring of TCP rtts, then a ton of htb bins + cake for each subscriber. shameless plugs: https://github.com/thebracket/cpumap-pping#cpumap-pping # everybody needs kathie nichols and simon 's pping!!! https://github.com/LibreQoE (even more shameless plug - ask your isp to try libreqos.io out - presently at 10k users, pushing 11gbit/sec, 24% of 16 cores! I'm really, really amazed by all this, I remember struggling to get cake to crack 50Mbits on a 600mhz mips box a decade back. Way to go everyone! - love, rip van winkle) Anyway... after adopting xdp fully over the past couple months... most of the CPU time is now spent in htb, and while I see htb has been successfully offloaded by the mlx5, it's not clear if that can pull from from cake as an underlying qdisc (?), or only a pfifo, nor how much buffering offloads like this introduce. ? Are there other cards with an htb offload now? Secondly - The xdp path is marvelous, but in trying to drive this transparent bridge to 100Gbit, I find myself wanting something old fashioned, and in my mind, simpler than a match pattern - is there any ethernet card that lets you do a TCAM mapping of a large (say, 128 thousand) set of IP addresses to an irq or queue ? 1.2.3.4/29 -> irq 8 (or hw queue 8) 1.2.4.1/32 -> irq 9 (or hw queue 9) a:b:c:d::/56 -> irq 8 (etc * 10s of thousands of other ip addresses, 1 or more LPM matches per subscriber) Needn't be big enough to fill a bgp table... This is different from RPS in that we don't want a rxhash to spread the load "evenly", but to always direct a set of IP addresses to a particular core, on a particular queue - which is setup to do all that htb-ing (32k unique bins per core, e.g. 1.9m bins on 64 cores) and cake-ing. Other ideas for steppingstones on the march to 100gbit forwarding through tons of cake gladly considered. We're going to be fiddling with the metadata stuff in xdp as well - 3 hw hashes for cake, a rx timestamp for codel will probably help there too. --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC