From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2a0c:4d80:42:2001::664]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2333F3B29D for ; Thu, 21 Jan 2021 06:14:36 -0500 (EST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1611227672; bh=o1ohV6MIlb10nwN7Hr1t8nYSfvT9RieTb2FfastdOfI=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=x5cUbQMlvw5mzLYj+70Ru4hU9c4ybPtOE+NPO1ZrcKtSyd0b/mxpcTjz9uFkH8+LG Uk2LZzwIMODE9YJhJHaZO5yqRAzTm0m/UMeuqnY7YRkJmgOavTWeeRBOh6+YlFiFqq Cow5GALy4uYcM269H3XYYvjQ/7GesoP3MIHd+C2cTculrdzJqCX1MkEF+SZLpaPhSE 8w9VNYAo1JjGddwFzE9ObUXRO8PnFjrinc9co4LWjVA3frQRh75PPylvTKIRJk2uWo HBemuZGTpxgcfL3K+4vMOBaYNB1kpyu+bi8p9WHJOTd2pTGzEDnLedqISzufrptjBG 7H8DuotCHtiDA== To: Robert Chacon , Jesper Dangaard Brouer Cc: bloat@lists.bufferbloat.net In-Reply-To: References: <87y2gvqp2i.fsf@toke.dk> <87o8hqs7q3.fsf@toke.dk> Date: Thu, 21 Jan 2021 12:14:31 +0100 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87bldik0dk.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Thanks to developers / htb+fq_codel ISP shaper X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2021 11:14:36 -0000 Robert Chacon writes: > Toke, > > Thank you very much for pointing me in the right direction. > I am having some fun in the lab tinkering with the 'mq' qdisc and Jesper's > xdp-cpumap-tc. > It seems I will need to use iptables or nftables to filter packets to > corresponding queues, since mq apparently cannot have u32 filters on its > root. > I will try to familiarize myself with iptables and nftables, and hopefully > get it working soon and report back. Thank you! Cool - adding in Jesper, maybe he has some input on this :) -Toke > On Fri, Jan 15, 2021 at 5:30 AM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> Robert Chacon writes: >> >> >> Cool! What kind of performance are you seeing? The README mentions be= ing >> >> limited by the BPF hash table size, but can you actually shape 2000 >> >> customers on one machine? On what kind of hardware and at what rate(s= )? >> > >> > On our production network our peak throughput is 1.5Gbps from 200 >> clients, >> > and it works very well. >> > We use a simple consumer-class AMD 2700X CPU in production because >> > utilization of the shaper VM is ~15% at 1.5Gbps load. >> > Customers get reliably capped within =C2=B12Mbps of their allocated >> htb/fq_codel >> > bandwidth, which is very helpful to control network congestion. >> > >> > Here are some graphs from RRUL performed on our test bench hypervisor: >> > >> https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000= _subs_4G.png >> > In that example, bandwidth for the "subscriber" client VM was set to >> 4Gbps. >> > 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of >> LibreQoS. >> > The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbps >> > utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network driv= ers >> > are used and most offloading types are enabled. >> > In our setup, VM networking multiqueue isn't enabled (it kept disrupti= ng >> > traffic flow), so 6Gbps is probably the most it can achieve like this. >> Our >> > qdiscs in this VM may be limited to one core because of that. >> >> I suspect the issue you had with multiqueue is that it requires per-CPU >> partitioning on a per-customer base to work well. This is possible to do >> with XDP, as Jesper demonstrates here: >> >> https://github.com/netoptimizer/xdp-cpumap-tc >> >> With this it should be possible to scale the hardware queues across >> multiple CPUs properly, and you should be able to go to much higher >> rates by just throwing more CPU cores at it. At least on bare metal; not >> sure if the VM virt-drivers have the needed support yet... >> >> -Toke >> > > > --=20 > [image: photograph] > > > *Robert Chac=C3=B3n* Owner > *M* (915) 730-1472 > *E* robert.chacon@jackrabbitwireless.com > *JackRabbit Wireless LLC* > P.O. Box 222111 > El Paso, TX 79913 > *jackrabbitwireless.com*