From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 83AE43B29D for ; Thu, 21 Jan 2021 00:50:37 -0500 (EST) Received: by mail-lj1-x22a.google.com with SMTP id e7so1110245ljg.10 for ; Wed, 20 Jan 2021 21:50:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jackrabbitwireless.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=aZYRUf50aYXPQiCVxD4shV5vXOt0ICwkUph+2eMjlGE=; b=ORiKDoTdbAn/B0Y1aW+IdWo1tOmzh3wXERlwDmI1xeQZAdKDKwrVw+gv+oph0nyS7P Ff6RntpNzWGTMAPdrihq8JqEZSB35De+aQXLHH4Qt25DsreexPzvA15BK8rUubbrHs7/ udUQDUmDmP9jVdIGG+Bwe9ci0HUl1DKkQrJUU8zHZ6WIM7VqhClM0VcJzZJEnYRplRLC H5p4SmcfGxUpSqwybC/Xr5lvZpuQAO5npAk7z1XzURd7iwiA8xXpNGGkPPp2J6FGrGRV uN//32zzPYsJg7zizy0Wqh4zSyix9O+NS6goJJN13W1RFv8cFHZCMmckImKRPWeAIbX+ KqRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=aZYRUf50aYXPQiCVxD4shV5vXOt0ICwkUph+2eMjlGE=; b=QvvsflFxNW6nzXM3IaDnuvyRKIbnp7w2zywBw8o5g/rYq9GA8y+LQUtVJ6QpgOPEgJ 7uwVDk2nwF7JAOonI7BM2XzeJAUwshtQuSyPTc3fqOm5q+SY1i2E0JJ45GTWLK2EsjlE seewrvAoWnF/BpAD2XuywsbSjBEX190CjrfmGhYtclUwb86jBTXuX74j1K3cA99/gQN2 h8btTVY7ANeu0sNBk79QJhj78eg0BA7Xy9B3h289RYZlaA+SKZrWM+CGUH5OYIQcBpcV ZDMGIBPXXukF9753ppO2b7r663tM5pXwrLvyItsy4vzD8Jlja4WQqhczkNxwAJQbB7le VAPg== X-Gm-Message-State: AOAM532PhBCCcmXFlNGG8JH80sTafDjdQptUl34FqbP6Lc5BrANzNfkt +5nPgSyPBJHgiQ40MhcvhHrd/0wFGKvxA+Mibxntz9oDObE= X-Google-Smtp-Source: ABdhPJz+jNVZSe5LvX1jVR3vrzGERkR6volbiG6nbtidddSo/h8rnnAJjb8020gqshp/L+PepmLaOs/q2wUgu106t4E= X-Received: by 2002:a2e:9a18:: with SMTP id o24mr6130761lji.311.1611208236346; Wed, 20 Jan 2021 21:50:36 -0800 (PST) MIME-Version: 1.0 References: <87y2gvqp2i.fsf@toke.dk> <87o8hqs7q3.fsf@toke.dk> In-Reply-To: <87o8hqs7q3.fsf@toke.dk> From: Robert Chacon Date: Wed, 20 Jan 2021 22:50:25 -0700 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: bloat@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="0000000000009641ca05b962a66d" Subject: Re: [Bloat] Thanks to developers / htb+fq_codel ISP shaper X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2021 05:50:37 -0000 --0000000000009641ca05b962a66d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Toke, Thank you very much for pointing me in the right direction. I am having some fun in the lab tinkering with the 'mq' qdisc and Jesper's xdp-cpumap-tc. It seems I will need to use iptables or nftables to filter packets to corresponding queues, since mq apparently cannot have u32 filters on its root. I will try to familiarize myself with iptables and nftables, and hopefully get it working soon and report back. Thank you! On Fri, Jan 15, 2021 at 5:30 AM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Robert Chacon writes: > > >> Cool! What kind of performance are you seeing? The README mentions bei= ng > >> limited by the BPF hash table size, but can you actually shape 2000 > >> customers on one machine? On what kind of hardware and at what rate(s)= ? > > > > On our production network our peak throughput is 1.5Gbps from 200 > clients, > > and it works very well. > > We use a simple consumer-class AMD 2700X CPU in production because > > utilization of the shaper VM is ~15% at 1.5Gbps load. > > Customers get reliably capped within =C2=B12Mbps of their allocated > htb/fq_codel > > bandwidth, which is very helpful to control network congestion. > > > > Here are some graphs from RRUL performed on our test bench hypervisor: > > > https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_= subs_4G.png > > In that example, bandwidth for the "subscriber" client VM was set to > 4Gbps. > > 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of > LibreQoS. > > The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbps > > utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network drive= rs > > are used and most offloading types are enabled. > > In our setup, VM networking multiqueue isn't enabled (it kept disruptin= g > > traffic flow), so 6Gbps is probably the most it can achieve like this. > Our > > qdiscs in this VM may be limited to one core because of that. > > I suspect the issue you had with multiqueue is that it requires per-CPU > partitioning on a per-customer base to work well. This is possible to do > with XDP, as Jesper demonstrates here: > > https://github.com/netoptimizer/xdp-cpumap-tc > > With this it should be possible to scale the hardware queues across > multiple CPUs properly, and you should be able to go to much higher > rates by just throwing more CPU cores at it. At least on bare metal; not > sure if the VM virt-drivers have the needed support yet... > > -Toke > --=20 [image: photograph] *Robert Chac=C3=B3n* Owner *M* (915) 730-1472 *E* robert.chacon@jackrabbitwireless.com *JackRabbit Wireless LLC* P.O. Box 222111 El Paso, TX 79913 *jackrabbitwireless.com* --0000000000009641ca05b962a66d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Toke,

Thank you very much fo= r pointing me in the right direction.
I am having some fun in the= lab tinkering with the 'mq' qdisc and Jesper's xdp-cpumap-tc.<= /div>
It seems I will need to use iptables or nftables to filter packet= s to corresponding queues, since mq apparently cannot have u32 filters on i= ts root.
I will try to familiarize myself with iptables and nftab= les, and hopefully get it working soon and report back. Thank you!

On Fri, Jan 15, 2021 at 5:30 AM Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk> wrote:
Robert Chacon <robert.chacon@jackr= abbitwireless.com> writes:

>> Cool! What kind of performance are you seeing? The README mentions= being
>> limited by the BPF hash table size, but can you actually shape 200= 0
>> customers on one machine? On what kind of hardware and at what rat= e(s)?
>
> On our production network our peak throughput is 1.5Gbps from 200 clie= nts,
> and it works very well.
> We use a simple consumer-class AMD 2700X CPU in production because
> utilization of the shaper VM is ~15% at 1.5Gbps load.
> Customers get reliably capped within =C2=B12Mbps of their allocated ht= b/fq_codel
> bandwidth, which is very helpful to control network congestion.
>
> Here are some graphs from RRUL performed on our test bench hypervisor:=
> https://raw= .githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_subs_4G.png
> In that example, bandwidth for the "subscriber" client VM wa= s set to 4Gbps.
> 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of Libre= QoS.
> The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbp= s
> utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network = drivers
> are used and most offloading types are enabled.
> In our setup, VM networking multiqueue isn't enabled (it kept disr= upting
> traffic flow), so 6Gbps is probably the most it can achieve like this.= Our
> qdiscs in this VM may be limited to one core because of that.

I suspect the issue you had with multiqueue is that it requires per-CPU
partitioning on a per-customer base to work well. This is possible to do with XDP, as Jesper demonstrates here:

https://github.com/netoptimizer/xdp-cpumap-tc

With this it should be possible to scale the hardware queues across
multiple CPUs properly, and you should be able to go to much higher
rates by just throwing more CPU cores at it. At least on bare metal; not sure if the VM virt-drivers have the needed support yet...

-Toke


--
=20 =20 =20
=20 3D"photograph"

Robert Cha= c=C3=B3n
 Owner

=20 		=20 =20 =20 =20 =20
M (915) 730= -1472
 =20 E robert.c= hacon@jackrabbitwireless.com
JackRabbit Wireless LLC =20 =09
P.O. Box 222111
 El Paso, TX 79913 =09
jack= rabbitwireless.com
--0000000000009641ca05b962a66d--