From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CDF203B29D for ; Thu, 14 Jan 2021 17:07:19 -0500 (EST) Received: by mail-ej1-x62c.google.com with SMTP id jx16so10464466ejb.10 for ; Thu, 14 Jan 2021 14:07:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jackrabbitwireless.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=C6BCRfSV0K7OAF0cObHUZydpKz0sIM6QgTs3G+Kyw8E=; b=lWkMicbEIXiGdMSJ+5L/zEiVuu9gpZvwrxABBCOGeptm71O+4s4A5q6z+UJ4dyETNs 0twDsU8uPODmazt6w7W7j5knD30Kg2ZVQ9jsoVBolAbqmG9HVs+Lpsp8izR9w0fsBW2Q YrPUJUYqFXkJumMW3VI8RnP22rTnpgjkgvntbu/017kWeHA0zT4PjbqqDHc9zFGYZKHy UXjK/nXJ7ZENAP1AH6SlkYha9c7fA8uwyQXBBOG4pkVD2zqZYmyM94cPf5CmPbEGGdJk R+slqDxPEdSrSzdqz7N5+eeuoZJ60DVKx3MIM4i1J0kcsDg7/Iu/zOYl7gXGd9pBPao5 w9QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=C6BCRfSV0K7OAF0cObHUZydpKz0sIM6QgTs3G+Kyw8E=; b=koKD9OizYTcUpj7oD142uleFMM+Hye64hoqq+MHQ36dJI9Jnwuz1x5Zz85uQtlnK4n HB1CRwSNmo1K9cuQ0STmaZ+LAHat1A1rIFGkVeGt+asvi9ql98CWBJHnM4lGd3u2qDff r18y3o5KqTAi+fRieIYZClSXJFNkfwhEBYbeA3CHOwhxxm8Sq7k06uuNqa6wDGiwfqFm L+vkbdh0NJAp8tu3mMVBBzz2m1gbRL5vkHFoPgRmxMrxYK2jJ1znYJQq5LF4vrhJr+3m sllC7wJsX9DN60tQXaonnPy/KcQh+bxps4btKvjbp5+ZDJOyNnNSyUSwg3wshwpAFgK8 xwyg== X-Gm-Message-State: AOAM532OVWcrN/QlansjUcPMohrYrvGNJ1vZUWktYeRZ3XXlOY4/YtPr VFZTND24fWfydWky4T+9xq+lWhaX7Q5l0VHhSGPKCw== X-Google-Smtp-Source: ABdhPJyf+RD5WhYIJygLWFEAi5HwDOSqoV4XY8MY76PeF01hKu5Q7+k1hzMwJ59sp8jv+d1HdydzSNzvVZ+y4h+wxkw= X-Received: by 2002:a17:906:4f0d:: with SMTP id t13mr6840504eju.10.1610662038800; Thu, 14 Jan 2021 14:07:18 -0800 (PST) MIME-Version: 1.0 References: <87y2gvqp2i.fsf@toke.dk> In-Reply-To: <87y2gvqp2i.fsf@toke.dk> From: Robert Chacon Date: Thu, 14 Jan 2021 15:07:07 -0700 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: bloat@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="000000000000ad24b805b8e37ab2" Subject: Re: [Bloat] Thanks to developers / htb+fq_codel ISP shaper X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2021 22:07:20 -0000 --000000000000ad24b805b8e37ab2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > Cool! What kind of performance are you seeing? The README mentions being > limited by the BPF hash table size, but can you actually shape 2000 > customers on one machine? On what kind of hardware and at what rate(s)? On our production network our peak throughput is 1.5Gbps from 200 clients, and it works very well. We use a simple consumer-class AMD 2700X CPU in production because utilization of the shaper VM is ~15% at 1.5Gbps load. Customers get reliably capped within =C2=B12Mbps of their allocated htb/fq_= codel bandwidth, which is very helpful to control network congestion. Here are some graphs from RRUL performed on our test bench hypervisor: https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_su= bs_4G.png In that example, bandwidth for the "subscriber" client VM was set to 4Gbps. 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of LibreQoS. The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbps utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network drivers are used and most offloading types are enabled. In our setup, VM networking multiqueue isn't enabled (it kept disrupting traffic flow), so 6Gbps is probably the most it can achieve like this. Our qdiscs in this VM may be limited to one core because of that. I suspect in a non-virtualized setup, or one with multiqueue, it can handle much more throughput. Either way for now it's surprising to me how well it works and I'm just grateful for it haha. Kudos to you and your peers for making fq_codel so efficient! - Robert On Thu, Jan 14, 2021 at 12:46 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Robert Chacon writes: > > > Hello everyone, > > > > I am new here, my name is Robert. I operate a small ISP in the US. I > wanted > > to post here to thank Dave T=C3=A4ht, as well as the dozens of contribu= tors to > > the fq_codel and cake projects. > > Thank you for reaching out! It's always fun to hear about real-world > deployments of this technology, and it's great to hear that it's working > well for you! :) > > > I created a simple python application that uses htb+fq_codel to shape m= y > > customers' traffic, and have seen great performance improvements. I am > > maintaining it as an open source project for other ISPs to use at > > https://github.com/rchac/LibreQoS > > Cool! What kind of performance are you seeing? The README mentions being > limited by the BPF hash table size, but can you actually shape 2000 > customers on one machine? On what kind of hardware and at what rate(s)? > > -Toke > --=20 [image: photograph] *Robert Chac=C3=B3n* Owner *M* (915) 730-1472 *E* robert.chacon@jackrabbitwireless.com *JackRabbit Wireless LLC* P.O. Box 222111 El Paso, TX 79913 *jackrabbitwireless.com* --000000000000ad24b805b8e37ab2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> Cool! What kind of performance are y= ou seeing? The README mentions being
> limited by the BPF hash table size, but can you actually shape 2000 > customers on one machine? On what kind of hardware and at what rate(s)= ?

On our production network our peak t= hroughput is 1.5Gbps from 200 clients, and it works very well.
We= use a simple consumer-class AMD 2700X CPU in production because utilizatio= n of the shaper VM is ~15% at 1.5Gbps load.
Customers get rel= iably capped within =C2=B12Mbps of their allocated htb/fq_codel bandwidth, = which is very helpful to control network congestion.

Here are some graphs from RRUL performed on our test bench hyperviso= r: https://raw.githubusercontent.com/rchac/LibreQoS/m= ain/docs/fq_codel_1000_subs_4G.png
In that example, bandwidth for the "subscriber" client VM was set t= o=20 4Gbps. 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of Lib= reQoS.
The test bench server has an AMD 3900X running Ubuntu = in Proxmox. 4Gbps utilizes 10% of the VM's 12 cores. Paravirtualized Vi= rtIO network drivers are used and most offloading types are enabled.
<= div>In our setup, VM networking multiqueue isn't enabled (it kept disrupting = traffic flow), so 6Gbps is probably the most it=20 can achieve like this. Our qdiscs in this VM may be limited to one core bec= ause of that.
I suspect in a non-virtualized setup, or one with = multiqueue, it can handle much more throughput.
Either way fo= r now it's surprising to me how well it works and I'm just grateful= for it haha.
Kudos to you and your peers for making fq_codel so = efficient!

- Robert

On Thu, Jan 14, 2= 021 at 12:46 PM Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk> wrote:
Robert Chacon <robert.chacon@jackrabbitwireless.com= > writes:

> Hello everyone,
>
> I am new here, my name is Robert. I operate a small ISP in the US. I w= anted
> to post here to thank Dave T=C3=A4ht, as well as the dozens of contrib= utors to
> the fq_codel and cake projects.

Thank you for reaching out! It's always fun to hear about real-world deployments of this technology, and it's great to hear that it's wo= rking
well for you! :)

> I created a simple python application that uses htb+fq_codel to shape = my
> customers' traffic, and have seen great performance improvements. = I am
> maintaining it as an open source project for other ISPs to use at
> https://github.com/rchac/LibreQoS

Cool! What kind of performance are you seeing? The README mentions being limited by the BPF hash table size, but can you actually shape 2000
customers on one machine? On what kind of hardware and at what rate(s)?

-Toke


--
=20 =20 =20
=20 3D"photograph"

Robert Cha= c=C3=B3n
 Owner

=20 		=20 =20 =20 =20 =20
M (915) 730= -1472
 =20 E robert.c= hacon@jackrabbitwireless.com
JackRabbit Wireless LLC =20 =09
P.O. Box 222111
 El Paso, TX 79913 =09
jack= rabbitwireless.com
--000000000000ad24b805b8e37ab2--