From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C16C23B29D for ; Wed, 20 Jan 2021 23:39:04 -0500 (EST) Received: by mail-io1-xd33.google.com with SMTP id y19so1589725iov.2 for ; Wed, 20 Jan 2021 20:39:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=S5615AuzFBgjEQMwSznLPbE23BvksGsar3BnZErlOug=; b=qYpO0BEmFXggDayPbpBNIVpnHySu2Luk96uvcRDiLzIiqEIbtcfUb+ALtwegRdz78q BHdloFCSUlkeCX4pUvgqim3PPXJTNtERIksqoWILOiBNdG4OXgokjl4i7eC7R3rRC4NW HBfWv/W6ybJVgSXu3M+NLMFZVNPCyb2+BdXjrvbcKOPPaGFpigyCUqLo/HB8+BCXWTZy p246Mb4qI3/kDnmBrepxSPWDqziPgBH18dfjqv6v9AL4Pl406zQFSavJHflWRJ4AevtT gS1UYQFiXwpPJYxBfwKIiQyVUYZiEe2V2N/UCc4IEka+Ajl5ETKs4r88sw+sysOqn9mN JAwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=S5615AuzFBgjEQMwSznLPbE23BvksGsar3BnZErlOug=; b=fOMwWGxB2tNj+blkNP5scIEpXHhIlMTorCehq96wa6sWZ4ZLlXYtIKucZT9EOqts7c r2wKTNsHjxczJ4n5Sg2H5QkTODvIDNK6NlkXjo6fYJEJSyW8itihWF9yPEBRWwlqkDzU Yzo7jJGggMOte5lg/TyLrZVyYnwIDpZ1dXnDZBVsddBe89S4GnemAMfJhx9aICMwSKRW N5ue4yubhYlizPeqCoUJNtvq/qp1UkKqlks3FCR79L14M9ovG6fvJLytoEYLkn1HS8S5 kooSvuEPg/dIFpT5t6/bQh2vF7kJxTmi8Z4QDq9CHteytY+0ZJ8NQ2z6cIfOP7e+jOFc OC4Q== X-Gm-Message-State: AOAM531voUfY9rTO0m3duVqklZ8amQ86FupED8OyVFOMMCicH8WcCYr3 41KTWsR3ySobRl13FrICSHZ3IsnqSM2k8/VjVqY= X-Google-Smtp-Source: ABdhPJyOoWozvD+CvYXU3ZLym4BrFwWSIv30OKf9Pw3hnEUHI5d5ciHq8kHw92OgS4NnDSCmzqxj7dUpcEpnA/q+/yU= X-Received: by 2002:a6b:e50e:: with SMTP id y14mr9148273ioc.161.1611203944248; Wed, 20 Jan 2021 20:39:04 -0800 (PST) MIME-Version: 1.0 References: <87y2gvqp2i.fsf@toke.dk> In-Reply-To: From: Dave Taht Date: Wed, 20 Jan 2021 20:38:52 -0800 Message-ID: To: Robert Chacon Cc: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= , bloat Content-Type: multipart/alternative; boundary="000000000000c1fa4a05b961a6ca" Subject: Re: [Bloat] Thanks to developers / htb+fq_codel ISP shaper X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2021 04:39:04 -0000 --000000000000c1fa4a05b961a6ca Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jan 14, 2021 at 2:07 PM Robert Chacon < robert.chacon@jackrabbitwireless.com> wrote: > > Cool! What kind of performance are you seeing? The README mentions bein= g > > limited by the BPF hash table size, but can you actually shape 2000 > > customers on one machine? On what kind of hardware and at what rate(s)? > > On our production network our peak throughput is 1.5Gbps from 200 clients= , > and it works very well. > We use a simple consumer-class AMD 2700X CPU in production because > utilization of the shaper VM is ~15% at 1.5Gbps load. > Customers get reliably capped within =C2=B12Mbps of their allocated > htb/fq_codel bandwidth, which is very helpful to control network congesti= on. > > Here are some graphs from RRUL performed on our test bench hypervisor: > https://raw.githubusercontent.com/rchac/LibreQoS/main/docs/fq_codel_1000_= subs_4G.png > In that example, bandwidth for the "subscriber" client VM was set to > 4Gbps. 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of > LibreQoS. > What I really love about this plot is that it is now very possible for your customers to play live music together with reasonable latencies and jitter, under load. Existing tools like "jacktrip" should "just work. For a cool talk about the jacktrip revolution: https://www.npr.org/2020/11/21/937043051/musicians-turn-to-new-software-to-= play-together-online I'm using cake to keep things under control on my testbed network, using ardour as the mixing tool, and achieving about 6ms of inherent latency. I am hoping to sink a bit of time into galene.org and various web browsers this year to finally get closer to what the lola project has been doing for a while on the video front. > The test bench server has an AMD 3900X running Ubuntu in Proxmox. 4Gbps > utilizes 10% of the VM's 12 cores. Paravirtualized VirtIO network drivers > are used and most offloading types are enabled. > In our setup, VM networking multiqueue isn't enabled (it kept disrupting > traffic flow), so 6Gbps is probably the most it can achieve like this. Ou= r > qdiscs in this VM may be limited to one core because of that. > I suspect in a non-virtualized setup, or one with multiqueue, it can > handle much more throughput. > Either way for now it's surprising to me how well it works and I'm just > grateful for it haha. > Kudos to you and your peers for making fq_codel so efficient! > > - Robert > > On Thu, Jan 14, 2021 at 12:46 PM Toke H=C3=B8iland-J=C3=B8rgensen > wrote: > >> Robert Chacon writes: >> >> > Hello everyone, >> > >> > I am new here, my name is Robert. I operate a small ISP in the US. I >> wanted >> > to post here to thank Dave T=C3=A4ht, as well as the dozens of contrib= utors >> to >> > the fq_codel and cake projects. >> >> Thank you for reaching out! It's always fun to hear about real-world >> deployments of this technology, and it's great to hear that it's working >> well for you! :) >> >> > I created a simple python application that uses htb+fq_codel to shape = my >> > customers' traffic, and have seen great performance improvements. I am >> > maintaining it as an open source project for other ISPs to use at >> > https://github.com/rchac/LibreQoS >> >> Cool! What kind of performance are you seeing? The README mentions being >> limited by the BPF hash table size, but can you actually shape 2000 >> customers on one machine? On what kind of hardware and at what rate(s)? >> >> -Toke >> > > > -- > [image: photograph] > > > *Robert Chac=C3=B3n* Owner > *M* (915) 730-1472 > *E* robert.chacon@jackrabbitwireless.com > *JackRabbit Wireless LLC* > P.O. Box 222111 > El Paso, TX 79913 > *jackrabbitwireless.com* > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --=20 "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 --000000000000c1fa4a05b961a6ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jan 14, 2021 at 2:07 PM Rober= t Chacon <robert= .chacon@jackrabbitwireless.com> wrote:
> Cool! = What kind of performance are you seeing? The README mentions being
> limited by the BPF hash table size, but can you actually shape 2000 > customers on one machine? On what kind of hardware and at what rate(s)= ?

On our production network our peak t= hroughput is 1.5Gbps from 200 clients, and it works very well.
We= use a simple consumer-class AMD 2700X CPU in production because utilizatio= n of the shaper VM is ~15% at 1.5Gbps load.
Customers get rel= iably capped within =C2=B12Mbps of their allocated htb/fq_codel bandwidth, = which is very helpful to control network congestion.

Here are some graphs from RRUL performed on our test bench hyperviso= r: https://raw.githubusercontent.co= m/rchac/LibreQoS/main/docs/fq_codel_1000_subs_4G.png
In that example, bandwidth for the "subscriber" client VM was set t= o=20 4Gbps. 1000 IPv4 IPs and 1000 IPv6 IPs were in the filter hash table of Lib= reQoS.

What I really love a= bout this plot is that it is now very possible for your customers to
<= div>play live music together with reasonable latencies and jitter, under lo= ad.

Existing tools like "jacktrip" shoul= d "just work. For a cool talk about the jacktrip
<= br>
I'm using cake to=C2=A0keep things under control on my te= stbed network, using ardour
as the mixing tool, and achieving abo= ut 6ms of inherent latency.

I am hoping to sink a = bit of time into galene.org and various w= eb browsers this
year to finally get closer to what the lola proj= ect has been doing for a while on the video front.=C2=A0=C2=A0
= =C2=A0=C2=A0
The test bench server has an AMD 3900X running Ub= untu in Proxmox. 4Gbps utilizes 10% of the VM's 12 cores. Paravirtualiz= ed VirtIO network drivers are used and most offloading types are enabled.
In our setup, VM networking multiqueue isn't enabled (it kept disrupting = traffic flow), so 6Gbps is probably the most it=20 can achieve like this. Our qdiscs in this VM may be limited to one core bec= ause of that.
I suspect in a non-virtualized setup, or one with = multiqueue, it can handle much more throughput.
Either way fo= r now it's surprising to me how well it works and I'm just grateful= for it haha.
Kudos to you and your peers for making fq_codel so = efficient!

- Robert

On Thu, Jan 14, 2= 021 at 12:46 PM Toke H=C3=B8iland-J=C3=B8rgensen <toke@toke.dk> wrote:
Robert Chacon <robert.chacon@jack= rabbitwireless.com> writes:

> Hello everyone,
>
> I am new here, my name is Robert. I operate a small ISP in the US. I w= anted
> to post here to thank Dave T=C3=A4ht, as well as the dozens of contrib= utors to
> the fq_codel and cake projects.

Thank you for reaching out! It's always fun to hear about real-world deployments of this technology, and it's great to hear that it's wo= rking
well for you! :)

> I created a simple python application that uses htb+fq_codel to shape = my
> customers' traffic, and have seen great performance improvements. = I am
> maintaining it as an open source project for other ISPs to use at
> https://github.com/rchac/LibreQoS

Cool! What kind of performance are you seeing? The README mentions being limited by the BPF hash table size, but can you actually shape 2000
customers on one machine? On what kind of hardware and at what rate(s)?

-Toke


--
=20 =20 =20
=20 3D"photograph"

Robert Cha= c=C3=B3n
 Owner

=20 		=20 =20 =20 =20 =20
M (915) 730= -1472
 =20 E robert.c= hacon@jackrabbitwireless.com
JackRabbit Wireless LLC =20 =09
P.O. Box 222111
 El Paso, TX 79913 =09
jack= rabbitwireless.com
_______________________________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat


--
"For a successful technolo= gy, reality must take precedence over public relations, for Mother Nature c= annot be fooled" - Richard Feynman

dave@taht.net <Dave T=C3=A4ht> CTO, TekLib= re, LLC Tel: 1-831-435-0729
--000000000000c1fa4a05b961a6ca--