From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x231.google.com (mail-lf0-x231.google.com [IPv6:2a00:1450:4010:c07::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 959873BA8E for ; Mon, 6 Mar 2017 13:09:01 -0500 (EST) Received: by mail-lf0-x231.google.com with SMTP id a6so76313256lfa.0 for ; Mon, 06 Mar 2017 10:09:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=dfGptOWIP1EJweNtRAkb+HbZL80ctw4fdNqrbwHfKUM=; b=AWAifK9lQxho4AYu67WFlM7P6z/6yp+1zatSsyQ7hq6ynmI3slBra/SJypGIoAeCQK rEgH9I9D1FcUAKlRDNIFi5QwUBEc86yL4mXf5AFVAJoUqRKrHjxHz7RSQaFa603eGBiw acRI61fnhqmErAYSeceqrWjXpDWXAOBCoSjb5Xyz/Q2bLSW8bX+jurs9zey4MpPMejt8 xs8DGI41KhhOXcxmQqA4Rr5XkHmB+PHB69uq4BSsd9Sjw4VYdyYROJT5RfxKposXtmyS 3FTkcKzn1W38vJ5cEq8N/vGtSZ8hSarB7r23a9e8uoCyxyz62mu9KHPOIZcUAb7db0AM fbiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=dfGptOWIP1EJweNtRAkb+HbZL80ctw4fdNqrbwHfKUM=; b=TBX0HFFueR1uT0iIxjTxP9SgndAlL5lg8UsEnf8FT18cXU8Ni/K7pCt3xbUzA/Dair lgqLwSNgVOJZhrfZQ2WOGiQBrIasTPhcn7wO+moHbkBWZi7VJIQ0HnJgKCd0Vd/JzZNp NN0VaevXXUow/4rRUwe3/xFjKSwQoJ8YupI/4JE2tl4z7kT73ZYL4j+mWeNXYY8azcBn s3JzNi1stDbzbEROgGQMpmRqEvInkMmg5erhmZj0RMZty8Kljw7CWlKNmlmJDxoEnzCq vomfe6BlCaDPo+Kxr2Ccryw9Q23tIRDZLNBKe1D+yOuWUSg5k+5lssJjwvrT+uYiboOp 1rfg== X-Gm-Message-State: AMke39kg7l8hxrLla9xLvKSu08WuMTPxeoFKqQ8pY8ffbB0/r18Ucp8aPAeYwfkroODM1bqLL8GYtvvwIzJhPA== X-Received: by 10.46.71.198 with SMTP id u189mr5767803lja.97.1488823740424; Mon, 06 Mar 2017 10:09:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.198.7 with HTTP; Mon, 6 Mar 2017 10:08:59 -0800 (PST) In-Reply-To: <43E2473F-9DFC-48D4-A4E2-A41802E8004F@gmail.com> References: <07479F0A-40DD-44E5-B67E-28117C7CF228@gmx.de> <1488400107.3610.1@smtp.autistici.org> <2B251BF1-C965-444D-A831-9981861E453E@gmx.de> <1488484262.16753.0@smtp.autistici.org> <98E899EF-5E66-42CC-88AA-79FA80A4F228@gmail.com> <2D2AE632-75BB-4DDE-B370-0996EFECF14B@gmail.com> <43E2473F-9DFC-48D4-A4E2-A41802E8004F@gmail.com> From: Benjamin Cronce Date: Mon, 6 Mar 2017 12:08:59 -0600 Message-ID: To: Jonathan Morton Cc: Dave Taht , "cake@lists.bufferbloat.net" , Eric Luehrsen Content-Type: multipart/alternative; boundary=001a114108702e6c97054a13cac8 Subject: Re: [Cake] [LEDE-DEV] Cake SQM killing my DIR-860L - was: [17.01] Kernel: bump to 4.4.51 X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Mar 2017 18:09:01 -0000 --001a114108702e6c97054a13cac8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Depends on how short of a timescale you're talking about. Shared global state that is being read and written to very quickly by multiple threads is bad enough for a single package system, but when you start getting to something like an AMD Ryzen or NUMA, shared global state becomes really expensive. Accuracy is expensive. Loosen the accuracy and gain scalability. I would be interested in the pseduo-code or high level of what state needs to be shared and how that state is used. I was also thinking more of some hybrid. Instead of a "token" representing a bucked amount of bandwidth that can be immediately used, I was thinking more of like a "future" of bandwidth that could be used. So instead of saying "here's a token of bandwidth", you have each core doing it's own deficit bandwidth shaping, but when a token is received, a core can temporarily increase its assigned shaping bandwidth. If I remember correctly, cake already supports having its bandwidth changed on the fly. Of course it may be simpler to say cake is meant to be used on no more than 8 cores with a non-numa CPU system with all cores having a shared low-latency cache connecting the cores. On Mon, Mar 6, 2017 at 8:44 AM, Jonathan Morton wrote: > > > On 6 Mar, 2017, at 15:30, Benjamin Cronce wrote: > > > > You could treat it like task stealing, except each core can generate > tokens that represent a quantum of bandwidth that is only valid for some > interval. > > You=E2=80=99re obviously thinking of a token-bucket based shaper here. C= AKE uses > a deficit-mode shaper which deliberately works a different way - it=E2=80= =99s more > accurate on short timescales, and this actually makes a positive differen= ce > in several important cases. > > The good news is that there probably is a way to explicitly and > efficiently share bandwidth in any desired ratio across different CAKE > instances, assuming a shared-memory location can be established. I don= =E2=80=99t > presently have the mental bandwidth to actually try doing that, though. > > - Jonathan Morton > > --001a114108702e6c97054a13cac8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Depends on how short of a timescale you're talking abo= ut. Shared global state that is being read and written to very quickly by m= ultiple threads is bad enough for a single package system, but when you sta= rt getting to something like an AMD Ryzen or NUMA, shared global state beco= mes really expensive. Accuracy is expensive. Loosen the accuracy and gain s= calability.

I would be interested in the pseduo-code or = high level of what state needs to be shared and how that state is used.

I was also thinking more of some hybrid. Instead of a= "token" representing a bucked amount of bandwidth that can be im= mediately used, I was thinking more of like a "future" of bandwid= th that could be used. So instead of saying "here's a token of ban= dwidth", you have each core doing it's own deficit bandwidth shapi= ng, but when a token is received, a core can temporarily increase its assig= ned shaping bandwidth. If I remember correctly, cake already supports havin= g its bandwidth changed on the fly.

Of course it m= ay be simpler to say cake is meant to be used on no more than 8 cores with = a non-numa CPU system with all cores having a shared low-latency cache conn= ecting the cores.

On Mon, Mar 6, 2017 at 8:44 AM, Jonathan Morton <chromatix= 99@gmail.com> wrote:

> On 6 Mar, 2017, at 15:30, Benjamin Cronce <bcronce@gmail.com> wrote:
>
> You could treat it like task stealing, except each core can generate t= okens that represent a quantum of bandwidth that is only valid for some int= erval.

You=E2=80=99re obviously thinking of a token-bucket based shaper her= e.=C2=A0 CAKE uses a deficit-mode shaper which deliberately works a differe= nt way - it=E2=80=99s more accurate on short timescales, and this actually = makes a positive difference in several important cases.

The good news is that there probably is a way to explicitly and efficiently= share bandwidth in any desired ratio across different CAKE instances, assu= ming a shared-memory location can be established.=C2=A0 I don=E2=80=99t pre= sently have the mental bandwidth to actually try doing that, though.

=C2=A0- Jonathan Morton


--001a114108702e6c97054a13cac8--