From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AA2D93B2A4 for ; Mon, 16 Sep 2019 09:28:53 -0400 (EDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id DBA0E220DB for ; Mon, 16 Sep 2019 09:28:52 -0400 (EDT) Received: from imap2 ([10.202.2.52]) by compute4.internal (MEProxy); Mon, 16 Sep 2019 09:28:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=althea.net; h= mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type:content-transfer-encoding; s=fm1; bh=n/po+ 3FUuskh3xFi7Sj10ZHyltA0vv42MkLuSAtwB88=; b=e1tQBWQ4JFxxQg81i3+Tk bsuGQO36G+S8iYkqKkQJpANHsLD72d/a9dE0+do9fPlKVoC1kT+Gr/WX2zLx6PPx JFBj41GVKoPHE1VG21vWSm485x0jnUyXNpqRCF0nMMPDL2F8FQf8bc3iv96OjPmd +acuvPP70RG6GVqWRDnC2WF6aUjtqQtXTVJXPp69crdQP1l8bI6X6XMm9mEhbzeL g2bKZlndhPLBlMGBANB7k4/d+xMoMaJwGUPd2X3feAoD7vlio3y2tqnX9q6+nC3B 1x4DFFfRTcOFU/2LVHWn0UuRGndYvyUX9crC7d3vx5MMdf0j0aY0Fk8ywWtmTEp4 A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=n/po+3FUuskh3xFi7Sj10ZHyltA0vv42MkLuSAtwB 88=; b=Qv5i+baD6bV5BcbFKrc1XIFDkifcYp5T5W0nBTZ/GN5xkj5SrBcH/SsXb fJEgMmXZx3r3HP2o+Eb6o8Jfoag6yT23LnqwdSvz1y1Wvo2pvNq/bbk8PO2/nuNa b7G619jEDFHWeEsGEZ6JxvIA3G4/OK6ahx1MznLK/xX01TgUk573GWDOPbAmo5qm HKehVmAHNhYHIcZwVFARS3QKuqzzNPFjLZL8W6oCqo4NK7W+hzjwEuz4uE3kv8Ur WkifH4Hz2cG8vblqLm6yLVOyKBhHL1OmHqn09QusrVO3DoV0bF8T26HTkdapp0ki TtulWOlYR4I69om+ehluN+x+oYAww== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrudefgdeihecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgfgsehtqh ertderreejnecuhfhrohhmpedflfhushhtihhnucfmihhlphgrthhrihgtkhdfuceojhhu shhtihhnsegrlhhthhgvrgdrnhgvtheqnecuffhomhgrihhnpegsuhhffhgvrhgslhhorg htrdhnvghtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehjuhhsthhinhesrghlthhhvggr rdhnvghtnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id DA215E00B3; Mon, 16 Sep 2019 09:28:51 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.7-237-gf35468d-fmstable-20190912v1 Mime-Version: 1.0 Message-Id: <93040b26-aa44-41b9-abec-7e1f637c0cb1@www.fastmail.com> In-Reply-To: References: <2825CE14-2109-4580-A086-9701F4D3ADF0@gmail.com> <18b1c174-b88d-4664-9aa8-9c42925fc14c@www.fastmail.com> <9a90111b-2389-4dc6-8409-18c40f895540@www.fastmail.com> <43F02160-E691-4393-A0C0-8AB4AD962700@gmail.com> <057ea708-6797-23f5-ef01-9d3d7b002578@newmedia-net.de> Date: Mon, 16 Sep 2019 09:28:31 -0400 From: "Justin Kilpatrick" To: cake@lists.bufferbloat.net Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] cake memory consumption X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Sep 2019 13:28:53 -0000 I'm not seeing anything like the memory usage you describe in a similar = situation.=20 OpenWRT 18.06.4 on a glb1300 and 10+ virtual interfaces with cake. Total= memory usage is 70MB for everything.=20 --=20 Justin Kilpatrick justin@althea.net On Mon, Sep 16, 2019, at 9:22 AM, Sebastian Gottschall wrote: >=20 > Am 16.09.2019 um 14:00 schrieb Dave Taht: > > I am puzzled as to why fq_codel_fast would use more ram than fq_code= l > > would, was sce (gso-splotting) enabled? > that can by typical error tollerance. he just used "free" for comparis= ation > > > > similarly, the differences between hfsc and htb are interesting. I > > don't get that either. > > > > How many cake instances are being created? > according to his config, i assume 7 > > > > And for the sake of discussion, what does cake standalone consume? > thats a rare condition for my testers. this is something for PC's but=20= > not for routers :-) > this is something i need to find out for myself on my routers > > > > On Mon, Sep 16, 2019 at 11:22 AM Sebastian Gottschall > > wrote: > >> after we found out serious out of memory issues on smaller embedded= devices (128 mb ram) we made some benchmarks with different schedulers > >> with the result that cake takes a serious amount of memory. we use = the out of tree cake module and we use it class based since we have comp= lex methods of doing qos per interface, per mac addresse or even per > > I note that I often thought about having mac address functionality > > might be a valuable mode for cake. > that wouldnt help. there are many variations with multiple different=20= > settings for different mac addresses. as far as i have seen cake is no= t=20 > designed to work like this. this is why we > have to use a class / qdisc tree in my case > > > >> ip/network. so its not just simple cake on a single interface solut= ion. we made some benchmarks with different schedulers. does anybody hav= e a solution for making that better? > > With such complexity required I'd stick to hfsc + fq_X rather than > > layer in cake. > yea. i told that too. but people complain that cake runs soooooooo muc= h=20 > better. or at least a little bit. hard to get around this argument > > > > Understanding the model (sh -x the tc commands for, say, hfsc + > > something and htb + something ) your users require, though, would be= > > helpful. We tried to design cake so that a jillion optimizations suc= h > > as ack prioritization, per network fq (instead per flow/per host) - > > but we couldn't possibly cover all use cases in it with out more > > feedback from the field. > > > > Still... such a big difference in memory use doesn't add up. Cake ha= s > > a larger fixed memory allocation > 4 mb max as i have seen. but by 7 its coming up to 28. but i still see= =20 > much more here. consider that i implemented the same limitation to=20 > fq_codel and also fq_codel_fast > (model specific. on bigger devices i dont restrict he memory to 4 mb) > > than fq_codel, but the rest is just packets which come from global m= emory. > > > > Can you point to a build and a couple targets we could try? I am > > presently travelling (in portugal) and won't > > be back online until later this week. > what do you mean with targets? the build for testing was always the=20= > same. i requested todo the test just with multiple schedulers which is= =20 > switchable in my gui. >=20 > what i can do is doing a tree like print to visualize how its builded=20= > (or i simple print you out the qdisc/class/filters) >=20 > the test itself was made on a tplink archer c7 v2. >=20 > >> HTB/FQ_CODEL -------=EF=83=A0 62M > >> HTB/SFQ -------=EF=83=A0 62M > >> HTB/PIE -------=EF=83=A0 62M > >> HTB/FQ_CODEL_FAST -------=EF=83=A0 67M > >> HTB/CAKE -------=EF=83=A0111M > >> > >> HFSC/FQ_CODEL_FAST -------=EF=83=A0 47M > >> HTB/PIE -------=EF=83=A0 49M > >> HTB/SFQ -------=EF=83=A0 50M > >> HFSC /FQ_CODEL -------=EF=83=A0 52M > >> HFSC/CAKE -------=EF=83=A0109M > >> > >> > >> consider that the benchmark doesnt show the real values. its system= overall and does not consider memory taken by the wireless driver for i= nstance which is about 45 mb of ram for ath10k > >> so this makes all even more worse unfortunatly since there is not t= hat many ram left for cake. just about 70mb maybe. > >> Am 08.09.2019 um 19:27 schrieb Jonathan Morton: > >> > >> You could also set it back to 'internet' and progressively reduce t= he > >> bandwidth parameter, making the Cake shaper into the actual bottlen= eck. > >> This is the correct fix for the problem, and you should notice an > >> instant improvement as soon as the bandwidth parameter is correct. > >> > >> Hand tuning this one link is not a problem. I'm searching for a set= of settings that will provide generally good performance across a wide = range of devices, links, and situations. > >> > >> From what you've indicated so far there's nothing as effective as = a correct bandwidth estimation if we consider the antenna (link) a black= box. Expecting the user to input expected throughput for every link and= then managing that information is essentially a non-starter. > >> > >> Radio tuning provides some improvement, but until ubiquiti starts s= hipping with Codel on non-router devices I don't think there's a good so= lution here. > >> > >> Any way to have the receiving device detect bloat and insert an ECN= ? > >> > >> That's what the qdisc itself is supposed to do. > >> > >> I don't think the time spent in the intermediate device is detectab= le at the kernel level but we keep track of latency for routing decision= s and could detect bloat with some accuracy, the problem is how to respo= nd. > >> > >> As long as you can detect which link the bloat is on (and in which = direction), you can respond by reducing the bandwidth parameter on that = half-link by a small amount. Since you have a cooperating network, main= taining a time standard on each node sufficient to observe one-way delay= s seems feasible, as is establishing a normal baseline latency for each = link. > >> > >> The characteristics of the bandwidth parameter being too high are e= asy to observe. Not only will the one-way delay go up, but the received= throughput in the same direction at the same time will be lower than co= nfigured. You might use the latter as a hint as to how far you need to = reduce the shaped bandwidth. > >> > >> Deciding when and by how much to *increase* bandwidth, which is pre= sumably desirable when link conditions improve, is a more difficult prob= lem when the link hardware doesn't cooperate by informing you of its sta= tus. (This is something you could reasonably ask Ubiquiti to address.) > >> > >> I would assume that link characteristics will change slowly, and ru= n an occasional explicit bandwidth probe to see if spare bandwidth is av= ailable. If that probe comes through without exhibiting bloat, *and* th= e link is otherwise loaded to capacity, then increase the shaper by an a= mount within the probe's capacity of measurement - and schedule a repeat= . > >> > >> A suitable probe might be 100x 1500b packets paced out over a secon= d, bypassing the shaper. This will occupy just over 1Mbps of bandwidth,= and can be expected to induce 10ms of delay if injected into a saturate= d 100Mbps link. Observe the delay experienced by each packet *and* the = quantity of other traffic that appears between them. Only if both are f= avourable can you safely open the shaper, by 1Mbps. > >> > >> Since wireless links can be expected to change their capacity over = time, due to eg. weather and tree growth, this seems to be more generall= y useful than a static guess. You could deploy a new link with a conser= vative "guess" of say 10Mbps, and just probe from there. > >> > >> - Jonathan Morton > >> _______________________________________________ > >> Cake mailing list > >> Cake@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/cake > >> > >> _______________________________________________ > >> Cake mailing list > >> Cake@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/cake > > > > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake >