From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id F17F03B2A4 for ; Mon, 16 Sep 2019 08:00:50 -0400 (EDT) Received: by mail-io1-xd44.google.com with SMTP id j4so77937594iog.11 for ; Mon, 16 Sep 2019 05:00:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=P9rAc0sJGhgUsIImweWTsJpv8BA9zDVP/lfpvXWUC8c=; b=nh4HUyZFJ0KoMjNumaIlz5Gx2ECuJyuecC5F/8541wXKBYXoalBSOd2EsGAYplM8q3 /TI98qCluN7hPOpTjwtWrlAgh266607pvxHf0r2Zgh/amWg6ndft1IPtHUs2+MAVMQC0 5ewpa8OpFmptz2h5XP3RE0jaSFAObI6yev8h+huIOND/+UjgmGmYEoGUSFAI8+2vQPkV jvqscYyw8jwOGOnAQgd7g9CnkRcyh+iF7XwYw40CAVc4+m8qAu1jhKDKqGWQ/o/waOD7 xZzyOy1bsMgon92p/UVt9HXFbsaO3YgBakSQc1Y2xOYPFHr01qGdIrdzOUHB3o2ekxQU CYYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=P9rAc0sJGhgUsIImweWTsJpv8BA9zDVP/lfpvXWUC8c=; b=UKyrGuL6aJUc8QvOMlc7FucQSYy9g2R9/GoBuuhdw5luFO4cSv9yAm3cR83BmuDcX6 H039CvLQmJP2jkJRb4jAOEG8fx0kKbS21tI2SJKRaaRO1sBE40na90JQZC8nDmr1ntww NVou4FCh1y/JN8f6nmCjEc2/3mZQ2qyGtq3XWqnDS0rj/MVEZGGU7c1xi/Pnd2Irl5f1 gnUhFgFfdtHjSvTno17yywrVrfEANDxX+1RSdtwNcIqlZJma/mojabeEAIixurIT0Nqa QPEm6nY7+PdURRHhy00GoSAXXAbriOwHmAdFMH8K50HC0qLYvyH2agJVEV2LU+J5SwOz VGLA== X-Gm-Message-State: APjAAAVPnU2ReHL0vzVqxZ6vlKxB6iEXQJQdZz1QagV/LWshYqxwauCG jXawcPTm3vq2CfcrGBjD9Pz9rUHdraHP09Ocfxsmtear X-Google-Smtp-Source: APXvYqx4tbTz9a/6ooC19JDrZaXDXjFcyfs5EvDOkcKE+p27u34jrxdLfSnCh3hKli2VmpXFl4WwBIObuMh9e4HOaZY= X-Received: by 2002:a6b:5814:: with SMTP id m20mr15462217iob.249.1568635250052; Mon, 16 Sep 2019 05:00:50 -0700 (PDT) MIME-Version: 1.0 References: <2825CE14-2109-4580-A086-9701F4D3ADF0@gmail.com> <18b1c174-b88d-4664-9aa8-9c42925fc14c@www.fastmail.com> <9a90111b-2389-4dc6-8409-18c40f895540@www.fastmail.com> <43F02160-E691-4393-A0C0-8AB4AD962700@gmail.com> <057ea708-6797-23f5-ef01-9d3d7b002578@newmedia-net.de> In-Reply-To: <057ea708-6797-23f5-ef01-9d3d7b002578@newmedia-net.de> From: Dave Taht Date: Mon, 16 Sep 2019 13:00:38 +0100 Message-ID: To: Sebastian Gottschall Cc: Cake List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] cake memory consumption X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Sep 2019 12:00:51 -0000 I am puzzled as to why fq_codel_fast would use more ram than fq_codel would, was sce (gso-splotting) enabled? similarly, the differences between hfsc and htb are interesting. I don't get that either. How many cake instances are being created? And for the sake of discussion, what does cake standalone consume? On Mon, Sep 16, 2019 at 11:22 AM Sebastian Gottschall wrote: > > after we found out serious out of memory issues on smaller embedded devic= es (128 mb ram) we made some benchmarks with different schedulers > with the result that cake takes a serious amount of memory. we use the ou= t of tree cake module and we use it class based since we have complex metho= ds of doing qos per interface, per mac addresse or even per I note that I often thought about having mac address functionality might be a valuable mode for cake. >ip/network. so its not just simple cake on a single interface solution. we= made some benchmarks with different schedulers. does anybody have a soluti= on for making that better? With such complexity required I'd stick to hfsc + fq_X rather than layer in cake. Understanding the model (sh -x the tc commands for, say, hfsc + something and htb + something ) your users require, though, would be helpful. We tried to design cake so that a jillion optimizations such as ack prioritization, per network fq (instead per flow/per host) - but we couldn't possibly cover all use cases in it with out more feedback from the field. Still... such a big difference in memory use doesn't add up. Cake has a larger fixed memory allocation than fq_codel, but the rest is just packets which come from global memory. Can you point to a build and a couple targets we could try? I am presently travelling (in portugal) and won't be back online until later this week. > > HTB/FQ_CODEL -------=EF=83=A0 62M > HTB/SFQ -------=EF=83=A0 62M > HTB/PIE -------=EF=83=A0 62M > HTB/FQ_CODEL_FAST -------=EF=83=A0 67M > HTB/CAKE -------=EF=83=A0111M > > HFSC/FQ_CODEL_FAST -------=EF=83=A0 47M > HTB/PIE -------=EF=83=A0 49M > HTB/SFQ -------=EF=83=A0 50M > HFSC /FQ_CODEL -------=EF=83=A0 52M > HFSC/CAKE -------=EF=83=A0109M > > > consider that the benchmark doesnt show the real values. its system overa= ll and does not consider memory taken by the wireless driver for instance w= hich is about 45 mb of ram for ath10k > so this makes all even more worse unfortunatly since there is not that ma= ny ram left for cake. just about 70mb maybe. > Am 08.09.2019 um 19:27 schrieb Jonathan Morton: > > You could also set it back to 'internet' and progressively reduce the > bandwidth parameter, making the Cake shaper into the actual bottleneck. > This is the correct fix for the problem, and you should notice an > instant improvement as soon as the bandwidth parameter is correct. > > Hand tuning this one link is not a problem. I'm searching for a set of se= ttings that will provide generally good performance across a wide range of = devices, links, and situations. > > From what you've indicated so far there's nothing as effective as a corre= ct bandwidth estimation if we consider the antenna (link) a black box. Expe= cting the user to input expected throughput for every link and then managin= g that information is essentially a non-starter. > > Radio tuning provides some improvement, but until ubiquiti starts shippin= g with Codel on non-router devices I don't think there's a good solution he= re. > > Any way to have the receiving device detect bloat and insert an ECN? > > That's what the qdisc itself is supposed to do. > > I don't think the time spent in the intermediate device is detectable at = the kernel level but we keep track of latency for routing decisions and cou= ld detect bloat with some accuracy, the problem is how to respond. > > As long as you can detect which link the bloat is on (and in which direct= ion), you can respond by reducing the bandwidth parameter on that half-link= by a small amount. Since you have a cooperating network, maintaining a ti= me standard on each node sufficient to observe one-way delays seems feasibl= e, as is establishing a normal baseline latency for each link. > > The characteristics of the bandwidth parameter being too high are easy to= observe. Not only will the one-way delay go up, but the received throughp= ut in the same direction at the same time will be lower than configured. Y= ou might use the latter as a hint as to how far you need to reduce the shap= ed bandwidth. > > Deciding when and by how much to *increase* bandwidth, which is presumabl= y desirable when link conditions improve, is a more difficult problem when = the link hardware doesn't cooperate by informing you of its status. (This = is something you could reasonably ask Ubiquiti to address.) > > I would assume that link characteristics will change slowly, and run an o= ccasional explicit bandwidth probe to see if spare bandwidth is available. = If that probe comes through without exhibiting bloat, *and* the link is ot= herwise loaded to capacity, then increase the shaper by an amount within th= e probe's capacity of measurement - and schedule a repeat. > > A suitable probe might be 100x 1500b packets paced out over a second, byp= assing the shaper. This will occupy just over 1Mbps of bandwidth, and can = be expected to induce 10ms of delay if injected into a saturated 100Mbps li= nk. Observe the delay experienced by each packet *and* the quantity of oth= er traffic that appears between them. Only if both are favourable can you = safely open the shaper, by 1Mbps. > > Since wireless links can be expected to change their capacity over time, = due to eg. weather and tree growth, this seems to be more generally useful = than a static guess. You could deploy a new link with a conservative "gues= s" of say 10Mbps, and just probe from there. > > - Jonathan Morton > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740