From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 33AF53B29E for ; Tue, 23 Jun 2020 10:41:43 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1592923302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1vq7k4gCAnEYjiSSLCBQjxYUqup6Q3xlwtC4udBa0x0=; b=HxmKkAd1MssrF+h36STx+gbiDY6xUzSa1CwQGSDf3AcA2Tl3wrgNO9MD2nMq8aR5zYOw0z tJ2Di8Ajl3TnY3WoFMSjkJFvDJZJd5LBNpVuoJhvZNdga+6inQXxa1C4B7zrmvA8vMimpP D3bB2Esqjzd3q35aFsmzre8w0+tIEUc= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-285-TBJI-FRhO-WZx_92FtIw8Q-1; Tue, 23 Jun 2020 10:41:38 -0400 X-MC-Unique: TBJI-FRhO-WZx_92FtIw8Q-1 Received: by mail-wm1-f70.google.com with SMTP id c66so4755440wma.8 for ; Tue, 23 Jun 2020 07:41:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=1vq7k4gCAnEYjiSSLCBQjxYUqup6Q3xlwtC4udBa0x0=; b=Qros9sgW3J+pHGxNrtg0rujH96/u7nsnqjaMJ+fqJul9pRzgJGQwuyvQS2UetgCU/Y D+35y2PvfiPhyF583a3VyDax0oOzW3RQWG0sUhZpB8Y/W8Mw8/MxkbLHs7cGnJ5DZKS3 9Ru/yDWMFxy4uzyXv1N/jteLV692KCJV0k84dUI4W1rxCaZvfsPMoL8z7mdaEadI5T5L YiZZE98ZlX5uTmyKbZalr7D/EkbsfzL4yk59IsW5hcx9sO/vlZv/HXxnlOINcQT315o7 eYgw7i9vopCPYmRpfIzIPSt2kV+9j32mEXqUoua9CLfpJl49d7up6nvsanYqZ7LE/EAf lQVw== X-Gm-Message-State: AOAM530d0glWkUj7X8WQ2koSwChrgsWbQGv3J/YdDGWOSGCjQ34kkUSw qzgGuYCThrp/U2rzUt5gETZ9MNE32JpQZnehrfuVVlg9Nn3UDtaRae+a+ZbFDYvxYg4/ZH3DiSF cOCE0Ag5ljek2MljdzVunJw== X-Received: by 2002:a1c:b4c2:: with SMTP id d185mr2194848wmf.57.1592923297481; Tue, 23 Jun 2020 07:41:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyJ15/etNgDYKS1RNNjG+o4VaYMidoiFjSCWWxakn3VmTupwHjdKAI/H6zB3g9q6CrkaNSrLQ== X-Received: by 2002:a1c:b4c2:: with SMTP id d185mr2194826wmf.57.1592923297166; Tue, 23 Jun 2020 07:41:37 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id g18sm3982074wme.17.2020.06.23.07.41.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jun 2020 07:41:36 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id C5420181502; Tue, 23 Jun 2020 16:41:34 +0200 (CEST) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= To: Jose Blanquicet Cc: cake@lists.bufferbloat.net, marco maniezzo In-Reply-To: References: <877dvzt84w.fsf@toke.dk> X-Clacks-Overhead: GNU Terry Pratchett Date: Tue, 23 Jun 2020 16:41:34 +0200 Message-ID: <87lfkdrgip.fsf@toke.dk> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2020 14:41:43 -0000 Jose Blanquicet writes: > Hi Toke, > > Thanks for your reply. > > On Mon, Jun 22, 2020 at 5:47 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: >> > We have an embedded system with limited CPU resources that acts as a >> > gateway to provide Internet access from LTE to a private USB-NCM >> > network (And also to a Wi-Fi private network but we will work on it >> > later). Our problem is that the bandwidth on LTE and USB link is >> > higher than what the system is able to handle thus it reaches 100% of >> > CPU load when we perform a simple speed test from a device on the >> > private network. >> >> What speeds were you getting without shaping? > > Between 35 and 40Mbps. > >> > Therefore, we want to limit the bandwidth to avoid system getting >> > saturated in such use-case. To do so, we thought to use the CAKE on >> > the USB interface. For instance, we tried: >> > >> > tc qdisc replace root dev eth0 cake bandwidth 20mbit ethernet >> > internet flowblind nonat besteffort nowash >> > >> > It worked correctly and the maximum rate was limited but there are two >> > things that are worrying us: >> > >> > 1) The maximum rate reached after applying CAKE was in between 12Mbps >> > and 15Mbps which is quite lower than the 20Mbps we are configuring, we >> > were expecting around 18-19. Why? Is there something in the parameters >> > we are doing wrong? Please take into account that our goal is to limit >> > the rate but adding as little CPU load as possible. >> >> Hmm, are you actually running out of CPU? I.e., is the CPU pegged at >> 100% when you hit this limit? What kind of platform are you running on? >> And what kernel and CAKE versions are you using? > > I checked the CPU with top and there is still free CPU to be used. We > also tried with lower values like 10 and it is again far away from the > configured limit. > > We have just a percentage of an ARM Cortex A7 (1.2GHz) because the > rest is reserved for modem. We are now trying to optimize all the > applications in the system but LTE<->WIFI/USB data transfer is indeed > the > use-case that puts our system in crisis. > > The kernel version is 3.18 and for we are using the latest commit on > master branch (9d79e2b) for CAKE. In case, we could change CAKE but > not the kernel version, at most some specific patches. Right, well if you're not running out of CPU I guess it could be a timing issue. The CAKE shaper relies on accurate timestamps and the qdisc watchdog timer to schedule the transmission of packets. A loaded system can simply miss deadlines, which would be consistent with the behaviour you're seeing. In fact, when looking into this a bit more, I came across this commit that seemed to observe the same behaviour in sch_fq: https://git.kernel.org/torvalds/c/fefa569a9d4b So I guess we could try to do something similar in CAKE. Could you please post the output of 'tc -s qdisc' after a test run? That should give some indication on how much the shaper is throttling... >> > 2) The CPU load added by CAKE was not negligible for our system. In >> > fact, we compared the CPU load when limitation was done by CAKE and by >> > the device on the private network, e.g. curl tool with parameter >> > "--limit-rate". As a result, we found that the CPU load when using >> > CAKE was 30%. Is there any way to make it lighter with a different >> > configuration? >> >> No, you've already turned off most of the features that might incur >> overhead, so I don't think there's anything more you can do >> configuration-wise to improve CPU load. Shaping does tend to use up a >> lot of CPU, so it's not too surprising you run into issues here. > > Could you please help us to identify which one is still active? We > thought we had already turned off all the features not needed to apply > a limitation with a single queue (Besteffor mode). Well the only thing more you can turn off by configuration is the shaper itself :) >> We did recently get a pull request whose author states that he was >> seeing a 1/3 improvement in performance from it. See: >> https://github.com/dtaht/sch_cake/pull/136 >> >> You could try this; if your ingress network device driver has the same >> issue with skbs being allocated in smaller bits, you may see a similar >> increase with this patch. For a quick test you could also just try >> commenting out the call to cake_handle_diffserv() entirely since you're >> running in besteffort mode anyway :) > > Interesting. We will try this, we commented out the call to > cake_handle_diffserv() as you said and just to be sure, we also > applied the 2nd commit of the PR. I will be back soon with news. OK, great! -Toke