From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [45.145.95.4]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B11C43B29D for ; Mon, 7 Jun 2021 18:20:04 -0400 (EDT) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1623104401; bh=pC0p1b2jen6T8umnBwZJwpOfvs1eFT0cAYTJzHviiew=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=LsdF3RJ74RLxe0WwHSahS1o9J/TUlqiCUdc7Fg8obyNOnMXelFReWCWQMlMWyQ6QT e0Pb5Bn+idp2GETwBb1ZSZqoY6VXopanehcnr77lu9gawmkUR0ifYcjN+j6I8HTpRg gD+YLSPSwLtR3pCNyC3gY5yOd9HIXCdM0Vbe+GnpCkLGRUgKwYh1X4GYk6bOLddWJ1 vIEIHwvbkZVKtPoLJlcAVzF8ob4+rHBmmCsAM/uHo5zxTwD24+05zYLr3ZWSaeiy+2 7sT/xK1Ws28vPkNrEknDoAS4LtxW/s1hpt26ct38rfBd/Z44w3tQ10Aqmx7TZmnsSx exxrRLdLJkJjw== To: Jonathan Morton , Rich Brown Cc: bloat In-Reply-To: References: <20210607133853.045a96d5@babalu> Date: Tue, 08 Jun 2021 00:20:00 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87zgw1tizz.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jun 2021 22:20:04 -0000 Jonathan Morton writes: >> On 7 Jun, 2021, at 8:28 pm, Rich Brown wrote: >> >> Saw this on the lartc mailing list... For my own information, does anyone have thoughts, esp. for this quote: >> >> "... when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on)..." > > This is probably the same phenomenon that limits most cheap CPE > devices to about 100Mbps or 300Mbps with software shaping, just on a > bigger scale due to running on fundamentally better hardware. > > My best theory to date on the root cause of this phenomenon is a > throughput bottleneck between the NIC and the system RAM via DMA, > which happens to be bypassed by a hardware forwarding engine within > the NIC (or in an external switch chip) when software shaping is > disabled. I note that 4.5Gbps is close to the capacity of a single > PCIe v2 lane, so checking the topology of the NIC's attachment to the > machine might help to confirm my theory. > > To avoid the problem, you'll either need to shape to a rate lower than > the bottleneck capacity, or eliminate the unexpected bottleneck by > implementing a faster connection to the NIC that can support > wire-speed transfers. I very much doubt this has anything to do with system bottlenecks. We hit the PCIe bottleneck when trying to push 100Gbit through a server, 5 Gbps is trivial for a modern device. Rather, as Jesper pointed out this sounds like root qdisc lock contention... -Toke