From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [IPv6:2a00:1450:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6F3163B29D for ; Mon, 7 Jun 2021 13:57:29 -0400 (EDT) Received: by mail-lf1-x12d.google.com with SMTP id m21so12008216lfg.13 for ; Mon, 07 Jun 2021 10:57:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=LOgXXDoZunFx4JqH7kZIVIn6z4z0/ZQLQicSLEnBJpE=; b=jivmpuvnsq6QZMJJZFSID06/ZR3DSf4c6v37wkLJtPaVYX1VuEoQ9Ko1yfY/5b+yHL A8yZsM3YHSIO88QvuUQCjrsk+womdViIiBLkZoOPnMKLdQVBfjqbc+CFdNry7IOH79f9 uURWEO2Ch8zG0wFpX6ed4diYe2JaAOwQsGDGpRN5pj9re1dDSIrnUTxL4mva3ahnaPK5 sQOsKbGiKbWEVc2MBlAg8xwnzrV/bD7cNOUpmw6Z4k+B5+TFbjENyxl81372m7NdA1Ir yVxPSkB8Zz8NdTecvR5olIxsOJfqjHdSIpa4ihmZTqpT64DIkkGLcsDmML2zvZ2BxhWm urAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=LOgXXDoZunFx4JqH7kZIVIn6z4z0/ZQLQicSLEnBJpE=; b=UOZJ/MyVIvrVkeHe0Yk0x1qJJ4nLph77YmlyeehyHkZyJHfd5nIK8hyCYDQfqSn2qv uojWFKQknpAZ5STFey+3a1JkGEsVeeJ7OeVWV2Y8XBMkT05TmnDRZCGd981bweINj/ba /zNDr5198yk9jK10kJQJHfaaclyszo4Z/MuRDtNUQWsoL5F4GqGtxlaFF32NZ4KwYHok 6q2ef2nu+2hJZBbo7+5kNfHgqPZqxhJJZf7dTuAyy8uD+QE2LEqPyqhG3ouCgX9QP5TD ur5nXJiJPVO95LSBq3reHKqOpZVtagX50Jy1AY8K4AHl2EbQ52M9WxOemem9QiR5+y4r wzrw== X-Gm-Message-State: AOAM530GNnXssQCxjWguFs5cvj/q73RtVNh4qITT9yGHYmmlZk5r9WZg D+mkcd+UxxVUcOllr1eBm7Q= X-Google-Smtp-Source: ABdhPJyKrcZugP+P8N0OrocPorUAp3fOs82pD2OP5RsHaRYKWZ1Kh60delcP0meSXkBmwgVOZzCuOw== X-Received: by 2002:a19:c34a:: with SMTP id t71mr12256675lff.499.1623088648087; Mon, 07 Jun 2021 10:57:28 -0700 (PDT) Received: from jonathartonsmbp.lan (87-93-133-133.bb.dnainternet.fi. [87.93.133.133]) by smtp.gmail.com with ESMTPSA id r27sm1572168lfp.95.2021.06.07.10.57.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jun 2021 10:57:27 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.7\)) From: Jonathan Morton In-Reply-To: Date: Mon, 7 Jun 2021 20:57:26 +0300 Cc: bloat Content-Transfer-Encoding: quoted-printable Message-Id: References: <20210607133853.045a96d5@babalu> To: Rich Brown X-Mailer: Apple Mail (2.3445.9.7) Subject: Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jun 2021 17:57:29 -0000 > On 7 Jun, 2021, at 8:28 pm, Rich Brown = wrote: >=20 > Saw this on the lartc mailing list... For my own information, does = anyone have thoughts, esp. for this quote: >=20 > "... when the speed comes to about 4.5Gbps download (upload is about = 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical = cores above 90% - 48x2.4Ghz if count that virtualization is on)..." This is probably the same phenomenon that limits most cheap CPE devices = to about 100Mbps or 300Mbps with software shaping, just on a bigger = scale due to running on fundamentally better hardware. My best theory to date on the root cause of this phenomenon is a = throughput bottleneck between the NIC and the system RAM via DMA, which = happens to be bypassed by a hardware forwarding engine within the NIC = (or in an external switch chip) when software shaping is disabled. I = note that 4.5Gbps is close to the capacity of a single PCIe v2 lane, so = checking the topology of the NIC's attachment to the machine might help = to confirm my theory. To avoid the problem, you'll either need to shape to a rate lower than = the bottleneck capacity, or eliminate the unexpected bottleneck by = implementing a faster connection to the NIC that can support wire-speed = transfers. - Jonathan Morton=