[Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Jonathan Morton]

> On 7 Jun, 2021, at 8:28 pm, Rich Brown <richb.hanover at gmail.com> wrote:
> 
> Saw this on the lartc mailing list... For my own information, does anyone have thoughts, esp. for this quote:
> 
> "... when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on)..."

This is probably the same phenomenon that limits most cheap CPE devices to about 100Mbps or 300Mbps with software shaping, just on a bigger scale due to running on fundamentally better hardware.

My best theory to date on the root cause of this phenomenon is a throughput bottleneck between the NIC and the system RAM via DMA, which happens to be bypassed by a hardware forwarding engine within the NIC (or in an external switch chip) when software shaping is disabled.  I note that 4.5Gbps is close to the capacity of a single PCIe v2 lane, so checking the topology of the NIC's attachment to the machine might help to confirm my theory.

To avoid the problem, you'll either need to shape to a rate lower than the bottleneck capacity, or eliminate the unexpected bottleneck by implementing a faster connection to the NIC that can support wire-speed transfers.

 - Jonathan Morton