[Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

Saw this on the lartc mailing list... For my own information, does anyone have thoughts, esp. for this quote:

"... when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on)..."

Thanks.

Rich


> Begin forwarded message:
> 
> From: "Ethy H. Brito" <ethy.brito@inexo.com.br>
> Subject: Traffic shaping at 10~300mbps at a 10Gbps link
> Date: June 7, 2021 at 12:38:53 PM EDT
> To: lartc <lartc@vger.kernel.org>
> 
> 
> Hi
> 
> I am having a hard time trying to shape 3000 users at ceil speeds from 10 to 300mbps in a 7/7Gbps link using HTB+SFQ+TC(filter by IP hashkey mask) for a few days now tweaking HTB and SFQ parameters with no luck so far.
> 
> Everything seems right, up 4Gbps overall download speed with shaping on.
> I have no significant packets delay, no dropped packets and no high CPU average loads (not more than 20% - htop info)
> 
> But when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in.
> CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on) and as a consequence packets are dropped (as reported by tc -s class sh ...), RTT goes above 200ms and a lots of ungry users. This goes from about 7PM to 11 PM every day.
> 
> If I turn shaping off, everything return to normality immediately and peaks of not more than 5Gbps (1 second average) are observed and a CPU load of about 5%. So I infer the uplink is not crowded.
> 
> I use one root HTB qdisc and one root (1:) HTB class.
> Then about 20~30 same level (1:xx) inner classes to (sort of) separate the users by region 
> And under these inner classes, goes the almost 3000 leaves (1:xxxx). 
> I have one class with about 900 users and this quantity decreases by the other inner classes having some of them with just one user.
> 
> Is the way I'm using HTB+SFQ+TC suitable for this job?
> 
> Since the script that creates the shaping environment is too long I do not post it here.
> 
> What can I inform you guys to help me solve this?
> Fragments of code, stats, some measurements? What?
> 
> Thanks.
> 
> Regards
> 
> Ethy


[-- Attachment #2: Type: text/html, Size: 4795 bytes --]

Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Jonathan Morton]

> On 7 Jun, 2021, at 8:28 pm, Rich Brown <richb.hanover@gmail.com> wrote:
> 
> Saw this on the lartc mailing list... For my own information, does anyone have thoughts, esp. for this quote:
> 
> "... when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on)..."

This is probably the same phenomenon that limits most cheap CPE devices to about 100Mbps or 300Mbps with software shaping, just on a bigger scale due to running on fundamentally better hardware.

My best theory to date on the root cause of this phenomenon is a throughput bottleneck between the NIC and the system RAM via DMA, which happens to be bypassed by a hardware forwarding engine within the NIC (or in an external switch chip) when software shaping is disabled.  I note that 4.5Gbps is close to the capacity of a single PCIe v2 lane, so checking the topology of the NIC's attachment to the machine might help to confirm my theory.

To avoid the problem, you'll either need to shape to a rate lower than the bottleneck capacity, or eliminate the unexpected bottleneck by implementing a faster connection to the NIC that can support wire-speed transfers.

 - Jonathan Morton

Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Jesper Dangaard Brouer]

Hi Rich,

Quote:
> > I use one root HTB qdisc and one root (1:) HTB class.

Sounds like a classical case of lock congestion on the TC-root qdisc
lock.  I have a solution using XDP here[1] combined with TC. Google
have also hit the problem, they solved it differently, specific to
their use-case.

[1] https://github.com/xdp-project/xdp-cpumap-tc


On Mon, 7 Jun 2021 13:28:10 -0400
Rich Brown <richb.hanover@gmail.com> wrote:

> Saw this on the lartc mailing list... For my own information, does
> anyone have thoughts, esp. for this quote:
> 
> "... when the speed comes to about 4.5Gbps download (upload is about
> 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz
> physical cores above 90% - 48x2.4Ghz if count that virtualization is
> on)..."
> 
> Thanks.
> 
> Rich
> 
> 
> > Begin forwarded message:
> > 
> > From: "Ethy H. Brito" <ethy.brito@inexo.com.br>
> > Subject: Traffic shaping at 10~300mbps at a 10Gbps link
> > Date: June 7, 2021 at 12:38:53 PM EDT
> > To: lartc <lartc@vger.kernel.org>
> > 
> > 
> > Hi
> > 
> > I am having a hard time trying to shape 3000 users at ceil speeds
> > from 10 to 300mbps in a 7/7Gbps link using HTB+SFQ+TC(filter by IP
> > hashkey mask) for a few days now tweaking HTB and SFQ parameters
> > with no luck so far.
> > 
> > Everything seems right, up 4Gbps overall download speed with
> > shaping on. I have no significant packets delay, no dropped packets
> > and no high CPU average loads (not more than 20% - htop info)
> > 
> > But when the speed comes to about 4.5Gbps download (upload is about
> > 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz
> > physical cores above 90% - 48x2.4Ghz if count that virtualization
> > is on) and as a consequence packets are dropped (as reported by tc
> > -s class sh ...), RTT goes above 200ms and a lots of ungry users.
> > This goes from about 7PM to 11 PM every day.
> > 
> > If I turn shaping off, everything return to normality immediately
> > and peaks of not more than 5Gbps (1 second average) are observed
> > and a CPU load of about 5%. So I infer the uplink is not crowded.
> > 
> >
> > I use one root HTB qdisc and one root (1:) HTB class.
> >
> > Then about 20~30 same level (1:xx) inner classes to (sort of) separate the users by region 
> > And under these inner classes, goes the almost 3000 leaves (1:xxxx). 
> >
> > I have one class with about 900 users and this quantity decreases
> > by the other inner classes having some of them with just one user.
> > 
> > Is the way I'm using HTB+SFQ+TC suitable for this job?
> > 
> > Since the script that creates the shaping environment is too long I do not post it here.
> > 
> > What can I inform you guys to help me solve this?
> > Fragments of code, stats, some measurements? What?
> > 
> > Thanks.
> > 
> > Regards
> > 
> > Ethy  
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 7 Jun, 2021, at 8:28 pm, Rich Brown <richb.hanover@gmail.com> wrote:
>> 
>> Saw this on the lartc mailing list... For my own information, does anyone have thoughts, esp. for this quote:
>> 
>> "... when the speed comes to about 4.5Gbps download (upload is about 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz physical cores above 90% - 48x2.4Ghz if count that virtualization is on)..."
>
> This is probably the same phenomenon that limits most cheap CPE
> devices to about 100Mbps or 300Mbps with software shaping, just on a
> bigger scale due to running on fundamentally better hardware.
>
> My best theory to date on the root cause of this phenomenon is a
> throughput bottleneck between the NIC and the system RAM via DMA,
> which happens to be bypassed by a hardware forwarding engine within
> the NIC (or in an external switch chip) when software shaping is
> disabled. I note that 4.5Gbps is close to the capacity of a single
> PCIe v2 lane, so checking the topology of the NIC's attachment to the
> machine might help to confirm my theory.
>
> To avoid the problem, you'll either need to shape to a rate lower than
> the bottleneck capacity, or eliminate the unexpected bottleneck by
> implementing a faster connection to the NIC that can support
> wire-speed transfers.

I very much doubt this has anything to do with system bottlenecks. We
hit the PCIe bottleneck when trying to push 100Gbit through a server, 5
Gbps is trivial for a modern device.

Rather, as Jesper pointed out this sounds like root qdisc lock
contention...

-Toke