Re: [Bloat] Fwd: Traffic shaping at 10~300mbps at a 10Gbps link

[Jesper Dangaard Brouer <brouer@redhat.com>]

Hi Rich,

Quote:
> > I use one root HTB qdisc and one root (1:) HTB class.

Sounds like a classical case of lock congestion on the TC-root qdisc
lock.  I have a solution using XDP here[1] combined with TC. Google
have also hit the problem, they solved it differently, specific to
their use-case.

[1] https://github.com/xdp-project/xdp-cpumap-tc


On Mon, 7 Jun 2021 13:28:10 -0400
Rich Brown <richb.hanover@gmail.com> wrote:

> Saw this on the lartc mailing list... For my own information, does
> anyone have thoughts, esp. for this quote:
> 
> "... when the speed comes to about 4.5Gbps download (upload is about
> 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz
> physical cores above 90% - 48x2.4Ghz if count that virtualization is
> on)..."
> 
> Thanks.
> 
> Rich
> 
> 
> > Begin forwarded message:
> > 
> > From: "Ethy H. Brito" <ethy.brito@inexo.com.br>
> > Subject: Traffic shaping at 10~300mbps at a 10Gbps link
> > Date: June 7, 2021 at 12:38:53 PM EDT
> > To: lartc <lartc@vger.kernel.org>
> > 
> > 
> > Hi
> > 
> > I am having a hard time trying to shape 3000 users at ceil speeds
> > from 10 to 300mbps in a 7/7Gbps link using HTB+SFQ+TC(filter by IP
> > hashkey mask) for a few days now tweaking HTB and SFQ parameters
> > with no luck so far.
> > 
> > Everything seems right, up 4Gbps overall download speed with
> > shaping on. I have no significant packets delay, no dropped packets
> > and no high CPU average loads (not more than 20% - htop info)
> > 
> > But when the speed comes to about 4.5Gbps download (upload is about
> > 500mbps), chaos kicks in. CPU load goes sky high (all 24x2.4Ghz
> > physical cores above 90% - 48x2.4Ghz if count that virtualization
> > is on) and as a consequence packets are dropped (as reported by tc
> > -s class sh ...), RTT goes above 200ms and a lots of ungry users.
> > This goes from about 7PM to 11 PM every day.
> > 
> > If I turn shaping off, everything return to normality immediately
> > and peaks of not more than 5Gbps (1 second average) are observed
> > and a CPU load of about 5%. So I infer the uplink is not crowded.
> > 
> >
> > I use one root HTB qdisc and one root (1:) HTB class.
> >
> > Then about 20~30 same level (1:xx) inner classes to (sort of) separate the users by region 
> > And under these inner classes, goes the almost 3000 leaves (1:xxxx). 
> >
> > I have one class with about 900 users and this quantity decreases
> > by the other inner classes having some of them with just one user.
> > 
> > Is the way I'm using HTB+SFQ+TC suitable for this job?
> > 
> > Since the script that creates the shaping environment is too long I do not post it here.
> > 
> > What can I inform you guys to help me solve this?
> > Fragments of code, stats, some measurements? What?
> > 
> > Thanks.
> > 
> > Regards
> > 
> > Ethy  
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer