From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [IPv6:2001:470:dc45:1000::1]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id F105D3BA8E for ; Sat, 2 Jun 2018 15:25:38 -0400 (EDT) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1527967536; bh=UsxtLWOinFg4bsJNpfqcik5v7ivxj6ztfrLiCRI28fI=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=dQBxbencMKM6+LtiOBSuRFBZFCvenXoP1+3Ih+aU5neTxhc7GmrWcjWQwqgYJLroZ fZ4izXUytjufDYmN8tPCRyR8zQuAlBzxMmp0aLaGYikwCZsifhqUyn033Wn+TvHOpJ kKueo4FgxBrFoEWQK35VhNB5jVC9iWtg+FcKkiZCNx3XzQlum3/tCIPq8GhjG5LHLd iSpUW91pgNOSNDaYRtQJCXTk3FoEmn+o0+CG50gQ9BLpix9P8pL5wGVFnZbCthRhDk ms85aKsXVGFg1RgsbFkBZ9cCkYe+8u656VaaxxVA7M1KDZhV2ryfizNdgidNx1Mz34 ex8RRN/7xOkjg== To: Dave Taht Cc: Jonathan Morton , Cake List In-Reply-To: References: <87muwe9z7j.fsf@toke.dk> <87k1ri9yve.fsf@toke.dk> <877enimhuh.fsf@toke.dk> <9A273CA6-FDF6-485D-B466-8B01D4573BD9@gmail.com> <871sdqmg8l.fsf@toke.dk> <87a7sdcan1.fsf@toke.dk> Date: Sat, 02 Jun 2018 21:25:35 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <877enhc7o0.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] Lockup at high speeds X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jun 2018 19:25:39 -0000 Dave Taht writes: > On Sat, Jun 2, 2018 at 11:21 AM, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >> Dave Taht writes: >> >>> How on earth are you getting these speeds? I'm stuck at a pathetic *4* >>> gbit here on the admittedly ancient 12 core boxes I'd got. >>> (I'll go rework my veth topology) >> >> Well, having actual hardware do the packet forwarding, and only having >> each machine process each packet once probably helps... >> >>> rcu stuff makes me nervous, what happens if that nat stuff is disabled = entirely. >>> >>> What if you made q.len and sparse/bullk flow counts atomic ops? >> >> The whole qdisc dequeue op is running while holding a global lock. The >> freezes I'm seeing are happening because the other cores get stuck >> waiting for the cake dequeue operation to complete (and thus release the >> qdisc lock). Which is why I suspect an infinite loop somewhere... > > Am I wrong in remembering that some qdisc stats are multi-core (like > qlen?), and summed at some other step? Well, I *think* you are... :) > If you pin the card or qdisc to a single core what happens? (Besides > it getting slower) No idea. Will try that, once i have more time to debug this; in a few weeks. I'm trying to get someone else to do something about it in the meantime, as you can maybe tell ;) -Toke