From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 9ADFD3B2A4 for ; Fri, 6 Nov 2020 09:13:35 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604672015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jJ4u8gnVO7UYFYQLzf+qU2fNxNvIY13EI+aeh2MRniE=; b=iFzNNyIaGrrd5TkGQa1DLnYOquZtye83DAgG+SIc21CkjxwTEijwbWZHxi5E6cQldzz3oM +G/q5SHVLFS2XHNJZe3ZBonqbhCXDko1a2TYj9SSpOeOaZCnl7a+mjSpm7UxBWxDyF9Q+e NSHVu7bv1rhjl8UxO5SNpchErJqVUII= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-295-gpW-SRPNPgiHWRnXoaU9Lg-1; Fri, 06 Nov 2020 09:13:32 -0500 X-MC-Unique: gpW-SRPNPgiHWRnXoaU9Lg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3DEA3188C122; Fri, 6 Nov 2020 14:13:31 +0000 (UTC) Received: from carbon (unknown [10.36.110.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA42A7512B; Fri, 6 Nov 2020 14:13:25 +0000 (UTC) Date: Fri, 6 Nov 2020 15:13:24 +0100 From: Jesper Dangaard Brouer To: Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNlbg==?= Cc: Thomas Rosenstein , Bloat , brouer@redhat.com, David Ahern Message-ID: <20201106151324.5f506574@carbon> In-Reply-To: <20201106135358.09f6c281@carbon> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 14:13:35 -0000 On Fri, 6 Nov 2020 13:53:58 +0100 Jesper Dangaard Brouer wrote: > [...] > > > > > > Could this be related to netlink? I have gobgpd running on these > > > routers, which injects routes via netlink. > > > But the churn rate during the tests is very minimal, maybe 30 - 40 > > > routes every second. > > Yes, this could be related. The internal data-structure for FIB > lookups is a fibtrie which is a compressed patricia tree, related to > radix tree idea. Thus, I can imagine that the kernel have to > rebuild/rebalance the tree with all these updates. Reading the kernel code. The IPv4 fib_trie code is very well tuned, fully RCU-ified, meaning read-side is lock-free. The resize() function code in net//ipv4/fib_trie.c have max_work limiter to avoid it uses too much time. And the update looks lockfree. The IPv6 update looks more scary, as it seems to take a "bh" spinlock that can block softirq from running code in net/ipv6/ip6_fib.c (spin_lock_bh(&f6i->fib6_table->tb6_lock). Have you tried to use 'perf record' to observe that is happening on the system while these latency incidents happen? (let me know if you want some cmdline hints) -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer