From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4F7683CB37 for ; Fri, 6 Nov 2020 15:19:49 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604693989; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fo72op5nnEOrTIR5JOcT1PTwFolZLamfL5r/tiA4Zds=; b=RQBtOpNiHFJsFkfk1q6SH/hmC5QtVXspEGwV+eIvB/2b8VpugfDJUXRd67bQ0var9I8OUm ff0lI6PwlRT/Qm0SYG20Pv2AwbDD02+6/TYEvYzwOcjerx4ThYZFR0zBwxS975LW3j1REF j5YZ22WiZQRHcY0LQ/cT/fVCCjzD0vc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-576-IDDvVik9O72SQuPECpJWRw-1; Fri, 06 Nov 2020 15:19:47 -0500 X-MC-Unique: IDDvVik9O72SQuPECpJWRw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 47ED5188C129; Fri, 6 Nov 2020 20:19:46 +0000 (UTC) Received: from carbon (unknown [10.36.110.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 00BAC5C1BD; Fri, 6 Nov 2020 20:19:41 +0000 (UTC) Date: Fri, 6 Nov 2020 21:19:40 +0100 From: Jesper Dangaard Brouer To: "Thomas Rosenstein" Cc: Bloat , brouer@redhat.com Message-ID: <20201106211940.4c30ccc9@carbon> In-Reply-To: <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 20:19:49 -0000 On Fri, 06 Nov 2020 18:04:49 +0100 "Thomas Rosenstein" wrote: > On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote: > > > On Fri, 6 Nov 2020 13:53:58 +0100 > > Jesper Dangaard Brouer wrote: > > > >> [...] > >>>> > >>>> Could this be related to netlink? I have gobgpd running on these > >>>> routers, which injects routes via netlink. > >>>> But the churn rate during the tests is very minimal, maybe 30 - 40 > >>>> routes every second. > >> > >> Yes, this could be related. The internal data-structure for FIB > >> lookups is a fibtrie which is a compressed patricia tree, related to > >> radix tree idea. Thus, I can imagine that the kernel have to > >> rebuild/rebalance the tree with all these updates. > > > > Reading the kernel code. The IPv4 fib_trie code is very well tuned, > > fully RCU-ified, meaning read-side is lock-free. The resize() > > function code in net//ipv4/fib_trie.c have max_work limiter to avoid it uses > > too much time. And the update looks lockfree. > > > > The IPv6 update looks more scary, as it seems to take a "bh" spinlock > > that can block softirq from running code in net/ipv6/ip6_fib.c > > (spin_lock_bh(&f6i->fib6_table->tb6_lock). > > I'm using ping on IPv4, but I'll try to see if IPv6 makes any > difference! I think you misunderstand me. I'm not asking you to use ping6. The gobgpd daemon updates will both update IPv4 and IPv6 routes, right. Updating IPv6 routes are more problematic than IPv4 routes. The IPv6 route tables update can potentially stall softirq from running, which was the latency tool was measuring... and it did show some outliers. > > Have you tried to use 'perf record' to observe that is happening on > > the system while these latency incidents happen? (let me know if you > > want some cmdline hints) > > Haven't tried this yet. If you have some hints what events to monitor > I'll take them! Okay to record everything (-a) on the system and save call-graph (-g), and run for 5 seconds (via profiling the sleep function). # perf record -g -a sleep 5 To view the result the simply use the 'perf report', but likely you want to use option --no-children as you are profiling the kernel (and not a userspace program you want to have grouped 'children' by). I also include the CPU column via '--sort cpu,comm,dso,symbol' and you can select/zoom-in-on a specific CPU via '-C zero-indexed-cpu-num'. # perf report --sort cpu,comm,dso,symbol --no-children When we ask you to provide the output, you can use the --stdio option, and provide txt-info via a pastebin link as it is very long. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer