From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 015F83CB35 for ; Fri, 6 Nov 2020 12:04:52 -0500 (EST) Received: by mail-wr1-x435.google.com with SMTP id p8so1256652wrx.5 for ; Fri, 06 Nov 2020 09:04:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=VehQuScRRB1e8Sp46EQfRrtgLwUaSY4ZbJOvs+kDrAg=; b=NtrALkd1FRg6B4RTVWlIPlTuT5vdtmh2EQU6l4SaveazFTmGogiuWVzT1H+1peShp4 JVi/OBgRQzkbZB3/PkxwOEbW+MY1fTIR3QseEuUYS/LqV7cXyBkPvp8/CLmOH0WkBoK6 +1RDfVMDvgHuO2XpTZ1dMBbazAutCuSqxnT3M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=VehQuScRRB1e8Sp46EQfRrtgLwUaSY4ZbJOvs+kDrAg=; b=XuWP2KTqzwuI9wxBSYS5dSrzfn5TgZs0Duyt46tPR/U6aiXHuMcfLtk5yU3V/ApzZ6 nAavFRAhG6erZnHEa2oXIDUv/Uf4jt3CimmikT0vb7UuNT21ekwsNTj3P0OlwEAjbML0 Coq/U1IrrXJheQxdDzAmobh8d7cESYTM4MItaP2sIB9hPKtjyvpmP9SyhzWL7x85dm9e dMGhHJ+f9tP962pF0uXWCoWtXQ3hfrwGZhh5gshNGqm72Mk51NgcOTxdXmuqlvqtgBzw +mcSL8u7RqvXXUAE0Ui/Bg+ajr3L8XyWjCmo34nkEH8BcS4uX8d81QDVjR60MBtCtXPA zvXg== X-Gm-Message-State: AOAM533tK6JtqZgqxKyFuC/vk3oOZoOo2cgL0o873KXERJdyEtEQTqLs edIiRpXQQiupORXAgwYOSAL6 X-Google-Smtp-Source: ABdhPJw5mUtoNTUgJDh3M7Fvv/lL5LT9mueKmvYn7fK4UyAnByRLRSj/v7d8vVCO+93vQIRwX5atRA== X-Received: by 2002:adf:ea11:: with SMTP id q17mr3939143wrm.251.1604682291734; Fri, 06 Nov 2020 09:04:51 -0800 (PST) Received: from [10.8.100.3] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id 90sm3131983wrl.30.2020.11.06.09.04.50 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Nov 2020 09:04:50 -0800 (PST) From: "Thomas Rosenstein" To: "Jesper Dangaard Brouer" Cc: Bloat Date: Fri, 06 Nov 2020 18:04:49 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> In-Reply-To: <20201106151324.5f506574@carbon> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; markup=markdown Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 17:04:53 -0000 On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote: > On Fri, 6 Nov 2020 13:53:58 +0100 > Jesper Dangaard Brouer wrote: > >> [...] >>>> >>>> Could this be related to netlink? I have gobgpd running on these >>>> routers, which injects routes via netlink. >>>> But the churn rate during the tests is very minimal, maybe 30 - 40 >>>> routes every second. >> >> Yes, this could be related. The internal data-structure for FIB >> lookups is a fibtrie which is a compressed patricia tree, related to >> radix tree idea. Thus, I can imagine that the kernel have to >> rebuild/rebalance the tree with all these updates. > > Reading the kernel code. The IPv4 fib_trie code is very well tuned, > fully RCU-ified, meaning read-side is lock-free. The resize() > function > code in net//ipv4/fib_trie.c have max_work limiter to avoid it uses > too > much time. And the update looks lockfree. > > The IPv6 update looks more scary, as it seems to take a "bh" spinlock > that can block softirq from running code in net/ipv6/ip6_fib.c > (spin_lock_bh(&f6i->fib6_table->tb6_lock). I'm using ping on IPv4, but I'll try to see if IPv6 makes any difference! > > Have you tried to use 'perf record' to observe that is happening on > the system while these latency incidents happen? (let me know if you > want some cmdline hints) Haven't tried this yet. If you have some hints what events to monitor I'll take them! > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer