From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 7C32A3B29E for ; Mon, 16 Nov 2020 07:49:35 -0500 (EST) Received: by mail-wm1-x32d.google.com with SMTP id 10so23613588wml.2 for ; Mon, 16 Nov 2020 04:49:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QaSSrtKKH/FrvxJyLj4/lSQaZU7iezrzzygdG0yJRt4=; b=GJSa7tQrHzx8tq+kTyf9B5Xb2yeOWOJcf/2B4o9UNLZ5wpvGoTFpPkJ5ccPJ2Vir1i YjE41VajRDjzHa5fzNp+8eHVgR2AUSR3klMh4P15gyewDWFWom3KdUMavHp9CP746V+F bj9JAawmmLVh7igOoZcbn4bvDAD7mn2IYNOvY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QaSSrtKKH/FrvxJyLj4/lSQaZU7iezrzzygdG0yJRt4=; b=WpwKh6zV3JCTkOQ0dTIqjWOSqvcFLZslnTtyazc5iiytP+fjUDwhEdIKqYG2ca3xH7 QUAaj9cd3vwU5ClPCu83FSZ8sQIx/5OKt/8od3heGhOZN380IJ4m9+ReKymyfW+i+3cg 5t4bDps8uw3OASVcOa0JPbd9lQwJVQqiDbkrGqF/OcnQlh0LpH+OmHjHqVfZv7WPA5WP SrJNwHlzsE3AJP2LPqU7rcV4/fWG7XEsg5AHiHB9QU3QNk8nSXpyXGJdjNtG8HvDGWar oPEN3fkZ1SbXeCwliWLg/wmAOaJdPLWg+7mAMdAD4cF+BZW4gQ+wu0e6tmt8yQclJig9 jqYQ== X-Gm-Message-State: AOAM533PEGiPsfIkLKWJxZLzhJxRElZHhIR4lAO+zNm5wz7XaUcDb/u7 w4L+YJB27x0CEVIKtWdVIqdL X-Google-Smtp-Source: ABdhPJyjTRmns0XuTP2/2p9lPwwnucP2TNMLxcECWLYqfcbVizE/kcifVmpFbDBL+z3MHNgDUcahxg== X-Received: by 2002:a1c:4d05:: with SMTP id o5mr15254732wmh.124.1605530973838; Mon, 16 Nov 2020 04:49:33 -0800 (PST) Received: from [10.37.129.2] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id c62sm19523519wme.22.2020.11.16.04.49.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Nov 2020 04:49:33 -0800 (PST) From: "Thomas Rosenstein" To: "Jesper Dangaard Brouer" Cc: Bufferbloat Date: Mon, 16 Nov 2020 13:49:32 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <20201116133438.19459da5@carbon> References: <20201116133438.19459da5@carbon> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; markup=markdown Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2020 12:49:35 -0000 On 16 Nov 2020, at 13:34, Jesper Dangaard Brouer wrote: > On Wed, 04 Nov 2020 16:23:12 +0100 > Thomas Rosenstein via Bloat wrote: > > [...] >> I have multiple routers which connect to multiple upstream providers, = >> I >> have noticed a high latency shift in icmp (and generally all = >> connection) >> if I run b2 upload-file --threads 40 (and I can reproduce this) >> >> What options do I have to analyze why this happens? >> >> General Info: >> >> Routers are connected between each other with 10G Mellanox Connect-X >> cards via 10G SPF+ DAC cables via a 10G Switch from fs.com >> Latency generally is around 0.18 ms between all routers (4). >> Throughput is 9.4 Gbit/s with 0 retransmissions when tested with = >> iperf3. >> 2 of the 4 routers are connected upstream with a 1G connection = >> (separate >> port, same network card) >> All routers have the full internet routing tables, i.e. 80k entries = >> for >> IPv6 and 830k entries for IPv4 >> Conntrack is disabled (-j NOTRACK) >> Kernel 5.4.60 (custom) >> 2x Xeon X5670 @ 2.93 Ghz > > I think I have spotted your problem... This CPU[1] Xeon X5670 is more > than 10 years old! It basically corresponds to the machines I used = > for > my presentation at LinuxCon 2009 see slides[2]. Only with large = > frames > and with massive scaling across all CPUs was I able to get close to > 10Gbit/s through these machines. And on top I had to buy low-latency > RAM memory-blocks to make it happen. > > As you can see on my slides[2], memory bandwidth and PCIe speeds was = > at > the limit for making it possible on the hardware level. I had to run > DDR3 memory at 1333MHz and tune the QuickPath Interconnect (QPI) to > 6.4GT/s (default 4.8GT/s). > > This generation Motherboards had both PCIe gen-1 and gen-2 slots. = > Only > the PCIe gen-2 slots had barely enough bandwidth. Maybe you = > physically > placed NIC in PCIe gen-1 slot? > > On top of this, you also have a NUMA system, 2x Xeon X5670, which can > result is A LOT of "funny" issue, that is really hard to = > troubleshoot... > Yes, I'm aware of the limits of what to expect, but as we agree 60 tcp = streams with not even 200 Mbits shouldn't overload the PCIex bus or the = cpus. Also, don't forget, no issues with Kernel 3.10. PCI slot is a Gen2, x8, so more than enough bandwidth there luckily ;) But yes, they are quite old... > > [1] = > https://ark.intel.com/content/www/us/en/ark/products/47920/intel-xeon-p= rocessor-x5670-12m-cache-2-93-ghz-6-40-gt-s-intel-qpi.html > > [2] = > https://people.netfilter.org/hawk/presentations/LinuxCon2009/LinuxCon20= 09_JesperDangaardBrouer_final.pdf > > -- = > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer