From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D0B073B29D for ; Fri, 6 Nov 2020 06:18:54 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604661534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8hv0zON9pNgM6gcd305joo35j4L0UB/Yfuhxf3RWPY=; b=Yn1D7XapRwr4hh6XtCZxsnls98VNal1mY6q32Ei5BUbj8HEX1N6b+o2T10QkZZVN433IJw +oV0vH9rGJXVy95z/FF/KLGf/dCTDpUhN2b4mLRLyUsSAL2372JpfAaOoctE9UWsDrPWgO ORu0uFY3x7+P6Q7WiN/2nKt65iPtldE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-130-kJ3FW1UINNS3Xmkdhmsslg-1; Fri, 06 Nov 2020 06:18:47 -0500 X-MC-Unique: kJ3FW1UINNS3Xmkdhmsslg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3470C186DD21; Fri, 6 Nov 2020 11:18:46 +0000 (UTC) Received: from carbon (unknown [10.36.110.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 125705D9CA; Fri, 6 Nov 2020 11:18:41 +0000 (UTC) Date: Fri, 6 Nov 2020 12:18:40 +0100 From: Jesper Dangaard Brouer To: "Thomas Rosenstein" Cc: Bloat , "Toke =?UTF-8?B?SMO4aWxhbmQtSsO4?= =?UTF-8?B?cmdlbnNlbg==?=" , brouer@redhat.com Message-ID: <20201106121840.7959ae4b@carbon> In-Reply-To: <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 11:18:54 -0000 On Fri, 06 Nov 2020 10:18:10 +0100 "Thomas Rosenstein" wrote: > >> I just tested 5.9.4 seems to also fix it partly, I have long > >> stretches where it looks good, and then some increases again. (3.10 > >> Stock has them too, but not so high, rather 1-3 ms) > >> That you have long stretches where latency looks good is interesting information. My theory is that your system have a periodic userspace process that does a kernel syscall that takes too long, blocking network card from processing packets. (Note it can also be a kernel thread). Another theory is the NIC HW does strange things, but it is not very likely. E.g. delaying the packets before generating the IRQ interrupt, which hide it from my IRQ-to-softirq latency tool. A question: What traffic control qdisc are you using on your system? What you looked at the obvious case if any of your qdisc report a large backlog? (during the incidents) > >> for example: > >> > >> 64 bytes from x.x.x.x: icmp_seq=10 ttl=64 time=0.169 ms > >> 64 bytes from x.x.x.x: icmp_seq=11 ttl=64 time=5.53 ms > >> 64 bytes from x.x.x.x: icmp_seq=12 ttl=64 time=9.44 ms > >> 64 bytes from x.x.x.x: icmp_seq=13 ttl=64 time=0.167 ms > >> 64 bytes from x.x.x.x: icmp_seq=14 ttl=64 time=3.88 ms > >> > >> and then again: > >> > >> 64 bytes from x.x.x.x: icmp_seq=15 ttl=64 time=0.569 ms > >> 64 bytes from x.x.x.x: icmp_seq=16 ttl=64 time=0.148 ms > >> 64 bytes from x.x.x.x: icmp_seq=17 ttl=64 time=0.286 ms > >> 64 bytes from x.x.x.x: icmp_seq=18 ttl=64 time=0.257 ms > >> 64 bytes from x.x.x.x: icmp_seq=19 ttl=64 time=0.220 ms These very low ping times tell me that you are measuring very close to the target machine, which is good. Here on the bufferbloat list, we are always suspicious of network equipment being use in these kind of setups. As experience tells us that this can be the cause of bufferbloat latency. You mention some fs.com switches (your desc below signature), can you tell us more? [...] > I have a feeling that maybe not all config options were correctly moved > to the newer kernel. > > Or there's a big bug somewhere ... (which would seem rather weird for me > to be the first one to discover this) I really appreciate that you report this. This is a periodic issue, that often result in people not reporting this. Even if we find this to be caused by some process running on your system, or a bad config, this it is really important that we find the root-cause. > I'll rebuild the 5.9 kernel on one of the 3.10 kernel and see if it > makes a difference ... -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer On Wed, 04 Nov 2020 16:23:12 +0100 Thomas Rosenstein via Bloat wrote: > General Info: > > Routers are connected between each other with 10G Mellanox Connect-X > cards via 10G SPF+ DAC cables via a 10G Switch from fs.com > Latency generally is around 0.18 ms between all routers (4). > Throughput is 9.4 Gbit/s with 0 retransmissions when tested with iperf3. > 2 of the 4 routers are connected upstream with a 1G connection (separate > port, same network card) > All routers have the full internet routing tables, i.e. 80k entries for > IPv6 and 830k entries for IPv4 > Conntrack is disabled (-j NOTRACK) > Kernel 5.4.60 (custom) > 2x Xeon X5670 @ 2.93 Ghz > 96 GB RAM