From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 568093B29E for ; Sat, 7 Nov 2020 11:46:36 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604767596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L36Jq/1LkhVcDXm39te9wH5BKM2+IW1DwTceEVnHKoU=; b=U7e7L4hm72U4JbM229Sia6zXe1ex2ZBCSoLi2/PVLtV+rwu+6FoYXXJ/IyV43OOLqEz3G9 hHWhZkhyW3QtANZQmSoI/X4pcUEbyV94HyaiP8rBb62Bw56zpFRcGUQwe4WgR+CbVwwsVW YUydWy3RcjOulxiC3FSGg7rgVJ/jJ+I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-388-rZb_3AJONjG-qkCLB3ynjw-1; Sat, 07 Nov 2020 11:46:30 -0500 X-MC-Unique: rZb_3AJONjG-qkCLB3ynjw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3EFEA57205; Sat, 7 Nov 2020 16:46:29 +0000 (UTC) Received: from carbon (unknown [10.36.110.8]) by smtp.corp.redhat.com (Postfix) with ESMTP id 399AA5B4B8; Sat, 7 Nov 2020 16:46:24 +0000 (UTC) Date: Sat, 7 Nov 2020 17:46:23 +0100 From: Jesper Dangaard Brouer To: "Thomas Rosenstein" Cc: Bloat , brouer@redhat.com Message-ID: <20201107174623.73f91841@carbon> In-Reply-To: References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> <20201106211940.4c30ccc9@carbon> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Nov 2020 16:46:36 -0000 On Sat, 07 Nov 2020 13:37:01 +0100 "Thomas Rosenstein" wrote: > On 6 Nov 2020, at 21:19, Jesper Dangaard Brouer wrote: > > > On Fri, 06 Nov 2020 18:04:49 +0100 > > "Thomas Rosenstein" wrote: > > > >> On 6 Nov 2020, at 15:13, Jesper Dangaard Brouer wrote: > >> > >> [...] > > > >>> Have you tried to use 'perf record' to observe that is happening on > >>> the system while these latency incidents happen? (let me know if > >>> you want some cmdline hints) > >> > >> Haven't tried this yet. If you have some hints what events to monitor > >> I'll take them! > > > > Okay to record everything (-a) on the system and save call-graph (-g), > > and run for 5 seconds (via profiling the sleep function). > > > > # perf record -g -a sleep 5 > > > > To view the result the simply use the 'perf report', but likely you > > want to use option --no-children as you are profiling the kernel (and > > not a userspace program you want to have grouped 'children' by). I > > also include the CPU column via '--sort cpu,comm,dso,symbol' and you > > can select/zoom-in-on a specific CPU via '-C zero-indexed-cpu-num'. > > > > # perf report --sort cpu,comm,dso,symbol --no-children > > > > When we ask you to provide the output, you can use the --stdio option, > > and provide txt-info via a pastebin link as it is very long. > > Here is the output from kernel 3.10_1127 (I updated to the really newest > in that branch): https://pastebin.com/5mxirXPw > Here is the output from kernel 5.9.4: https://pastebin.com/KDZ2Ei2F Did the latency issue happen during this this perf recording? (it is obviously important that you record during the issue) Finding a single latency event/issue in perf.data is like looking in for needle in a haystack. It does give hints and e.g. tell me details like what part of the kernel are used/activated. I can see that you have likely configured XPS (Transmit Packet Steering). Is this on purpose? -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer