From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2238A3B2A4 for ; Thu, 12 Nov 2020 08:31:20 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605187880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TDFq3A8VPB0c7ARSZddhIeDdu+S+OZM8Oc2u8L3r5qo=; b=NiiJGZM7jHVV0H/vgYcK2pOxeBMDD6ZBbz3FedCHcF72uxFrIPzE77Ssu4oAHzn26U9nU2 XelTmAAGbL6QkguPoJwhUMezfgPrvPhkCjskNe3JqT9b3NBJFKkE5TH74qhQp84VjdGhz9 UvwphKdBBmRD/ZYrOa63S8jP54BD6Ro= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-589-lYWHOXdCP1CbE8M_d6PI-w-1; Thu, 12 Nov 2020 08:31:18 -0500 X-MC-Unique: lYWHOXdCP1CbE8M_d6PI-w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8C356AF063; Thu, 12 Nov 2020 13:31:17 +0000 (UTC) Received: from carbon (unknown [10.36.110.8]) by smtp.corp.redhat.com (Postfix) with ESMTP id 324AC6EF6E; Thu, 12 Nov 2020 13:31:12 +0000 (UTC) Date: Thu, 12 Nov 2020 14:31:11 +0100 From: Jesper Dangaard Brouer To: "Thomas Rosenstein" Cc: Bufferbloat , "Saeed Mahameed" , "Tariq Toukan" , brouer@redhat.com Message-ID: <20201112143111.2bb697bf@carbon> In-Reply-To: References: <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> <20201106211940.4c30ccc9@carbon> <6963be0e-3eb5-5875-b53c-66033f50dc2d@gmail.com> <12D28386-7C00-4A31-91E4-37083C1674F9@creamfinance.com> <20201109092428.293104ea@carbon> <7723D882-4DAB-4A70-9D00-DF1976872AC2@creamfinance.com> <20201109124030.71216677@carbon> <27110D8E-77DF-4D10-A5EA-6430DBD55BC7@creamfinance.com> <20201112110508.59f402e7@carbon> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Nov 2020 13:31:21 -0000 On Thu, 12 Nov 2020 12:26:20 +0100 "Thomas Rosenstein" wrote: > > Long-shot theory(2): The NIC driver IRQ interrupt coalesce > > adaptive-IRQ (adj via ethtool -C) is wrong. The reason it is wrong is > > because of TSO frames, due packet-to-byte ratio isn't taken into > > account. > > here are the coalese parameters: (are the same for 3.10 and 5.4.60) > > ethtool -c eth4 > Coalesce parameters for eth4: > Adaptive RX: on TX: on Notice "Adaptive" setting is on. My long-shot theory(2) is that this adaptive algorithm in the driver code can guess wrong (due to not taking TSO into account) and cause issues for Try to turn this adaptive algorithm off: ethtool -C eth4 adaptive-rx off adaptive-tx off > stats-block-usecs: 0 > sample-interval: 0 > pkt-rate-low: 0 > pkt-rate-high: 0 > > rx-usecs: 32 When you run off "adaptive-rx" you will get 31250 interrupts/sec (calc 1/(32/10^6) = 31250). > rx-frames: 64 You likely want to reduce this. > rx-usecs-irq: 0 > rx-frames-irq: 0 > > tx-usecs: 8 > tx-frames: 128 These 128 frames also seems too large. > tx-usecs-irq: 0 > tx-frames-irq: 0 > [...] > > > > You can test theory(2) in two ways, via disabling TSO and via > > adjusting > > ethtool --show-coalesce|--coalesce (-c|-C). The --coalesce change can > > change part of theory(1), the NIC wake-up/IRQ interval. What is the NIC ring size settings? ethtool -g eth4 -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer