From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B99D53B29E for ; Mon, 16 Nov 2020 06:56:51 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605527811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NMZ9hVxp+8TNx8lzzlF5kVRGGByB54U+dTWCJ5mqY0M=; b=WLullWR0OIyDUO7PGswgEQ3SZMzZS+cvX8dKczBF5FktJDYoMQrqyKN00sb828eq4EPkXq 0tb0Qk3EfKwKU45JwdMhiqqRdp27mi7rhX6f32YFqPRsu284kv/44uYICQ06HaccWBFIf4 c1aoHMF3ihYkiYZOa1zMDn+T+JOYOiE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-187-6ZjF9NYGMu2azEb5qS1m2Q-1; Mon, 16 Nov 2020 06:56:47 -0500 X-MC-Unique: 6ZjF9NYGMu2azEb5qS1m2Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DC95C8030CD; Mon, 16 Nov 2020 11:56:45 +0000 (UTC) Received: from carbon (unknown [10.36.110.8]) by smtp.corp.redhat.com (Postfix) with ESMTP id A4E8E5D9D2; Mon, 16 Nov 2020 11:56:41 +0000 (UTC) Date: Mon, 16 Nov 2020 12:56:40 +0100 From: Jesper Dangaard Brouer To: "Thomas Rosenstein" Cc: Bufferbloat , "Saeed Mahameed" , "Tariq Toukan" , brouer@redhat.com Message-ID: <20201116125640.0840429b@carbon> In-Reply-To: <8B5A1292-72E2-4715-838C-5823AFA6B7BD@creamfinance.com> References: <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> <20201106211940.4c30ccc9@carbon> <6963be0e-3eb5-5875-b53c-66033f50dc2d@gmail.com> <12D28386-7C00-4A31-91E4-37083C1674F9@creamfinance.com> <20201109092428.293104ea@carbon> <7723D882-4DAB-4A70-9D00-DF1976872AC2@creamfinance.com> <20201109124030.71216677@carbon> <27110D8E-77DF-4D10-A5EA-6430DBD55BC7@creamfinance.com> <20201112110508.59f402e7@carbon> <20201112143111.2bb697bf@carbon> <20201112164236.3f94702a@carbon> <8B5A1292-72E2-4715-838C-5823AFA6B7BD@creamfinance.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=brouer@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2020 11:56:51 -0000 On Fri, 13 Nov 2020 07:31:26 +0100 "Thomas Rosenstein" wrote: > On 12 Nov 2020, at 16:42, Jesper Dangaard Brouer wrote: > > > On Thu, 12 Nov 2020 14:42:59 +0100 > > "Thomas Rosenstein" wrote: > > > >>> Notice "Adaptive" setting is on. My long-shot theory(2) is that > >>> this > >>> adaptive algorithm in the driver code can guess wrong (due to not > >>> taking TSO into account) and cause issues for > >>> > >>> Try to turn this adaptive algorithm off: > >>> > >>> ethtool -C eth4 adaptive-rx off adaptive-tx off > >>> > > [...] > >>>> > >>>> rx-usecs: 32 > >>> > >>> When you run off "adaptive-rx" you will get 31250 interrupts/sec > >>> (calc 1/(32/10^6) = 31250). > >>> > >>>> rx-frames: 64 > > [...] > >>>> tx-usecs-irq: 0 > >>>> tx-frames-irq: 0 > >>>> > >>> [...] > >> > >> I have now updated the settings to: > >> > >> ethtool -c eth4 > >> Coalesce parameters for eth4: > >> Adaptive RX: off TX: off > >> stats-block-usecs: 0 > >> sample-interval: 0 > >> pkt-rate-low: 0 > >> pkt-rate-high: 0 > >> > >> rx-usecs: 0 > > > > Please put a value in rx-usecs, like 20 or 10. > > The value 0 is often used to signal driver to do adaptive. > > Ok, put it now to 10. Setting it to 10 is a little aggressive, as you ask it to generate 100,000 interrupts per sec. (Watch with 'vmstat 1' to see it.) 1/(10/10^6) = 100000 interrupts/sec > Goes a bit quicker (transfer up to 26 MB/s), but discards and pci stalls > are still there. Why are you measuring in (26) MBytes/sec ? (equal 208 Mbit/s) If you still have ethtool PHY-discards, then you still have a problem. > Ping times are noticable improved: Okay so this means these changes did have a positive effect. So, this can be related to OS is not getting activated fast-enough by NIC interrupts. > 64 bytes from x.x.x.x: icmp_seq=39 ttl=64 time=0.172 ms > 64 bytes from x.x.x.x: icmp_seq=40 ttl=64 time=0.414 ms > 64 bytes from x.x.x.x: icmp_seq=41 ttl=64 time=0.183 ms > 64 bytes from x.x.x.x: icmp_seq=42 ttl=64 time=1.41 ms > 64 bytes from x.x.x.x: icmp_seq=43 ttl=64 time=0.172 ms > 64 bytes from x.x.x.x: icmp_seq=44 ttl=64 time=0.228 ms > 64 bytes from x.x.x.x: icmp_seq=46 ttl=64 time=0.120 ms > 64 bytes from x.x.x.x: icmp_seq=47 ttl=64 time=1.47 ms > 64 bytes from x.x.x.x: icmp_seq=48 ttl=64 time=0.162 ms > 64 bytes from x.x.x.x: icmp_seq=49 ttl=64 time=0.160 ms > 64 bytes from x.x.x.x: icmp_seq=50 ttl=64 time=0.158 ms > 64 bytes from x.x.x.x: icmp_seq=51 ttl=64 time=0.113 ms Can you try to test if disabling TSO, GRO and GSO makes a difference? ethtool -K eth4 gso off gro off tso off -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer