From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3BC623B2A4 for ; Thu, 12 Nov 2020 08:43:03 -0500 (EST) Received: by mail-wm1-x32b.google.com with SMTP id 10so5317440wml.2 for ; Thu, 12 Nov 2020 05:43:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=EvZk971Y6HvY4VraGQBVn1ZSXac55plV7AEZMLBSkSw=; b=hoUasoW+FZ3sV/7BsJ+SYI7utHozmCwjNbUbwy1fgjNOuwljKXGWT0qHK453YibPno AKaMMczf5/ElNpYezkOf0Jvt3hBRQWCzxTAsbhSXqC0xhe+7we8cZr6tOIqjaVBRL84E Upqd2ofPIkPLzBIolR96AIzWtQq5ntZbavR+M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=EvZk971Y6HvY4VraGQBVn1ZSXac55plV7AEZMLBSkSw=; b=uWky2Zzn5UyZHjNZq0p6OSG6rh5VFG3krivdO5c+ofwBecQpNA05VZQxGty8bBQDZm /fP2q6ZmzSkCIJdyk05kzBf5192DOJsxyxv4P0f6lkOBg3D2oZqcNE9uUu2IqLkB25Xd M+Jz7hDXy1go6BKMW0ZcFgPkpCTqlOpIoL3soVO6mDyO3RTyBsxIhhRd9LNrtaqUwiKq Ep6A9zvSsJ9KtB6bu0/1a3ViK1/JzEp3gr3LR/t0tbcbVVM9yDvAZ9y/ABMBsWm26Bav +yIVm9CoqQnKHjIGRqFTXhnlbSjS9ipW3bM80vRyRf/Mi1obK/JEKU7NwOLy+bS1ChHo uSxQ== X-Gm-Message-State: AOAM530ItcpDU+Dh8s3zKzWEgsuLRxrrSSX3sEnJxSD5Qc7I9tj3zN0o TuzuolR3MfK9jBoN+CdR3knH X-Google-Smtp-Source: ABdhPJwL8fguNNtoqG9pBsgse2+UnqT9wOetRbR1wnCuKeYe1aGmvg/uA9PKz2lUQYeSjVajQykVCg== X-Received: by 2002:a7b:c20d:: with SMTP id x13mr9315869wmi.167.1605188582102; Thu, 12 Nov 2020 05:43:02 -0800 (PST) Received: from [10.37.129.2] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id t7sm6696737wrx.42.2020.11.12.05.43.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Nov 2020 05:43:01 -0800 (PST) From: "Thomas Rosenstein" To: "Jesper Dangaard Brouer" Cc: Bufferbloat , "Saeed Mahameed" , "Tariq Toukan" Date: Thu, 12 Nov 2020 14:42:59 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <20201112143111.2bb697bf@carbon> References: <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> <20201106135358.09f6c281@carbon> <20201106151324.5f506574@carbon> <1E70B6D2-1212-43FA-989A-03B657EEE2F2@creamfinance.com> <20201106211940.4c30ccc9@carbon> <6963be0e-3eb5-5875-b53c-66033f50dc2d@gmail.com> <12D28386-7C00-4A31-91E4-37083C1674F9@creamfinance.com> <20201109092428.293104ea@carbon> <7723D882-4DAB-4A70-9D00-DF1976872AC2@creamfinance.com> <20201109124030.71216677@carbon> <27110D8E-77DF-4D10-A5EA-6430DBD55BC7@creamfinance.com> <20201112110508.59f402e7@carbon> <20201112143111.2bb697bf@carbon> MIME-Version: 1.0 Content-Type: text/plain; markup=markdown Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Nov 2020 13:43:03 -0000 On 12 Nov 2020, at 14:31, Jesper Dangaard Brouer wrote: > On Thu, 12 Nov 2020 12:26:20 +0100 > "Thomas Rosenstein" wrote: > >>> Long-shot theory(2): The NIC driver IRQ interrupt coalesce >>> adaptive-IRQ (adj via ethtool -C) is wrong. The reason it is wrong is >>> because of TSO frames, due packet-to-byte ratio isn't taken into >>> account. >> >> here are the coalese parameters: (are the same for 3.10 and 5.4.60) >> >> ethtool -c eth4 >> Coalesce parameters for eth4: >> Adaptive RX: on TX: on > > Notice "Adaptive" setting is on. My long-shot theory(2) is that this > adaptive algorithm in the driver code can guess wrong (due to not > taking TSO into account) and cause issues for > > Try to turn this adaptive algorithm off: > > ethtool -C eth4 adaptive-rx off adaptive-tx off > >> stats-block-usecs: 0 >> sample-interval: 0 >> pkt-rate-low: 0 >> pkt-rate-high: 0 >> >> rx-usecs: 32 > > When you run off "adaptive-rx" you will get 31250 interrupts/sec > (calc 1/(32/10^6) = 31250). > >> rx-frames: 64 > > You likely want to reduce this. > >> rx-usecs-irq: 0 >> rx-frames-irq: 0 >> >> tx-usecs: 8 >> tx-frames: 128 > > These 128 frames also seems too large. > >> tx-usecs-irq: 0 >> tx-frames-irq: 0 >> > [...] I have now updated the settings to: ethtool -c eth4 Coalesce parameters for eth4: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 0 rx-frames: 32 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 16 tx-frames: 32 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 but it makes absolutely no difference, also discard keep coming as before. >>> >>> You can test theory(2) in two ways, via disabling TSO and via >>> adjusting >>> ethtool --show-coalesce|--coalesce (-c|-C). The --coalesce change can >>> change part of theory(1), the NIC wake-up/IRQ interval. here's the current nic settings: ethtool -k eth4 Features for eth4: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: off udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: on tx-gre-csum-segmentation: on tx-ipxip4-segmentation: on tx-ipxip6-segmentation: on tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] tx-udp-segmentation: on fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off rx-all: off tx-vlan-stag-hw-insert: on rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: on [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: on tls-hw-tx-offload: off [fixed] tls-hw-rx-offload: off [fixed] rx-gro-hw: off [fixed] tls-hw-record: off [fixed] > > What is the NIC ring size settings? > > ethtool -g eth4 > ethtool -g eth4 Ring parameters for eth4: Pre-set maximums: RX: 8192 RX Mini: 0 RX Jumbo: 0 TX: 8192 Current hardware settings: RX: 1024 RX Mini: 0 RX Jumbo: 0 TX: 1024 > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer