From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EA9B03B29D for ; Thu, 5 Nov 2020 07:22:15 -0500 (EST) Received: by mail-wr1-x431.google.com with SMTP id w14so1512868wrs.9 for ; Thu, 05 Nov 2020 04:22:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C2TP3LCmWF8G2tUpZaj/mKyi6kAr2Ugcb4QkwfafbLQ=; b=Gpk5f3JiHq6+tJ9m4Bqh/ts5wXFybpFyOF6nHxB0Uf9cMUn5KcZ5W/3SJvVAGtBfdt 1bXbFvwmZtZ5inCjfLtAZCg3I4ynijUAxB3nls1OqOPDM+/pKZnniof4KUPw5e7jcOI3 hQAmByO+QChgIIGom6R0/Rf1iZwEV9C63DzzU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C2TP3LCmWF8G2tUpZaj/mKyi6kAr2Ugcb4QkwfafbLQ=; b=HEnwqvSCK2ERO23F/f55jcnprc3+VakXt0yHUMxy0xYsRxHx0IV5G2jwvm8KyN4s7v 4BQMAqEjrYiZ8XpEJPaUOjWCxOA/1+MZryRTOLCuNwxE7cL633nrWKm2gEPMAMNxspXJ LCnjVLhDyRJnjYAaX/yVatJZd1fjtW6JCnqUfhHMq+Y034+hoDuav4orFh1FhombdiF2 9NYM3jjEP92P+TQSgNXs2QkYE8A5ezQmF3flEC2ABKs5rlbVLegKA0cdvOcpVr2/eNT7 HQ3cSGmW/XSSNGmEllWRnXqgDmoDHRDHVKoO1plbQnO8PUn8yjgUFXHhRbeFBc9lxynk mHJw== X-Gm-Message-State: AOAM5333pObrHHBUgVjxqL/sF874w1NS/6ajQmHFon4R95FvMbk6VIYw K9/mQ71j+j/UGjYO44Qknmic X-Google-Smtp-Source: ABdhPJxBlAUARtxDNqkARnylLpXsThFEfWPEe5vWkb5aB+SN/vvmkH9UBRebwaT6EbgfvX6v/inF7w== X-Received: by 2002:adf:e610:: with SMTP id p16mr2815577wrm.302.1604578934837; Thu, 05 Nov 2020 04:22:14 -0800 (PST) Received: from [10.8.100.3] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id z191sm2449388wme.30.2020.11.05.04.22.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Nov 2020 04:22:14 -0800 (PST) From: "Thomas Rosenstein" To: "Toke =?utf-8?b?SMO4aWxhbmQtSsO4cmdlbnNlbg==?=" Cc: bloat@lists.bufferbloat.net Date: Thu, 05 Nov 2020 13:22:10 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <87sg9ot5f1.fsf@toke.dk> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 12:22:16 -0000 On 5 Nov 2020, at 12:21, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > "Thomas Rosenstein" writes: > >>> If so, this sounds more like a driver issue, or maybe something to = >>> do >>> with scheduling. Does it only happen with ICMP? You could try this >>> tool >>> for a userspace UDP measurement: >> >> It happens with all packets, therefore the transfer to backblaze with = >> 40 >> threads goes down to ~8MB/s instead of >60MB/s > > Huh, right, definitely sounds like a kernel bug; or maybe the new = > kernel > is getting the hardware into a state where it bugs out when there are > lots of flows or something. > > You could try looking at the ethtool stats (ethtool -S) while running > the test and see if any error counters go up. Here's a handy script to > monitor changes in the counters: > > https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool= _stats.pl > >> I'll try what that reports! >> >>> Also, what happens if you ping a host on the internet (*through* the >>> router instead of *to* it)? >> >> Same issue, but twice pronounced, as it seems all interfaces are >> affected. >> So, ping on one interface and the second has the issue. >> Also all traffic across the host has the issue, but on both sides, so >> ping to the internet increased by 2x > > Right, so even an unloaded interface suffers? But this is the same = > NIC, > right? So it could still be a hardware issue... > >> Yep default that CentOS ships, I just tested 4.12.5 there the issue = >> also >> does not happen. So I guess I can bisect it then...(really don't want = >> to >> =F0=9F=98=83) > > Well that at least narrows it down :) I just tested 5.9.4 seems to also fix it partly, I have long stretches = where it looks good, and then some increases again. (3.10 Stock has them = too, but not so high, rather 1-3 ms) for example: 64 bytes from x.x.x.x: icmp_seq=3D10 ttl=3D64 time=3D0.169 ms 64 bytes from x.x.x.x: icmp_seq=3D11 ttl=3D64 time=3D5.53 ms 64 bytes from x.x.x.x: icmp_seq=3D12 ttl=3D64 time=3D9.44 ms 64 bytes from x.x.x.x: icmp_seq=3D13 ttl=3D64 time=3D0.167 ms 64 bytes from x.x.x.x: icmp_seq=3D14 ttl=3D64 time=3D3.88 ms and then again: 64 bytes from x.x.x.x: icmp_seq=3D15 ttl=3D64 time=3D0.569 ms 64 bytes from x.x.x.x: icmp_seq=3D16 ttl=3D64 time=3D0.148 ms 64 bytes from x.x.x.x: icmp_seq=3D17 ttl=3D64 time=3D0.286 ms 64 bytes from x.x.x.x: icmp_seq=3D18 ttl=3D64 time=3D0.257 ms 64 bytes from x.x.x.x: icmp_seq=3D19 ttl=3D64 time=3D0.220 ms 64 bytes from x.x.x.x: icmp_seq=3D20 ttl=3D64 time=3D0.125 ms 64 bytes from x.x.x.x: icmp_seq=3D21 ttl=3D64 time=3D0.188 ms 64 bytes from x.x.x.x: icmp_seq=3D22 ttl=3D64 time=3D0.202 ms 64 bytes from x.x.x.x: icmp_seq=3D23 ttl=3D64 time=3D0.195 ms 64 bytes from x.x.x.x: icmp_seq=3D24 ttl=3D64 time=3D0.177 ms 64 bytes from x.x.x.x: icmp_seq=3D25 ttl=3D64 time=3D0.242 ms 64 bytes from x.x.x.x: icmp_seq=3D26 ttl=3D64 time=3D0.339 ms 64 bytes from x.x.x.x: icmp_seq=3D27 ttl=3D64 time=3D0.183 ms 64 bytes from x.x.x.x: icmp_seq=3D28 ttl=3D64 time=3D0.221 ms 64 bytes from x.x.x.x: icmp_seq=3D29 ttl=3D64 time=3D0.317 ms 64 bytes from x.x.x.x: icmp_seq=3D30 ttl=3D64 time=3D0.210 ms 64 bytes from x.x.x.x: icmp_seq=3D31 ttl=3D64 time=3D0.242 ms 64 bytes from x.x.x.x: icmp_seq=3D32 ttl=3D64 time=3D0.127 ms 64 bytes from x.x.x.x: icmp_seq=3D33 ttl=3D64 time=3D0.217 ms 64 bytes from x.x.x.x: icmp_seq=3D34 ttl=3D64 time=3D0.184 ms For me it looks now that there was some fix between 5.4.60 and 5.9.4 ... = anyone can pinpoint it? > >>> >>> How did you configure the new kernel? Did you start from scratch, or >>> is >>> it based on the old centos config? >> >> first oldconfig and from there then added additional options for IB, >> NVMe, etc (which I don't really need on the routers) > > OK, so you're probably building with roughly the same options in terms > of scheduling granularity etc. That's good. Did you enable spectre > mitigations etc on the new kernel? What's the output of > `tail /sys/devices/system/cpu/vulnerabilities/*` ? mitigations are off =3D=3D> /sys/devices/system/cpu/vulnerabilities/itlb_multihit <=3D=3D KVM: Vulnerable =3D=3D> /sys/devices/system/cpu/vulnerabilities/l1tf <=3D=3D Mitigation: PTE Inversion; VMX: vulnerable =3D=3D> /sys/devices/system/cpu/vulnerabilities/mds <=3D=3D Vulnerable; SMT vulnerable =3D=3D> /sys/devices/system/cpu/vulnerabilities/meltdown <=3D=3D Vulnerable =3D=3D> /sys/devices/system/cpu/vulnerabilities/spec_store_bypass <=3D=3D= Vulnerable =3D=3D> /sys/devices/system/cpu/vulnerabilities/spectre_v1 <=3D=3D Vulnerable: __user pointer sanitization and usercopy barriers only; no = swapgs barriers =3D=3D> /sys/devices/system/cpu/vulnerabilities/spectre_v2 <=3D=3D Vulnerable, STIBP: disabled =3D=3D> /sys/devices/system/cpu/vulnerabilities/srbds <=3D=3D Not affected =3D=3D> /sys/devices/system/cpu/vulnerabilities/tsx_async_abort <=3D=3D Not affected Grub Boot options are: crashkernel=3D896M rd.lvm.lv=3Dcl/root net.ifnames= =3D0 = biosdevname=3D0 scsi_mod.use_blk_mq=3D1 dm_mod.use_blk_mq=3Dy mitigations= =3Doff = console=3Dtty0 console=3DttyS1,115200 > > -Toke