From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5E45C3B29D for ; Thu, 5 Nov 2020 07:41:32 -0500 (EST) Received: by mail-wm1-x32f.google.com with SMTP id s13so1471140wmh.4 for ; Thu, 05 Nov 2020 04:41:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=u2WcVcFk3uc+lcjQ7DaDYLvpld3bDjpLsItTOvNuhwk=; b=RE+PsJJS/iCm3iLkKSDyugahmC7hOMPBvMHqPJC4xX8ud1qTI66/0Lg7GUJetLznrY zuBN0li6Ai7ijOg0xbjVoOevqfnwYnX5+brL1y7bPyPA/8ukN0jWTQss6WfvG9wS96xg CezdMl68/3bVZp/L/qa6X7GKITxPMXd9LmF3I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u2WcVcFk3uc+lcjQ7DaDYLvpld3bDjpLsItTOvNuhwk=; b=LFquHEdetTrTfMs3XlSrdbjn5enCI+PuwKLVUe/j8w5tU2ThYhdK99T8bVrQWUfMOZ W+mCi2Vi9GjK1zShp0O2KlWazSpEh9Fxw3izyJVcZA0w2ABJNU3boj7pX0vGcPW730HB w4xvHRtZiGgxQKWgDzn+mdnQcmsVNUdcYacQPNbJPqUZ1Wr8zvcvkvycxCJnaULGiCbf wtQsmNX53g3PCyWnKn9aUKmCdmVfcYKqk3d1nXyBURFYh1HoOgSPXQ6Fh9ZZNAIju2OQ nngCbduplI3Vpx0AfXdi8ubWX4cCPx4WL03czWoFVCvcBdmw7ffAKBJNl1EwYMHDDK3R PLog== X-Gm-Message-State: AOAM533fs+r+eUIXBx0KC2N0hKG2u9Y1T+CAwJNMen8wSgLv1igAd/Jc GO7aJ6Hf4tgcpX13xQS0eTLQ X-Google-Smtp-Source: ABdhPJx/WKPeh6bi4XdXHyfkm6WgF9MpEpNOS5WgXfuCOcYnc4NTDLdqJp7IABM1iuze/eid/n11Nw== X-Received: by 2002:a05:600c:2282:: with SMTP id 2mr2712478wmf.154.1604580091299; Thu, 05 Nov 2020 04:41:31 -0800 (PST) Received: from [10.8.100.3] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id o205sm2483868wma.25.2020.11.05.04.41.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Nov 2020 04:41:30 -0800 (PST) From: "Thomas Rosenstein" To: "Toke =?utf-8?b?SMO4aWxhbmQtSsO4cmdlbnNlbg==?=" Cc: bloat@lists.bufferbloat.net Date: Thu, 05 Nov 2020 13:41:29 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <87eel8t1un.fsf@toke.dk> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <87eel8t1un.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 12:41:32 -0000 On 5 Nov 2020, at 13:38, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > "Thomas Rosenstein" writes: > >> On 5 Nov 2020, at 12:21, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >> >>> "Thomas Rosenstein" writes: >>> >>>>> If so, this sounds more like a driver issue, or maybe something to >>>>> do >>>>> with scheduling. Does it only happen with ICMP? You could try this >>>>> tool >>>>> for a userspace UDP measurement: >>>> >>>> It happens with all packets, therefore the transfer to backblaze = >>>> with >>>> 40 >>>> threads goes down to ~8MB/s instead of >60MB/s >>> >>> Huh, right, definitely sounds like a kernel bug; or maybe the new >>> kernel >>> is getting the hardware into a state where it bugs out when there = >>> are >>> lots of flows or something. >>> >>> You could try looking at the ethtool stats (ethtool -S) while = >>> running >>> the test and see if any error counters go up. Here's a handy script = >>> to >>> monitor changes in the counters: >>> >>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethto= ol_stats.pl >>> >>>> I'll try what that reports! >>>> >>>>> Also, what happens if you ping a host on the internet (*through* = >>>>> the >>>>> router instead of *to* it)? >>>> >>>> Same issue, but twice pronounced, as it seems all interfaces are >>>> affected. >>>> So, ping on one interface and the second has the issue. >>>> Also all traffic across the host has the issue, but on both sides, = >>>> so >>>> ping to the internet increased by 2x >>> >>> Right, so even an unloaded interface suffers? But this is the same >>> NIC, >>> right? So it could still be a hardware issue... >>> >>>> Yep default that CentOS ships, I just tested 4.12.5 there the issue >>>> also >>>> does not happen. So I guess I can bisect it then...(really don't = >>>> want >>>> to >>>> =F0=9F=98=83) >>> >>> Well that at least narrows it down :) >> >> I just tested 5.9.4 seems to also fix it partly, I have long = >> stretches >> where it looks good, and then some increases again. (3.10 Stock has = >> them >> too, but not so high, rather 1-3 ms) >> >> for example: >> >> 64 bytes from x.x.x.x: icmp_seq=3D10 ttl=3D64 time=3D0.169 ms >> 64 bytes from x.x.x.x: icmp_seq=3D11 ttl=3D64 time=3D5.53 ms >> 64 bytes from x.x.x.x: icmp_seq=3D12 ttl=3D64 time=3D9.44 ms >> 64 bytes from x.x.x.x: icmp_seq=3D13 ttl=3D64 time=3D0.167 ms >> 64 bytes from x.x.x.x: icmp_seq=3D14 ttl=3D64 time=3D3.88 ms >> >> and then again: >> >> 64 bytes from x.x.x.x: icmp_seq=3D15 ttl=3D64 time=3D0.569 ms >> 64 bytes from x.x.x.x: icmp_seq=3D16 ttl=3D64 time=3D0.148 ms >> 64 bytes from x.x.x.x: icmp_seq=3D17 ttl=3D64 time=3D0.286 ms >> 64 bytes from x.x.x.x: icmp_seq=3D18 ttl=3D64 time=3D0.257 ms >> 64 bytes from x.x.x.x: icmp_seq=3D19 ttl=3D64 time=3D0.220 ms >> 64 bytes from x.x.x.x: icmp_seq=3D20 ttl=3D64 time=3D0.125 ms >> 64 bytes from x.x.x.x: icmp_seq=3D21 ttl=3D64 time=3D0.188 ms >> 64 bytes from x.x.x.x: icmp_seq=3D22 ttl=3D64 time=3D0.202 ms >> 64 bytes from x.x.x.x: icmp_seq=3D23 ttl=3D64 time=3D0.195 ms >> 64 bytes from x.x.x.x: icmp_seq=3D24 ttl=3D64 time=3D0.177 ms >> 64 bytes from x.x.x.x: icmp_seq=3D25 ttl=3D64 time=3D0.242 ms >> 64 bytes from x.x.x.x: icmp_seq=3D26 ttl=3D64 time=3D0.339 ms >> 64 bytes from x.x.x.x: icmp_seq=3D27 ttl=3D64 time=3D0.183 ms >> 64 bytes from x.x.x.x: icmp_seq=3D28 ttl=3D64 time=3D0.221 ms >> 64 bytes from x.x.x.x: icmp_seq=3D29 ttl=3D64 time=3D0.317 ms >> 64 bytes from x.x.x.x: icmp_seq=3D30 ttl=3D64 time=3D0.210 ms >> 64 bytes from x.x.x.x: icmp_seq=3D31 ttl=3D64 time=3D0.242 ms >> 64 bytes from x.x.x.x: icmp_seq=3D32 ttl=3D64 time=3D0.127 ms >> 64 bytes from x.x.x.x: icmp_seq=3D33 ttl=3D64 time=3D0.217 ms >> 64 bytes from x.x.x.x: icmp_seq=3D34 ttl=3D64 time=3D0.184 ms >> >> >> For me it looks now that there was some fix between 5.4.60 and 5.9.4 = >> ... >> anyone can pinpoint it? > > $ git log --no-merges --oneline v5.4.60..v5.9.4|wc -l > 72932 > > Only 73k commits; should be easy, right? :) > > (In other words no, I have no idea; I'd suggest either (a) asking on > netdev, (b) bisecting or (c) using 5.9+ and just making peace with not > knowing). Guess I'll go the easy route and let it be ... I'll update all routers to the 5.9.4 and see if it fixes the traffic = flow - will report back once more after that. > >>>>> How did you configure the new kernel? Did you start from scratch, = >>>>> or >>>>> is >>>>> it based on the old centos config? >>>> >>>> first oldconfig and from there then added additional options for = >>>> IB, >>>> NVMe, etc (which I don't really need on the routers) >>> >>> OK, so you're probably building with roughly the same options in = >>> terms >>> of scheduling granularity etc. That's good. Did you enable spectre >>> mitigations etc on the new kernel? What's the output of >>> `tail /sys/devices/system/cpu/vulnerabilities/*` ? >> >> mitigations are off > > Right, I just figured maybe you were hitting some threshold that > involved a lot of indirect calls which slowed things down due to > mitigations. Guess not, then... > Thanks for the support :) > -Toke