From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BCBC13B2A4 for ; Fri, 6 Nov 2020 07:01:41 -0500 (EST) Received: by mail-wm1-x336.google.com with SMTP id c16so1096487wmd.2 for ; Fri, 06 Nov 2020 04:01:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=creamfinance.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ibFAjeGrc3grilkM6XBoPWh+dz/m9Ny0SnTWsUeQw+Q=; b=mFi/HsVaFHF3g48y8zjKd7kdYbzmYJqS2VYjRzelzP2Ala44WEc+NHmg7yiUSjSpjz OzeTQqnMpJMoXw+vAID7O4MK5IWO5kipismR4/nTDQlSDU8RZYm8zwRVbTBBgWB8bqHY 2En72l6lVe5x+XKSwBdwQr5ZF9IYaYbii9sxQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ibFAjeGrc3grilkM6XBoPWh+dz/m9Ny0SnTWsUeQw+Q=; b=csFnfiX6C5cK/xG7cKeoeH9aIEN1jX0+QOwy+X8OhW9PuEukfV7NGGMS6shpizK9+V J2Ib5q+RgmQha4gmGu1xUhXPgXlcGhmOlSzvIj/vUic8hOoGOyjxTUjy3rVyEklRG+96 X+1eIISP4RookYpdFHPNEURwPZfBpn9fCvZ1YjQ9w/rZ7VDpVpSw/L018DaplLu5taAX 6kiuOFeCe4Io9Gh0DpMtaJK662acREThc30gishqNKwAOxpq5uUETyuvdImwhCcWIgQk RgNCxWoWaRJ8t8hEQepaJ7jc5cz4TWohIiX4lAUDbMCuoXMgFQ26VhnRad+GDZOYSkiF YZzw== X-Gm-Message-State: AOAM530Ldtn9cBCP9Syq+zos/RdVfR9GkTVhEF+aFauSCfUCRqc8iDTy a5taEYwLnWDUW9GgQui14ipV X-Google-Smtp-Source: ABdhPJw7qkEjGnYvCL8r0oAaLGeLZf30XZTbjpVjZaGsEN+BWy18O36kYOGF+AtOxeWD3lyfyzyqqQ== X-Received: by 2002:a1c:4957:: with SMTP id w84mr2095030wma.84.1604664100497; Fri, 06 Nov 2020 04:01:40 -0800 (PST) Received: from [10.8.100.3] (ip-185.208.132.9.cf-it.at. [185.208.132.9]) by smtp.gmail.com with ESMTPSA id c185sm2009190wma.44.2020.11.06.04.01.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Nov 2020 04:01:39 -0800 (PST) From: "Thomas Rosenstein" To: "Toke =?utf-8?b?SMO4aWxhbmQtSsO4cmdlbnNlbg==?=" Cc: "Jesper Dangaard Brouer" , Bloat Date: Fri, 06 Nov 2020 13:01:38 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: <56413531-311F-40CC-B78F-25A39D9F48BE@creamfinance.com> In-Reply-To: <87blgaso84.fsf@toke.dk> References: <87imalumps.fsf@toke.dk> <871rh8vf1p.fsf@toke.dk> <81ED2A33-D366-42FC-9344-985FEE8F11BA@creamfinance.com> <87sg9ot5f1.fsf@toke.dk> <20201105143317.78276bbc@carbon> <11812D44-BD46-4CA4-BA39-6080BD88F163@creamfinance.com> <20201106121840.7959ae4b@carbon> <87blgaso84.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit Subject: Re: [Bloat] Router congestion, slow ping/ack times with kernel 5.4.60 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 12:01:41 -0000 On 6 Nov 2020, at 12:45, Toke Høiland-Jørgensen wrote: > "Thomas Rosenstein" writes: > >> On 6 Nov 2020, at 12:18, Jesper Dangaard Brouer wrote: >> >>> On Fri, 06 Nov 2020 10:18:10 +0100 >>> "Thomas Rosenstein" wrote: >>> >>>>>> I just tested 5.9.4 seems to also fix it partly, I have long >>>>>> stretches where it looks good, and then some increases again. >>>>>> (3.10 >>>>>> Stock has them too, but not so high, rather 1-3 ms) >>>>>> >>> >>> That you have long stretches where latency looks good is interesting >>> information. My theory is that your system have a periodic >>> userspace >>> process that does a kernel syscall that takes too long, blocking >>> network card from processing packets. (Note it can also be a kernel >>> thread). >> >> The weird part is, I first only updated router-02 and pinged to >> router-04 (out of traffic flow), there I noticed these long stretches >> of >> ok ping. >> >> When I updated also router-03 and router-04, the old behaviour kind >> of >> was back, this confused me. >> >> Could this be related to netlink? I have gobgpd running on these >> routers, which injects routes via netlink. >> But the churn rate during the tests is very minimal, maybe 30 - 40 >> routes every second. >> >> Otherwise we got: salt-minion, collectd, node_exporter, sshd > > collectd may be polling the interface stats; try turning that off? I can, but shouldn't that also influence iperf3 performance then? > >>> >>> Another theory is the NIC HW does strange things, but it is not very >>> likely. E.g. delaying the packets before generating the IRQ >>> interrupt, >>> which hide it from my IRQ-to-softirq latency tool. >>> >>> A question: What traffic control qdisc are you using on your system? >> >> kernel 4+ uses pfifo, but there's no dropped packets >> I have also tested with fq_codel, same behaviour and also no >> weirdness >> in the packets queue itself >> >> kernel 3.10 uses mq, and for the vlan interfaces noqueue > > Do you mean that you only have a single pfifo qdisc on kernel 4+? Why > is > it not using mq? oh, actually, I just noticed that's a remnant of the previous tests, I had net.core.default_qdisc = fq_codel in the sysctl.conf... so disregard my previous wrong info so all kernel by default look like that, mq + pfifo_fast: qdisc noqueue 0: dev lo root refcnt 2 qdisc mq 0: dev eth0 root qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev eth1 root qdisc pfifo_fast 0: dev eth1 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev eth2 root qdisc pfifo_fast 0: dev eth2 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth2 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev eth3 root qdisc pfifo_fast 0: dev eth3 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth3 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc mq 0: dev eth4 root qdisc pfifo_fast 0: dev eth4 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :11 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :12 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :13 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :14 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :15 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :16 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :17 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 parent :18 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc noqueue 0: dev eth4.2300 root refcnt 2 qdisc noqueue 0: dev eth4.2503 root refcnt 2 > > Was there anything in the ethtool stats? no, just rx / tx increase, not a single err > > -Toke