From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5BDE03B29E for ; Wed, 25 Jan 2017 19:10:29 -0500 (EST) Received: by mail-pf0-x241.google.com with SMTP id 19so15289705pfo.3 for ; Wed, 25 Jan 2017 16:10:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=fZ6jIyMyVtRqWZgpQq9D+zskaZAttkMkm9g5pKkZ+pU=; b=HyAXtnSDWp6u0bSRy5lewCQjKBi5MvIxU97tgKaC00PsNIeTUopfLXGC/Cy/W5h2T6 6KRhJV7h0jhfH7IankspbttkZAIJ69xeJe59KQvhsbC9IJ6eeu/yO9QDhwZrDctL3LoI rzJtvCdJQ5zVNb7X29GjrNHmeS9NqrFG8sf3jIl2lc/xKOvB2aeAgOGJUp2NE/PZ4QFw V/9sNBzzr4hxqhCRCue+42oOkhPOAa/2DdSg9EwOhmUcUkDP4Kg8bVKsNlLcvsEdnkgn BHtvC6B8TUpDsZ2TgfnPtDTKUuto+Cp+//HUe6WqiCSnnCKQxvW1TWr8fH/r9y3+GyAl AY9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=fZ6jIyMyVtRqWZgpQq9D+zskaZAttkMkm9g5pKkZ+pU=; b=MCJkJZvj/dnFgGzNVwUcVgd2E4I0++az7xRn5eAq2yOo7a6HDwXV1ubpi3G/wEsQLa 036eEljGJJ7OFRhQ/EFq86rvKRcF/k2OjNUgCQ06JlqSTeUA0lhWliHHFThWUTOr18A7 bDZ9VJOvY2i09UIzrgOfDZTqBOO92aY1JpppAbLvRhACACznQh9Mqszj9C12UEYkC0u+ jlYym12cEa+P6IYybFUwkfrTibosF4wC+jsUEdDanzwolhNZvv1mrxABuLc3LsZYjYft r1kvdUwaqI1huMUDoR7k9byfCaWlo2eCWFEFG+Pis4vRxDl7U9xkn+/SXY4UaePkoRwg B92A== X-Gm-Message-State: AIkVDXLk7GmdC0v2pYpSu7JHf+F0FH8nn/e8buPafrSTrrETm6ZmN5ABMVMFWvfjmlNhjg== X-Received: by 10.98.215.27 with SMTP id b27mr90893pfh.70.1485389428425; Wed, 25 Jan 2017 16:10:28 -0800 (PST) Received: from ?IPv6:2620:0:1000:1704:901f:c554:68ed:6757? ([2620:0:1000:1704:901f:c554:68ed:6757]) by smtp.googlemail.com with ESMTPSA id g70sm3623759pfb.50.2017.01.25.16.10.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 16:10:27 -0800 (PST) Message-ID: <1485389427.5145.92.camel@edumazet-glaptop3.roam.corp.google.com> From: Eric Dumazet To: Hans-Kristian Bakke Cc: Neal Cardwell , bloat Date: Wed, 25 Jan 2017 16:10:27 -0800 In-Reply-To: References: <1485387201.5145.81.camel@edumazet-glaptop3.roam.corp.google.com> <1485388385.5145.87.camel@edumazet-glaptop3.roam.corp.google.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jan 2017 00:10:29 -0000 On Thu, 2017-01-26 at 00:56 +0100, Hans-Kristian Bakke wrote: > These are just the fq settings as they get applied when having fq as > default qdiscs. I guess there are room for improvements on those > default settings depending on use case. > > > For future reference: should I increase the limit on drops or is it > okay as it is? For your use case, increasing flow_limit to 1000 or even 2000 on eth0 would be absolutely fine, since most of your traffic is going to be encapsulated. Note that this setup (vpn) was probably breaking back pressure (TCP Small Queues is relying on this), so adding FQ/pacing probably helps, even with Cubic. > > On 26 January 2017 at 00:53, Eric Dumazet > wrote: > On Thu, 2017-01-26 at 00:47 +0100, Hans-Kristian Bakke wrote: > > > > > > I did record the qdisc settings, but I didn't capture the > stats, but > > throttling is definitively active when I watch the tc -s > stats in > > realtime when testing (looking at tun1) > > > > > > ​tc -s qdisc show > > qdisc noqueue 0: dev lo root refcnt 2 > > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > > backlog 0b 0p requeues 0 > > qdisc fq 8007: dev eth0 root refcnt 2 limit 10000p > flow_limit 100p > > buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum > 15140 > > refill_delay 40.0ms > > Sent 1420855729 bytes 969198 pkt (dropped 134, overlimits 0 > requeues > > 0) > > backlog 0b 0p requeues 0 > > 124 flows (123 inactive, 0 throttled) > > 0 gc, 0 highprio, 3 throttled, 3925 ns latency, 134 > flows_plimit > > You seem to hit the "flow_limit 100" maybe because all packets > are going > through a single encap flow. ( 134 drops ) > > > qdisc fq 8008: dev tun1 root refcnt 2 limit 10000p > flow_limit 100p > > buckets 1024 orphan_mask 1023 quantum 3000 initial_quantum > 15000 > > refill_delay 40.0ms > > Sent 1031289740 bytes 741181 pkt (dropped 0, overlimits 0 > requeues 0) > > backlog 101616b 3p requeues 0 > > 16 flows (15 inactive, 1 throttled), next packet delay > 351937 ns > > 0 gc, 0 highprio, 58377 throttled, 12761 ns latency > > ​ > > > > Looks good, although latency seems a bit high, thanks ! > > > > > > On 26 January 2017 at 00:33, Eric Dumazet > > > wrote: > > > > On Thu, 2017-01-26 at 00:04 +0100, Hans-Kristian > Bakke wrote: > > > I can do that. I guess I should do the capture > from tun1 as > > that is > > > the place that the tcp-traffic is visible? My > non-virtual > > nic is only > > > seeing OpenVPN encapsulated UDP-traffic. > > > > > > > But is FQ installed at the point TCP sockets are ? > > > > You should give us "tc -s qdisc show xxx" so that > we can > > check if > > pacing (throttling) actually happens. > > > > > > > On 25 January 2017 at 23:48, Neal Cardwell > > > > > wrote: > > > On Wed, Jan 25, 2017 at 5:38 PM, > Hans-Kristian Bakke > > > wrote: > > > Actually.. the 1-4 mbit/s results > with fq > > sporadically > > > appears again as I keep testing > but it is > > most likely > > > caused by all the unknowns between > me an my > > > testserver. But still, changing to > > pfifo_qdisc seems > > > to normalize the throughput again > with BBR, > > could this > > > be one of those times where BBR > and pacing > > actually is > > > getting hurt for playing nice in > some very > > variable > > > bottleneck on the way? > > > > > > > > > Possibly. Would you be able to take a > tcpdump trace > > of each > > > trial (headers only would be ideal), and > post on a > > web site > > > somewhere a pcap trace for one of the slow > trials? > > > > > > > > > For example: > > > > > > > > > tcpdump -n -w /tmp/out.pcap -s 120 -i > eth0 -c > > 1000000 & > > > > > > > > > > > > thanks, > > > neal > > > > > > > > > > > > > > > On 25 January 2017 at 23:01, Neal > Cardwell > > > wrote: > > > On Wed, Jan 25, 2017 at > 3:54 PM, > > Hans-Kristian > > > Bakke > wrote: > > > Hi > > > > > > > > > Kernel 4.9 finally > landed in > > Debian > > > testing so I could > finally > > test BBR in > > > a real life > environment that > > I have > > > struggled with > getting any > > kind of > > > performance out > of. > > > > > > > > > The challenge at > hand is UDP > > based > > > OpenVPN through > europe at > > around 35 ms > > > rtt to my > VPN-provider with > > plenty of > > > available bandwith > available > > in both > > > ends and > everything > > completely unknown > > > in between. After > tuning the > > > UDP-buffers up to > make room > > for my 500 > > > mbit/s symmetrical > bandwith > > at 35 ms > > > the download part > seemed to > > work > > > nicely at an > unreliable 150 > > to 300 > > > mbit/s, while the > upload was > > stuck at > > > 30 to 60 mbit/s. > > > > > > > > > Just by activating > BBR the > > bandwith > > > instantly shot up > to around > > 150 mbit/s > > > using a fat tcp > test to a > > public > > > iperf3 server > located near > > my VPN exit > > > point in the > Netherlands. > > Replace BBR > > > with qubic again > and the > > performance > > > is once again all > over the > > place > > > ranging from very > bad to > > bad, but > > > never better than > 1/3 of > > BBRs "steady > > > state". In other > words > > "instant WIN!" > > > > > > > > > Glad to hear it. Thanks > for the test > > report! > > > > > > However, seeing > the > > requirement of fq > > > and pacing for BBR > and > > noticing that I > > > am running > pfifo_fast within > > a VM with > > > virtio NIC on a > Proxmox VE > > host with > > > fq_codel on all > physical > > interfaces, I > > > was surprised to > see that it > > worked so > > > well. > > > I then replaced > pfifo_fast > > with fq and > > > the performance > went right > > down to > > > only 1-4 mbit/s > from around > > 150 > > > mbit/s. Removing > the fq > > again regained > > > the performance at > once. > > > > > > > > > I have got some > questions to > > you guys > > > that know a lot > more than me > > about > > > these things: > > > 1. Do fq (and > fq_codel) even > > work > > > reliably in a VM? > What is > > the best > > > choice for default > qdisc to > > use in a > > > VM in general? > > > > > > > > > Eric covered this one. We > are not > > aware of > > > specific issues with fq in > VM > > environments. > > > And we have tested that > fq works > > sufficiently > > > well on Google Cloud VMs. > > > > > > 2. Why do BBR > immediately > > "fix" all my > > > issues with upload > through > > that > > > "unreliable" big > BDP link > > with > > > pfifo_fast when fq > pacing is > > a > > > requirement? > > > > > > > > > For BBR, pacing is part of > the > > design in order > > > to make BBR more "gentle" > in terms > > of the rate > > > at which it sends, in > order to put > > less > > > pressure on buffers and > keep packet > > loss > > > lower. This is > particularly > > important when a > > > BBR flow is restarting > from idle. In > > this case > > > BBR starts with a full > cwnd, and it > > counts on > > > pacing to pace out the > packets at > > the > > > estimated bandwidth, so > that the > > queue can > > > stay relatively short and > yet the > > pipe can be > > > filled immediately. > > > > > > > > > Running BBR without pacing > makes BBR > > more > > > aggressive, particularly > in > > restarting from > > > idle, but also in the > steady state, > > where BBR > > > tries to use pacing to > keep the > > queue short. > > > > > > > > > For bulk transfer tests > with one > > flow, running > > > BBR without pacing will > likely cause > > higher > > > queues and loss rates at > the > > bottleneck, which > > > may negatively impact > other traffic > > sharing > > > that bottleneck. > > > > > > 3. Could fq_codel > on the > > physical host > > > be the reason that > it still > > works? > > > > > > > > > Nope, fq_codel does not > implement > > pacing. > > > > > > 4. Do BBR _only_ > work with > > fq pacing > > > or could fq_codel > be used as > > a > > > replacement? > > > > > > > > > Nope, BBR needs pacing to > work > > correctly, and > > > currently fq is the only > Linux qdisc > > that > > > implements pacing. > > > > > > 5. Is BBR perhaps > modified > > to do the > > > right thing > without having > > to change > > > the qdisc in the > current > > kernel 4.9? > > > > > > > > > Nope. Linux 4.9 contains > the initial > > public > > > release of BBR from > September 2016. > > And there > > > have been no code changes > since then > > (just > > > expanded comments). > > > > > > > > > Thanks for the test > report! > > > > > > > > > neal > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > > > > > > > > > > > >