From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-x243.google.com (mail-pg0-x243.google.com [IPv6:2607:f8b0:400e:c05::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D78123B29E for ; Wed, 25 Jan 2017 18:53:07 -0500 (EST) Received: by mail-pg0-x243.google.com with SMTP id 204so20905912pge.2 for ; Wed, 25 Jan 2017 15:53:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=2UDwcJsMCCFFznfkaqMuXSv1YMYkvekzF22ktB7QlZg=; b=pb5VST43aDfwclCggMB2teH7CnL7OjWCfZIaEecA7GgyuwdZS2L7pkAKv5lrTsmDyA rBF+c5JTepbjmvRhxJIZT5WaUhS7ZOGkGX+i1gKQz6tbMPSyXVqpxv/mk5WeCx3ICCTU SGUbcNF50GjVQ3BSoqrVgRQcnaYRWU6cQ5aSdGX8rBd/rcTRZpcLKba48rxUkpI0xKH0 fbD+A/9X4YzC/4OXWUoIcVN7dyrRct0AXE3M4UffYg4yKGIm2BHmEVqdqRaqm9Qvnhx4 WSyVyCe4RBtewXMd8hFr63PWNEEomH6EmJwYDE6r5KwyIT1Fmapb16+Tf0apvdE1iQqi ejow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=2UDwcJsMCCFFznfkaqMuXSv1YMYkvekzF22ktB7QlZg=; b=APsmcAp4pgpgadkpGn1xa2HD35H6xoPG5vLizuanRfUZ00GelrS9ytbIifbkwgxhKt 65khKVk5NYejbwFjI8ZqxSOoVDdO33SEbNATnmj+L7n1EWaXxdBZ5yTUuWeR3KjKoFb7 kByQ95BbYWtKlre/gqU2xtgZpYSt1PABhEKhAJUWwSVaMAYq34Lc2bQOMMiLTj1hhPWZ FzmOLjqWgXfxTjW719r4gxFOWKNSa5XOzoXswomVSLQtzCL6j+N+B9pHbiOXDsqx7dhb pmEGJq56VS9GtT6D/dKemPONl6wVegq5DKg4Y/AubQCE0UOUrR508kCa66Xnhpf4aOoC 3ayQ== X-Gm-Message-State: AIkVDXIYp0m8/TuDdlDPrREFUztSUJE9d1yidaG9x6H1NVLz5CuxdDdP1aSuPry2FsDreQ== X-Received: by 10.99.174.4 with SMTP id q4mr3303pgf.186.1485388386911; Wed, 25 Jan 2017 15:53:06 -0800 (PST) Received: from ?IPv6:2620:0:1000:1704:901f:c554:68ed:6757? ([2620:0:1000:1704:901f:c554:68ed:6757]) by smtp.googlemail.com with ESMTPSA id g85sm3626574pfe.38.2017.01.25.15.53.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 15:53:06 -0800 (PST) Message-ID: <1485388385.5145.87.camel@edumazet-glaptop3.roam.corp.google.com> From: Eric Dumazet To: Hans-Kristian Bakke Cc: Neal Cardwell , bloat Date: Wed, 25 Jan 2017 15:53:05 -0800 In-Reply-To: References: <1485387201.5145.81.camel@edumazet-glaptop3.roam.corp.google.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 23:53:08 -0000 On Thu, 2017-01-26 at 00:47 +0100, Hans-Kristian Bakke wrote: > > > I did record the qdisc settings, but I didn't capture the stats, but > throttling is definitively active when I watch the tc -s stats in > realtime when testing (looking at tun1) > > > ​tc -s qdisc show > qdisc noqueue 0: dev lo root refcnt 2 > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > qdisc fq 8007: dev eth0 root refcnt 2 limit 10000p flow_limit 100p > buckets 1024 orphan_mask 1023 quantum 3028 initial_quantum 15140 > refill_delay 40.0ms > Sent 1420855729 bytes 969198 pkt (dropped 134, overlimits 0 requeues > 0) > backlog 0b 0p requeues 0 > 124 flows (123 inactive, 0 throttled) > 0 gc, 0 highprio, 3 throttled, 3925 ns latency, 134 flows_plimit You seem to hit the "flow_limit 100" maybe because all packets are going through a single encap flow. ( 134 drops ) > qdisc fq 8008: dev tun1 root refcnt 2 limit 10000p flow_limit 100p > buckets 1024 orphan_mask 1023 quantum 3000 initial_quantum 15000 > refill_delay 40.0ms > Sent 1031289740 bytes 741181 pkt (dropped 0, overlimits 0 requeues 0) > backlog 101616b 3p requeues 0 > 16 flows (15 inactive, 1 throttled), next packet delay 351937 ns > 0 gc, 0 highprio, 58377 throttled, 12761 ns latency > ​ > Looks good, although latency seems a bit high, thanks ! > > > On 26 January 2017 at 00:33, Eric Dumazet > wrote: > > On Thu, 2017-01-26 at 00:04 +0100, Hans-Kristian Bakke wrote: > > I can do that. I guess I should do the capture from tun1 as > that is > > the place that the tcp-traffic is visible? My non-virtual > nic is only > > seeing OpenVPN encapsulated UDP-traffic. > > > > But is FQ installed at the point TCP sockets are ? > > You should give us "tc -s qdisc show xxx" so that we can > check if > pacing (throttling) actually happens. > > > > On 25 January 2017 at 23:48, Neal Cardwell > > > wrote: > > On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke > > wrote: > > Actually.. the 1-4 mbit/s results with fq > sporadically > > appears again as I keep testing but it is > most likely > > caused by all the unknowns between me an my > > testserver. But still, changing to > pfifo_qdisc seems > > to normalize the throughput again with BBR, > could this > > be one of those times where BBR and pacing > actually is > > getting hurt for playing nice in some very > variable > > bottleneck on the way? > > > > > > Possibly. Would you be able to take a tcpdump trace > of each > > trial (headers only would be ideal), and post on a > web site > > somewhere a pcap trace for one of the slow trials? > > > > > > For example: > > > > > > tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c > 1000000 & > > > > > > > > thanks, > > neal > > > > > > > > > > On 25 January 2017 at 23:01, Neal Cardwell > > wrote: > > On Wed, Jan 25, 2017 at 3:54 PM, > Hans-Kristian > > Bakke wrote: > > Hi > > > > > > Kernel 4.9 finally landed in > Debian > > testing so I could finally > test BBR in > > a real life environment that > I have > > struggled with getting any > kind of > > performance out of. > > > > > > The challenge at hand is UDP > based > > OpenVPN through europe at > around 35 ms > > rtt to my VPN-provider with > plenty of > > available bandwith available > in both > > ends and everything > completely unknown > > in between. After tuning the > > UDP-buffers up to make room > for my 500 > > mbit/s symmetrical bandwith > at 35 ms > > the download part seemed to > work > > nicely at an unreliable 150 > to 300 > > mbit/s, while the upload was > stuck at > > 30 to 60 mbit/s. > > > > > > Just by activating BBR the > bandwith > > instantly shot up to around > 150 mbit/s > > using a fat tcp test to a > public > > iperf3 server located near > my VPN exit > > point in the Netherlands. > Replace BBR > > with qubic again and the > performance > > is once again all over the > place > > ranging from very bad to > bad, but > > never better than 1/3 of > BBRs "steady > > state". In other words > "instant WIN!" > > > > > > Glad to hear it. Thanks for the test > report! > > > > However, seeing the > requirement of fq > > and pacing for BBR and > noticing that I > > am running pfifo_fast within > a VM with > > virtio NIC on a Proxmox VE > host with > > fq_codel on all physical > interfaces, I > > was surprised to see that it > worked so > > well. > > I then replaced pfifo_fast > with fq and > > the performance went right > down to > > only 1-4 mbit/s from around > 150 > > mbit/s. Removing the fq > again regained > > the performance at once. > > > > > > I have got some questions to > you guys > > that know a lot more than me > about > > these things: > > 1. Do fq (and fq_codel) even > work > > reliably in a VM? What is > the best > > choice for default qdisc to > use in a > > VM in general? > > > > > > Eric covered this one. We are not > aware of > > specific issues with fq in VM > environments. > > And we have tested that fq works > sufficiently > > well on Google Cloud VMs. > > > > 2. Why do BBR immediately > "fix" all my > > issues with upload through > that > > "unreliable" big BDP link > with > > pfifo_fast when fq pacing is > a > > requirement? > > > > > > For BBR, pacing is part of the > design in order > > to make BBR more "gentle" in terms > of the rate > > at which it sends, in order to put > less > > pressure on buffers and keep packet > loss > > lower. This is particularly > important when a > > BBR flow is restarting from idle. In > this case > > BBR starts with a full cwnd, and it > counts on > > pacing to pace out the packets at > the > > estimated bandwidth, so that the > queue can > > stay relatively short and yet the > pipe can be > > filled immediately. > > > > > > Running BBR without pacing makes BBR > more > > aggressive, particularly in > restarting from > > idle, but also in the steady state, > where BBR > > tries to use pacing to keep the > queue short. > > > > > > For bulk transfer tests with one > flow, running > > BBR without pacing will likely cause > higher > > queues and loss rates at the > bottleneck, which > > may negatively impact other traffic > sharing > > that bottleneck. > > > > 3. Could fq_codel on the > physical host > > be the reason that it still > works? > > > > > > Nope, fq_codel does not implement > pacing. > > > > 4. Do BBR _only_ work with > fq pacing > > or could fq_codel be used as > a > > replacement? > > > > > > Nope, BBR needs pacing to work > correctly, and > > currently fq is the only Linux qdisc > that > > implements pacing. > > > > 5. Is BBR perhaps modified > to do the > > right thing without having > to change > > the qdisc in the current > kernel 4.9? > > > > > > Nope. Linux 4.9 contains the initial > public > > release of BBR from September 2016. > And there > > have been no code changes since then > (just > > expanded comments). > > > > > > Thanks for the test report! > > > > > > neal > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > > >