From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B57863B29E for ; Wed, 25 Jan 2017 18:33:23 -0500 (EST) Received: by mail-pg0-x244.google.com with SMTP id 75so20875898pgf.3 for ; Wed, 25 Jan 2017 15:33:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=3wLbr33Gro9Ols57QHo3U5pKQEuVmPYN6j4/lwLv2Po=; b=LkEvtCmwLBIAfIhb05UKUG9cpQY64sz1giY2f1M49DUsyKaBiOeey27WvcOqZzFIF+ USMJ8oDHqrvdYQWEMx6+dbUONrOyjE4XB97jyxtlhI7aM6Fv35IkFLTz1PxdIXsnHCdr Zq5XNMyeR1XSDiFXRMdOfoAeD2mq8EnP6odnuByLLOjNxbsU6FstCiO8zeaydvHW5QI5 jLNn2/g/ccUNiXGnHINok2nkAW1b8jcxiJKC0gG999AbBRpmthYlLVTvsZ6YQMwoZ4p8 WmGz0UZs4pYCRU3oESb5INTFkgLL2mWcHpwcV/QPQlxhW9ze6RziYROk7vV6wOWUJO8m VTdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=3wLbr33Gro9Ols57QHo3U5pKQEuVmPYN6j4/lwLv2Po=; b=erOoVW4bi/fHUnMByMeLzNiw3fisEgyvB1BDPMUi3Ywykm6mWu1hYuyV4pja48bBAH Q7TsvAWCzrAULVnRGizPh/kfq47x9Hj9CVeIb35B0g+/Eekr6LuGEC/oEI8Z5/ZyHVcA 14t1yilxryFpwC/+0MGu1GIRLLzVwTDo5FfJ4CH2R3AmKNM2sC7PTDjXrdYN6w7a6VnV CPPf5+WMiIiIJjGs0ph/tTPwq3E/6h4dgWvFHeMR+nt/PZz7p+roLqTv5nHyVdoq9gaY O1xuLGxPLWEOKf+4o2W+Qap0ch3ebUYerY1RdFFch90TJKDTjCXMkSomKe4GYYqYSbxj eq6g== X-Gm-Message-State: AIkVDXJRCSvGTfYxOJ+5h1dEw4JpUe0KZxJUXYqKe+iH6cikY0mo5Zb1i5MJAhunF6O2tg== X-Received: by 10.98.194.22 with SMTP id l22mr49684845pfg.178.1485387202840; Wed, 25 Jan 2017 15:33:22 -0800 (PST) Received: from ?IPv6:2620:0:1000:1704:901f:c554:68ed:6757? ([2620:0:1000:1704:901f:c554:68ed:6757]) by smtp.googlemail.com with ESMTPSA id p66sm3484454pfb.88.2017.01.25.15.33.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 15:33:22 -0800 (PST) Message-ID: <1485387201.5145.81.camel@edumazet-glaptop3.roam.corp.google.com> From: Eric Dumazet To: Hans-Kristian Bakke Cc: Neal Cardwell , bloat Date: Wed, 25 Jan 2017 15:33:21 -0800 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 23:33:23 -0000 On Thu, 2017-01-26 at 00:04 +0100, Hans-Kristian Bakke wrote: > I can do that. I guess I should do the capture from tun1 as that is > the place that the tcp-traffic is visible? My non-virtual nic is only > seeing OpenVPN encapsulated UDP-traffic. > But is FQ installed at the point TCP sockets are ? You should give us "tc -s qdisc show xxx" so that we can check if pacing (throttling) actually happens. > On 25 January 2017 at 23:48, Neal Cardwell > wrote: > On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke > wrote: > Actually.. the 1-4 mbit/s results with fq sporadically > appears again as I keep testing but it is most likely > caused by all the unknowns between me an my > testserver. But still, changing to pfifo_qdisc seems > to normalize the throughput again with BBR, could this > be one of those times where BBR and pacing actually is > getting hurt for playing nice in some very variable > bottleneck on the way? > > > Possibly. Would you be able to take a tcpdump trace of each > trial (headers only would be ideal), and post on a web site > somewhere a pcap trace for one of the slow trials? > > > For example: > > > tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 & > > > > thanks, > neal > > > > > On 25 January 2017 at 23:01, Neal Cardwell > wrote: > On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian > Bakke wrote: > Hi > > > Kernel 4.9 finally landed in Debian > testing so I could finally test BBR in > a real life environment that I have > struggled with getting any kind of > performance out of. > > > The challenge at hand is UDP based > OpenVPN through europe at around 35 ms > rtt to my VPN-provider with plenty of > available bandwith available in both > ends and everything completely unknown > in between. After tuning the > UDP-buffers up to make room for my 500 > mbit/s symmetrical bandwith at 35 ms > the download part seemed to work > nicely at an unreliable 150 to 300 > mbit/s, while the upload was stuck at > 30 to 60 mbit/s. > > > Just by activating BBR the bandwith > instantly shot up to around 150 mbit/s > using a fat tcp test to a public > iperf3 server located near my VPN exit > point in the Netherlands. Replace BBR > with qubic again and the performance > is once again all over the place > ranging from very bad to bad, but > never better than 1/3 of BBRs "steady > state". In other words "instant WIN!" > > > Glad to hear it. Thanks for the test report! > > However, seeing the requirement of fq > and pacing for BBR and noticing that I > am running pfifo_fast within a VM with > virtio NIC on a Proxmox VE host with > fq_codel on all physical interfaces, I > was surprised to see that it worked so > well. > I then replaced pfifo_fast with fq and > the performance went right down to > only 1-4 mbit/s from around 150 > mbit/s. Removing the fq again regained > the performance at once. > > > I have got some questions to you guys > that know a lot more than me about > these things: > 1. Do fq (and fq_codel) even work > reliably in a VM? What is the best > choice for default qdisc to use in a > VM in general? > > > Eric covered this one. We are not aware of > specific issues with fq in VM environments. > And we have tested that fq works sufficiently > well on Google Cloud VMs. > > 2. Why do BBR immediately "fix" all my > issues with upload through that > "unreliable" big BDP link with > pfifo_fast when fq pacing is a > requirement? > > > For BBR, pacing is part of the design in order > to make BBR more "gentle" in terms of the rate > at which it sends, in order to put less > pressure on buffers and keep packet loss > lower. This is particularly important when a > BBR flow is restarting from idle. In this case > BBR starts with a full cwnd, and it counts on > pacing to pace out the packets at the > estimated bandwidth, so that the queue can > stay relatively short and yet the pipe can be > filled immediately. > > > Running BBR without pacing makes BBR more > aggressive, particularly in restarting from > idle, but also in the steady state, where BBR > tries to use pacing to keep the queue short. > > > For bulk transfer tests with one flow, running > BBR without pacing will likely cause higher > queues and loss rates at the bottleneck, which > may negatively impact other traffic sharing > that bottleneck. > > 3. Could fq_codel on the physical host > be the reason that it still works? > > > Nope, fq_codel does not implement pacing. > > 4. Do BBR _only_ work with fq pacing > or could fq_codel be used as a > replacement? > > > Nope, BBR needs pacing to work correctly, and > currently fq is the only Linux qdisc that > implements pacing. > > 5. Is BBR perhaps modified to do the > right thing without having to change > the qdisc in the current kernel 4.9? > > > Nope. Linux 4.9 contains the initial public > release of BBR from September 2016. And there > have been no code changes since then (just > expanded comments). > > > Thanks for the test report! > > > neal > > > > > > > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat