From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x235.google.com (mail-oi0-x235.google.com [IPv6:2607:f8b0:4003:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2F2763B29E for ; Wed, 25 Jan 2017 17:48:59 -0500 (EST) Received: by mail-oi0-x235.google.com with SMTP id w204so126478891oiw.0 for ; Wed, 25 Jan 2017 14:48:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=RJfE4eE8Qlc1i8+LH4awE+VlPt4/To1/xI7VOJMlTxc=; b=kDdIZq69Gfn8PlZiU6ncRjoOqjprDcGp3FosUO/S+zbL3nOWyg3H7fsJ1JdaM4zUGE MzbkVs7Uz4uxzZSA3jhrv9SkHD2FGvKdYlPrTliq3/mYnEyY0arUCbDtq4L19hM5KgnR nleAubYv1LjdY9/mOddRvtw4bRXfFR3BgkYABEqs8eoHHdNlOPU4xcqFk6NxvFusQoP8 2n+QGsFLmfzX4JNyFlS83pnK2PkZBnTkvIu0NhMTPgDXAnWVh6G/Y2cNKJtr+95zNAGe VK48SIvWGsFxzHc5CiFWUhUoP1NMYnfLkHDsMrcduTTJhp/56xrRVK3TszqTHeqJApyO a6+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RJfE4eE8Qlc1i8+LH4awE+VlPt4/To1/xI7VOJMlTxc=; b=Z+5eF9SIL4MOVQJk/k+PXPrdQiYJa8hu4n78wxUg6rWdKcDEtuXiWgu9Upu8O5eOvp /ch/TMT8QGsVLwc4NxTn73JGMvcpdN4vdK3hyXFpQlzBGsVUTGVjzZYEqSzFAf8TValR fpY2XhmKwAmie72oFToG6OYUPaYzvC9sSeojhqCvGp7T9otTq/NCjd4xWe+p0WA9vH5d SaMU3T2iaQ+CKUBrYSS/JmN/J3qsxI90EWqC/g9O/Iy3RIxPe4IQtI5t2Gv+i0395EMz xnkx1lI9hEqnvEULh9nyY5rVn2yrsSDZbV+6VYu1N+28T+ub/o4QWPm2n8u5YB3jTHQU koPQ== X-Gm-Message-State: AIkVDXKIekd48rRFKnnLSZLKxmdlHi1NkfZ/+Rw/nfkRkXnQ33p9R/hWiXFtQe/XURLvETmWrC/hq/PJ6s+g4qBh X-Received: by 10.202.108.5 with SMTP id h5mr22231589oic.29.1485384537644; Wed, 25 Jan 2017 14:48:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.244.203 with HTTP; Wed, 25 Jan 2017 14:48:27 -0800 (PST) In-Reply-To: References: From: Neal Cardwell Date: Wed, 25 Jan 2017 17:48:27 -0500 Message-ID: To: Hans-Kristian Bakke Cc: bloat Content-Type: multipart/alternative; boundary=001a1142dfaeb904de0546f30982 Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 22:48:59 -0000 --001a1142dfaeb904de0546f30982 Content-Type: text/plain; charset=UTF-8 On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke wrote: > Actually.. the 1-4 mbit/s results with fq sporadically appears again as I > keep testing but it is most likely caused by all the unknowns between me an > my testserver. But still, changing to pfifo_qdisc seems to normalize the > throughput again with BBR, could this be one of those times where BBR and > pacing actually is getting hurt for playing nice in some very variable > bottleneck on the way? > Possibly. Would you be able to take a tcpdump trace of each trial (headers only would be ideal), and post on a web site somewhere a pcap trace for one of the slow trials? For example: tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 & thanks, neal > > On 25 January 2017 at 23:01, Neal Cardwell wrote: > >> On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke >> wrote: >> >>> Hi >>> >>> Kernel 4.9 finally landed in Debian testing so I could finally test BBR >>> in a real life environment that I have struggled with getting any kind of >>> performance out of. >>> >>> The challenge at hand is UDP based OpenVPN through europe at around 35 >>> ms rtt to my VPN-provider with plenty of available bandwith available in >>> both ends and everything completely unknown in between. After tuning the >>> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms >>> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s, >>> while the upload was stuck at 30 to 60 mbit/s. >>> >>> Just by activating BBR the bandwith instantly shot up to around 150 >>> mbit/s using a fat tcp test to a public iperf3 server located near my VPN >>> exit point in the Netherlands. Replace BBR with qubic again and the >>> performance is once again all over the place ranging from very bad to bad, >>> but never better than 1/3 of BBRs "steady state". In other words "instant >>> WIN!" >>> >> >> Glad to hear it. Thanks for the test report! >> >> >>> However, seeing the requirement of fq and pacing for BBR and noticing >>> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE >>> host with fq_codel on all physical interfaces, I was surprised to see that >>> it worked so well. >>> I then replaced pfifo_fast with fq and the performance went right down >>> to only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained >>> the performance at once. >>> >>> I have got some questions to you guys that know a lot more than me about >>> these things: >>> >> 1. Do fq (and fq_codel) even work reliably in a VM? What is the best >>> choice for default qdisc to use in a VM in general? >>> >> >> Eric covered this one. We are not aware of specific issues with fq in VM >> environments. And we have tested that fq works sufficiently well on Google >> Cloud VMs. >> >> >>> 2. Why do BBR immediately "fix" all my issues with upload through that >>> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement? >>> >> >> For BBR, pacing is part of the design in order to make BBR more "gentle" >> in terms of the rate at which it sends, in order to put less pressure on >> buffers and keep packet loss lower. This is particularly important when a >> BBR flow is restarting from idle. In this case BBR starts with a full cwnd, >> and it counts on pacing to pace out the packets at the estimated bandwidth, >> so that the queue can stay relatively short and yet the pipe can be filled >> immediately. >> >> Running BBR without pacing makes BBR more aggressive, particularly in >> restarting from idle, but also in the steady state, where BBR tries to use >> pacing to keep the queue short. >> >> For bulk transfer tests with one flow, running BBR without pacing will >> likely cause higher queues and loss rates at the bottleneck, which may >> negatively impact other traffic sharing that bottleneck. >> >> >>> 3. Could fq_codel on the physical host be the reason that it still works? >>> >> >> Nope, fq_codel does not implement pacing. >> >> >>> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a >>> replacement? >>> >> >> Nope, BBR needs pacing to work correctly, and currently fq is the only >> Linux qdisc that implements pacing. >> >> >>> 5. Is BBR perhaps modified to do the right thing without having to >>> change the qdisc in the current kernel 4.9? >>> >> >> Nope. Linux 4.9 contains the initial public release of BBR from September >> 2016. And there have been no code changes since then (just expanded >> comments). >> >> Thanks for the test report! >> >> neal >> >> > --001a1142dfaeb904de0546f30982 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On W= ed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke <hkbakke@gmail.com>= ; wrote:
Actually.. the 1-4= mbit/s results with fq sporadically appears again as I keep testing but it= is most likely caused by all the unknowns between me an my testserver. But= still, changing to pfifo_qdisc seems to normalize the throughput again wit= h BBR, could this be one of those times where BBR and pacing actually is ge= tting hurt for playing nice in some very variable bottleneck on the way?

Possibly. Would you be able to ta= ke a tcpdump trace of each trial (headers only would be ideal), and post on= a web site somewhere a pcap trace for one of the slow trials?
For example:

=C2=A0 =C2=A0tcpdump -n = -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 &

thanks,
neal

=C2=A0

On= 25 January 2017 at 23:01, Neal Cardwell <ncardwell@google.com><= /span> wrote:
On Wed= , Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkbakke@gmail.com><= /span> wrote:
Hi

Kernel 4.9 finally landed in Debian testing so I could fin= ally test BBR in a real life environment that I have struggled with getting= any kind of performance out of.

The challe= nge at hand is UDP based OpenVPN through europe at around 35 ms rtt to my V= PN-provider with plenty of available bandwith available in both ends and ev= erything completely unknown in between. After tuning the UDP-buffers up to = make room for my 500 mbit/s symmetrical bandwith at 35 ms the download part= seemed to work nicely at an unreliable 150 to 300 mbit/s, while the upload= was stuck at 30 to 60 mbit/s.=C2=A0

Just b= y activating BBR the bandwith instantly shot up to around 150 mbit/s using = a fat tcp test to a public iperf3 server located near my VPN exit point in = the Netherlands. Replace BBR with qubic again and the performance is once a= gain all over the place ranging from very bad to bad, but never better than= 1/3 of BBRs "steady state". In other words "instant WIN!&qu= ot;

Glad to hear it. Tha= nks for the test report!
=C2=A0
However, seeing the requirement of fq and pacing for BBR an= d noticing that I am running pfifo_fast within a VM with virtio NIC on a Pr= oxmox VE host with fq_codel on all physical interfaces, I was surprised to = see that it worked so well.
I then replaced pfifo_fast with fq and the performance went right down = to only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained t= he performance at once.
=
I have got some que= stions to you guys that know a lot more than me about these things:=C2=A0
=
1. Do fq (and fq_codel) even work rel= iably in a VM? What is the best choice for default qdisc to use in a VM in = general?

Eric covered th= is one. We are not aware of specific issues with fq in VM environments. And= =C2=A0we have tested that fq works sufficiently well on Google Cloud VMs.<= /div>
=C2=A0
2. Why do= BBR immediately "fix" all my issues with upload through that &qu= ot;unreliable" big BDP link with pfifo_fast when fq pacing is a requir= ement?

For BBR, pacing i= s part of the design in order to make BBR more "gentle" in terms = of the rate at which it sends, in order to put less pressure on buffers and= keep packet loss lower. This is particularly important when a BBR flow is = restarting from idle. In this case BBR starts with a full cwnd, and it coun= ts on pacing to pace out the packets at the estimated bandwidth, so that th= e queue can stay relatively short and yet the pipe can be filled immediatel= y.

Running BBR without pacing makes BBR more aggre= ssive, particularly in restarting from idle, but also in the steady state, = where BBR tries to use pacing to keep the queue short.

=
For bulk transfer tests with one flow, running BBR without pacing will= likely cause higher queues and loss rates at the bottleneck, which may neg= atively impact other traffic sharing that bottleneck.
=C2= =A0
3. Could fq_codel on the phy= sical host be the reason that it still works?
=
Nope, fq_codel does not implement pacing.
=
=C2=A0
4. Do BBR _only_ wor= k with fq pacing or could fq_codel be used as a replacement?

Nope, BBR needs pacing to work correct= ly, and currently fq is the only Linux qdisc that implements pacing.
<= span>
=C2=A0
5. Is BBR perha= ps modified to do the right thing without having to change the qdisc in the= current kernel 4.9?

Nop= e. Linux 4.9 contains the initial public release of BBR from September 2016= . And there have been no code changes since then (just expanded comments).<= /div>

Thanks for the test report!

neal



--001a1142dfaeb904de0546f30982--