From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-x241.google.com (mail-ot0-x241.google.com [IPv6:2607:f8b0:4003:c0f::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 96E1D3B29E for ; Wed, 25 Jan 2017 17:38:08 -0500 (EST) Received: by mail-ot0-x241.google.com with SMTP id 36so25308579otx.3 for ; Wed, 25 Jan 2017 14:38:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=b4gGLNV1OjUdcqYfjojD3Mu1BsSODiV7zkcsPgNmPvc=; b=OReMDAiTfd37MIhXJLdn6KHG1Cgr8+gMYbtEMqU8QFb0h5s4UFJMhAQPEY20mLR+4z oEk7UltQsyVXQhy/9SorkXhNowyYenlOP6oVIHJVPIPeBdZsjTxF4CLiQnV5HPGRyRLa UlQt/zFWApNktcKDyEyaeDGjPQn4NxMggz0GMA3ejEQUrJfN/AFES/MnS5WeubmB7MTn VrebXg0oZSqU0RxJoOZtNcWkQ7cNF5B+MkDX0xsJHkIt9JESIxYnEN+TvnlwY1ZqxN4s 0m8dn3DKjCqKoFwoBWMB1AgDp4q1bCaybgYxGy4kfr/05afp3oLM6HKR6LKjmDUHu9KG J3Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=b4gGLNV1OjUdcqYfjojD3Mu1BsSODiV7zkcsPgNmPvc=; b=i6Xdf3cYev5rMvIFrETs0VjI+b9Q0BHo2zJ9m93axnSXQuJjDryMUmS26C4njD2D2M UGrscQky3M7yVWX2WtHHFnNxNW7nUHfeehEgzm7MtzRj0G2TE+ShnALyKfGLjZUu8pBs jRJ5aLGVaD+GYMB6imqveANf5x1+3zodlSD2h1Kuyb4UAor1ULyhvXMstx/f/mqX9+9+ +B97xq/uXGUXq17ZsueBpEqliiXlaAbnERuvwAdv5OGNraTsAeel2BfGnAdTEwgUpkrv 91E7kIEoZF8i7mhBzsqROziYbYmJ9KNVa8AB+4fWe1NlhkohsF5KcRhXGMpK9pZT2Pp/ On6A== X-Gm-Message-State: AIkVDXKMs6CJPE62azRklbL7LOXhXA3aILXgyUDYBV4Nm0EVHuOiN0N5BBO7Zd9vWgWrnvagBWpKQr3NN/F6Yg== X-Received: by 10.157.4.1 with SMTP id 1mr18138218otc.205.1485383888013; Wed, 25 Jan 2017 14:38:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.1.21 with HTTP; Wed, 25 Jan 2017 14:38:07 -0800 (PST) In-Reply-To: References: From: Hans-Kristian Bakke Date: Wed, 25 Jan 2017 23:38:07 +0100 Message-ID: To: Neal Cardwell Cc: bloat Content-Type: multipart/alternative; boundary=001a113b136c0013270546f2e3e7 Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 22:38:08 -0000 --001a113b136c0013270546f2e3e7 Content-Type: text/plain; charset=UTF-8 Actually.. the 1-4 mbit/s results with fq sporadically appears again as I keep testing but it is most likely caused by all the unknowns between me an my testserver. But still, changing to pfifo_qdisc seems to normalize the throughput again with BBR, could this be one of those times where BBR and pacing actually is getting hurt for playing nice in some very variable bottleneck on the way? On 25 January 2017 at 23:01, Neal Cardwell wrote: > On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke > wrote: > >> Hi >> >> Kernel 4.9 finally landed in Debian testing so I could finally test BBR >> in a real life environment that I have struggled with getting any kind of >> performance out of. >> >> The challenge at hand is UDP based OpenVPN through europe at around 35 ms >> rtt to my VPN-provider with plenty of available bandwith available in both >> ends and everything completely unknown in between. After tuning the >> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms >> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s, >> while the upload was stuck at 30 to 60 mbit/s. >> >> Just by activating BBR the bandwith instantly shot up to around 150 >> mbit/s using a fat tcp test to a public iperf3 server located near my VPN >> exit point in the Netherlands. Replace BBR with qubic again and the >> performance is once again all over the place ranging from very bad to bad, >> but never better than 1/3 of BBRs "steady state". In other words "instant >> WIN!" >> > > Glad to hear it. Thanks for the test report! > > >> However, seeing the requirement of fq and pacing for BBR and noticing >> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE >> host with fq_codel on all physical interfaces, I was surprised to see that >> it worked so well. >> I then replaced pfifo_fast with fq and the performance went right down to >> only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained the >> performance at once. >> >> I have got some questions to you guys that know a lot more than me about >> these things: >> > 1. Do fq (and fq_codel) even work reliably in a VM? What is the best >> choice for default qdisc to use in a VM in general? >> > > Eric covered this one. We are not aware of specific issues with fq in VM > environments. And we have tested that fq works sufficiently well on Google > Cloud VMs. > > >> 2. Why do BBR immediately "fix" all my issues with upload through that >> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement? >> > > For BBR, pacing is part of the design in order to make BBR more "gentle" > in terms of the rate at which it sends, in order to put less pressure on > buffers and keep packet loss lower. This is particularly important when a > BBR flow is restarting from idle. In this case BBR starts with a full cwnd, > and it counts on pacing to pace out the packets at the estimated bandwidth, > so that the queue can stay relatively short and yet the pipe can be filled > immediately. > > Running BBR without pacing makes BBR more aggressive, particularly in > restarting from idle, but also in the steady state, where BBR tries to use > pacing to keep the queue short. > > For bulk transfer tests with one flow, running BBR without pacing will > likely cause higher queues and loss rates at the bottleneck, which may > negatively impact other traffic sharing that bottleneck. > > >> 3. Could fq_codel on the physical host be the reason that it still works? >> > > Nope, fq_codel does not implement pacing. > > >> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a >> replacement? >> > > Nope, BBR needs pacing to work correctly, and currently fq is the only > Linux qdisc that implements pacing. > > >> 5. Is BBR perhaps modified to do the right thing without having to change >> the qdisc in the current kernel 4.9? >> > > Nope. Linux 4.9 contains the initial public release of BBR from September > 2016. And there have been no code changes since then (just expanded > comments). > > Thanks for the test report! > > neal > > --001a113b136c0013270546f2e3e7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Actually.. the 1-4 mbit/s results with fq sporadically appears = again as I keep testing but it is most likely caused by all the unknowns be= tween me an my testserver. But still, changing to pfifo_qdisc seems to norm= alize the throughput again with BBR, could this be one of those times where= BBR and pacing actually is getting hurt for playing nice in some very vari= able bottleneck on the way?

On 25 January 2017 at 23:01, Neal Cardwell <ncar= dwell@google.com> wrote:
On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkbakke= @gmail.com> wrote:
Hi

Kernel 4.9 finally landed in Debian testing so I could fin= ally test BBR in a real life environment that I have struggled with getting= any kind of performance out of.

The challe= nge at hand is UDP based OpenVPN through europe at around 35 ms rtt to my V= PN-provider with plenty of available bandwith available in both ends and ev= erything completely unknown in between. After tuning the UDP-buffers up to = make room for my 500 mbit/s symmetrical bandwith at 35 ms the download part= seemed to work nicely at an unreliable 150 to 300 mbit/s, while the upload= was stuck at 30 to 60 mbit/s.=C2=A0

Just b= y activating BBR the bandwith instantly shot up to around 150 mbit/s using = a fat tcp test to a public iperf3 server located near my VPN exit point in = the Netherlands. Replace BBR with qubic again and the performance is once a= gain all over the place ranging from very bad to bad, but never better than= 1/3 of BBRs "steady state". In other words "instant WIN!&qu= ot;

Glad to hear it. Tha= nks for the test report!
=C2=A0
However, seeing the requirement of fq and pacing for BBR and notici= ng that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE= host with fq_codel on all physical interfaces, I was surprised to see that= it worked so well.
I th= en replaced pfifo_fast with fq and the performance went right down to only = 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained the perfo= rmance at once.

I have got some questions t= o you guys that know a lot more than me about these things:=C2=A0
1. Do fq (and fq_codel) even work reliably in a VM? What is the = best choice for default qdisc to use in a VM in general?

Eric covered this one. We are not aware of= specific issues with fq in VM environments. And =C2=A0we have tested that = fq works sufficiently well on Google Cloud VMs.
= =C2=A0
2. Why do BBR immediately "fix" al= l my issues with upload through that "unreliable" big BDP link wi= th pfifo_fast when fq pacing is a requirement?

For BBR, pacing is part of the design in order to ma= ke BBR more "gentle" in terms of the rate at which it sends, in o= rder to put less pressure on buffers and keep packet loss lower. This is pa= rticularly important when a BBR flow is restarting from idle. In this case = BBR starts with a full cwnd, and it counts on pacing to pace out the packet= s at the estimated bandwidth, so that the queue can stay relatively short a= nd yet the pipe can be filled immediately.

Running= BBR without pacing makes BBR more aggressive, particularly in restarting f= rom idle, but also in the steady state, where BBR tries to use pacing to ke= ep the queue short.

For bulk transfer tests with o= ne flow, running BBR without pacing will likely cause higher queues and los= s rates at the bottleneck, which may negatively impact other traffic sharin= g that bottleneck.
=C2=A0
3. Could fq_codel on the physical host be the reason that it still works?=

Nope, fq_codel does not= implement pacing.
=C2=A0
4. Do BBR _only_ work with fq pacing or could fq_codel be used as a repla= cement?

Nope, BBR needs = pacing to work correctly, and currently fq is the only Linux qdisc that imp= lements pacing.
=C2=A0
5= . Is BBR perhaps modified to do the right thing without having to change th= e qdisc in the current kernel 4.9?

<= /span>
Nope. Linux 4.9 contains the initial public release of BBR from = September 2016. And there have been no code changes since then (just expand= ed comments).

Thanks for the test report!

neal
<= div>

--001a113b136c0013270546f2e3e7--