From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8986A3B29E for ; Wed, 25 Jan 2017 18:31:57 -0500 (EST) Received: by mail-oi0-x22c.google.com with SMTP id j15so126779849oih.2 for ; Wed, 25 Jan 2017 15:31:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=2W3dINlsJxM8PN9XWepKZu7XkMl2Fg8YbW2EkV9Js9Q=; b=EtwT4hpAwAwLNPi6fwLQ+cGWAn6sAUVNqC3jr+HrjmcbLf4lKQOUPsE2k24tKbssCf hkjK52esY6yr86StxsCRORQEPdx5Kmfyv5oPQ+KdtN8rIsshbC0EtuuLidRc1q6YNXAP 5gDMwVsIGhQUWLNLXJS1z3271x7SnYa2fNrn12WuA24wJmI+a8F+Xr2zZJpxwEJztHBL enOcE2Q4Ew5k02z3mgoWxXvZSLtEEaVTDGihn/nYexgHWCdleCu5lyGBEtvruEIUXhPc 3UC3uqpQ8xYQDHCFt1w8iEqeUEz4l0B1Pv6Xnay0S0ZFx01hlJ+dq7JAjQlv+PYS6dIl TajA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=2W3dINlsJxM8PN9XWepKZu7XkMl2Fg8YbW2EkV9Js9Q=; b=VeVJ2zGM0SQBQSp9T9i5vVtmLK6ZUGjMETOiyQzOF3WU51PSfHkjc56bKZxUvGC5kT miaFH38g7pqB8UbJDuJvWUNi5Wyre0J5eGMT9WealYtpsN0zjgKvPjS6a+iZ/pUcNN7c P2X80k+JfdWsIEFFhhSi/5tYUOTIXM5wI3yTieNhO5+HiViw86TqOW21FssMI6kN7ARO emaEGxQZygmSxL6NmSAL8D4ljpw0M3ucWQd9z3H09AUWce9PIMLkhTRMAsYNL0a9MM3v oCgOL2von8RSSFF6tnzsecq3RnnNn86wqZFQP0L/rdT5JBAYkOGgd3fZzZpfCp9e8Oqa QXUw== X-Gm-Message-State: AIkVDXItpwao4nDO6l5Eba5LXHqsAIr3asYz1P+y0LZ+ctpDTySemJKwMpI0bMR0CInIuxWAavhLVKRbnMscaQ== X-Received: by 10.202.73.78 with SMTP id w75mr18859345oia.19.1485387116916; Wed, 25 Jan 2017 15:31:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.1.21 with HTTP; Wed, 25 Jan 2017 15:31:56 -0800 (PST) In-Reply-To: References: From: Hans-Kristian Bakke Date: Thu, 26 Jan 2017 00:31:56 +0100 Message-ID: To: Neal Cardwell Cc: bloat Content-Type: multipart/alternative; boundary=001a113df3a47536f30546f3a3c4 Subject: Re: [Bloat] Initial tests with BBR in kernel 4.9 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jan 2017 23:31:57 -0000 --001a113df3a47536f30546f3a3c4 Content-Type: text/plain; charset=UTF-8 Okay, here is some captures. It took a while as I of course had to create a script for it as the public iperf3 server is not always free to do tests. I think there are enough context with the folder names to make sense of it. https://owncloud.proikt.com/index.php/s/eY6eZmjDlznar0N On 26 January 2017 at 00:04, Hans-Kristian Bakke wrote: > I can do that. I guess I should do the capture from tun1 as that is the > place that the tcp-traffic is visible? My non-virtual nic is only seeing > OpenVPN encapsulated UDP-traffic. > > On 25 January 2017 at 23:48, Neal Cardwell wrote: > >> On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke >> wrote: >> >>> Actually.. the 1-4 mbit/s results with fq sporadically appears again as >>> I keep testing but it is most likely caused by all the unknowns between me >>> an my testserver. But still, changing to pfifo_qdisc seems to normalize the >>> throughput again with BBR, could this be one of those times where BBR and >>> pacing actually is getting hurt for playing nice in some very variable >>> bottleneck on the way? >>> >> >> Possibly. Would you be able to take a tcpdump trace of each trial >> (headers only would be ideal), and post on a web site somewhere a pcap >> trace for one of the slow trials? >> >> For example: >> >> tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 & >> >> thanks, >> neal >> >> >> >>> >>> On 25 January 2017 at 23:01, Neal Cardwell wrote: >>> >>>> On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke >>> > wrote: >>>> >>>>> Hi >>>>> >>>>> Kernel 4.9 finally landed in Debian testing so I could finally test >>>>> BBR in a real life environment that I have struggled with getting any kind >>>>> of performance out of. >>>>> >>>>> The challenge at hand is UDP based OpenVPN through europe at around 35 >>>>> ms rtt to my VPN-provider with plenty of available bandwith available in >>>>> both ends and everything completely unknown in between. After tuning the >>>>> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms >>>>> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s, >>>>> while the upload was stuck at 30 to 60 mbit/s. >>>>> >>>>> Just by activating BBR the bandwith instantly shot up to around 150 >>>>> mbit/s using a fat tcp test to a public iperf3 server located near my VPN >>>>> exit point in the Netherlands. Replace BBR with qubic again and the >>>>> performance is once again all over the place ranging from very bad to bad, >>>>> but never better than 1/3 of BBRs "steady state". In other words "instant >>>>> WIN!" >>>>> >>>> >>>> Glad to hear it. Thanks for the test report! >>>> >>>> >>>>> However, seeing the requirement of fq and pacing for BBR and noticing >>>>> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE >>>>> host with fq_codel on all physical interfaces, I was surprised to see that >>>>> it worked so well. >>>>> I then replaced pfifo_fast with fq and the performance went right down >>>>> to only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained >>>>> the performance at once. >>>>> >>>>> I have got some questions to you guys that know a lot more than me >>>>> about these things: >>>>> >>>> 1. Do fq (and fq_codel) even work reliably in a VM? What is the best >>>>> choice for default qdisc to use in a VM in general? >>>>> >>>> >>>> Eric covered this one. We are not aware of specific issues with fq in >>>> VM environments. And we have tested that fq works sufficiently well on >>>> Google Cloud VMs. >>>> >>>> >>>>> 2. Why do BBR immediately "fix" all my issues with upload through that >>>>> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement? >>>>> >>>> >>>> For BBR, pacing is part of the design in order to make BBR more >>>> "gentle" in terms of the rate at which it sends, in order to put less >>>> pressure on buffers and keep packet loss lower. This is particularly >>>> important when a BBR flow is restarting from idle. In this case BBR starts >>>> with a full cwnd, and it counts on pacing to pace out the packets at the >>>> estimated bandwidth, so that the queue can stay relatively short and yet >>>> the pipe can be filled immediately. >>>> >>>> Running BBR without pacing makes BBR more aggressive, particularly in >>>> restarting from idle, but also in the steady state, where BBR tries to use >>>> pacing to keep the queue short. >>>> >>>> For bulk transfer tests with one flow, running BBR without pacing will >>>> likely cause higher queues and loss rates at the bottleneck, which may >>>> negatively impact other traffic sharing that bottleneck. >>>> >>>> >>>>> 3. Could fq_codel on the physical host be the reason that it still >>>>> works? >>>>> >>>> >>>> Nope, fq_codel does not implement pacing. >>>> >>>> >>>>> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a >>>>> replacement? >>>>> >>>> >>>> Nope, BBR needs pacing to work correctly, and currently fq is the only >>>> Linux qdisc that implements pacing. >>>> >>>> >>>>> 5. Is BBR perhaps modified to do the right thing without having to >>>>> change the qdisc in the current kernel 4.9? >>>>> >>>> >>>> Nope. Linux 4.9 contains the initial public release of BBR from >>>> September 2016. And there have been no code changes since then (just >>>> expanded comments). >>>> >>>> Thanks for the test report! >>>> >>>> neal >>>> >>>> >>> >> > --001a113df3a47536f30546f3a3c4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Okay, here is some captures. It took a while as I of course had= to create a script for it as the public iperf3 server is not always free t= o do tests.=C2=A0
I think there are enough context with the folder names to= make sense of it.

https://owncloud.proikt.com/index.php/s/eY6eZmjDlznar0N

On 26 January 2017 at 00:04, Hans-Kristian Bakke <hkbakke@gmail.com&g= t; wrote:
I can do tha= t. I guess I should do the capture from tun1 as that is the place that the = tcp-traffic is visible? My non-virtual nic is only seeing OpenVPN encapsula= ted UDP-traffic.

On 25 January 2017 at 23= :48, Neal Cardwell <ncardwell@google.com> wrote:
On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian = Bakke <hkbakke@gmail.com> wrote:
Actually.. the 1-4 mbit/s results with fq sporadically appear= s again as I keep testing but it is most likely caused by all the unknowns = between me an my testserver. But still, changing to pfifo_qdisc seems to no= rmalize the throughput again with BBR, could this be one of those times whe= re BBR and pacing actually is getting hurt for playing nice in some very va= riable bottleneck on the way?

Possibly. Would you be able to take a tcpdump trace of each trial (he= aders only would be ideal), and post on a web site somewhere a pcap trace f= or one of the slow trials?

For example:
=
=C2=A0 =C2=A0tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1= 000000 &

thanks,
neal

=C2=A0

On 25 January 2017 at= 23:01, Neal Cardwell <ncardwell@google.com> wrote:
On Wed, Jan 25, 2017 at 3= :54 PM, Hans-Kristian Bakke <hkbakke@gmail.com> wrote:
Hi

Kern= el 4.9 finally landed in Debian testing so I could finally test BBR in a re= al life environment that I have struggled with getting any kind of performa= nce out of.

The challenge at hand is UDP ba= sed OpenVPN through europe at around 35 ms rtt to my VPN-provider with plen= ty of available bandwith available in both ends and everything completely u= nknown in between. After tuning the UDP-buffers up to make room for my 500 = mbit/s symmetrical bandwith at 35 ms the download part seemed to work nicel= y at an unreliable 150 to 300 mbit/s, while the upload was stuck at 30 to 6= 0 mbit/s.=C2=A0

Just by activating BBR the = bandwith instantly shot up to around 150 mbit/s using a fat tcp test to a p= ublic iperf3 server located near my VPN exit point in the Netherlands. Repl= ace BBR with qubic again and the performance is once again all over the pla= ce ranging from very bad to bad, but never better than 1/3 of BBRs "st= eady state". In other words "instant WIN!"

Glad to hear it. Thanks for the test repo= rt!
=C2=A0
Howev= er, seeing the requirement of fq and pacing for BBR and noticing that I am = running pfifo_fast within a VM with virtio NIC on a Proxmox VE host with fq= _codel on all physical interfaces, I was surprised to see that it worked so= well.
I then replaced p= fifo_fast with fq and the performance went right down to only 1-4 mbit/s fr= om around 150 mbit/s. Removing the fq again regained the performance at onc= e.

I have got some questions to you guys t= hat know a lot more than me about these things:=C2=A0
1. Do fq (and fq_codel) even work reliably in a VM? What = is the best choice for default qdisc to use in a VM in general?
=

Eric covered this one. We are not a= ware of specific issues with fq in VM environments. And =C2=A0we have teste= d that fq works sufficiently well on Google Cloud VMs.
=C2= =A0
2. Why do BBR immediately &q= uot;fix" all my issues with upload through that "unreliable"= big BDP link with pfifo_fast when fq pacing is a requirement?
<= /blockquote>

For BBR, pacing is part of the desig= n in order to make BBR more "gentle" in terms of the rate at whic= h it sends, in order to put less pressure on buffers and keep packet loss l= ower. This is particularly important when a BBR flow is restarting from idl= e. In this case BBR starts with a full cwnd, and it counts on pacing to pac= e out the packets at the estimated bandwidth, so that the queue can stay re= latively short and yet the pipe can be filled immediately.

Running BBR without pacing makes BBR more aggressive, particularly= in restarting from idle, but also in the steady state, where BBR tries to = use pacing to keep the queue short.

For bulk trans= fer tests with one flow, running BBR without pacing will likely cause highe= r queues and loss rates at the bottleneck, which may negatively impact othe= r traffic sharing that bottleneck.
=C2=A0
3. Could fq_codel on the physical host be the rea= son that it still works?

Nope, fq_codel does not implement pacing.
=C2=A0
4. Do BBR _only_ work with fq pacing or= could fq_codel be used as a replacement?

=
Nope, BBR needs pacing to work correctly, and currently f= q is the only Linux qdisc that implements pacing.
=C2=A0
5. Is BBR perhaps modified to do t= he right thing without having to change the qdisc in the current kernel 4.9= ?

Nope. Linux 4.9 contai= ns the initial public release of BBR from September 2016. And there have be= en no code changes since then (just expanded comments).

Thanks for the test report!

neal





--001a113df3a47536f30546f3a3c4--