Re: [Bloat] Initial tests with BBR in kernel 4.9

[Neal Cardwell <ncardwell@google.com>]

[-- Attachment #1: Type: text/plain, Size: 3540 bytes --]

On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkbakke@gmail.com>
wrote:

> Hi
>
> Kernel 4.9 finally landed in Debian testing so I could finally test BBR in
> a real life environment that I have struggled with getting any kind of
> performance out of.
>
> The challenge at hand is UDP based OpenVPN through europe at around 35 ms
> rtt to my VPN-provider with plenty of available bandwith available in both
> ends and everything completely unknown in between. After tuning the
> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms
> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s,
> while the upload was stuck at 30 to 60 mbit/s.
>
> Just by activating BBR the bandwith instantly shot up to around 150 mbit/s
> using a fat tcp test to a public iperf3 server located near my VPN exit
> point in the Netherlands. Replace BBR with qubic again and the performance
> is once again all over the place ranging from very bad to bad, but never
> better than 1/3 of BBRs "steady state". In other words "instant WIN!"
>

Glad to hear it. Thanks for the test report!


> However, seeing the requirement of fq and pacing for BBR and noticing that
> I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE host
> with fq_codel on all physical interfaces, I was surprised to see that it
> worked so well.
> I then replaced pfifo_fast with fq and the performance went right down to
> only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained the
> performance at once.
>
> I have got some questions to you guys that know a lot more than me about
> these things:
>
1. Do fq (and fq_codel) even work reliably in a VM? What is the best choice
> for default qdisc to use in a VM in general?
>

Eric covered this one. We are not aware of specific issues with fq in VM
environments. And  we have tested that fq works sufficiently well on Google
Cloud VMs.


> 2. Why do BBR immediately "fix" all my issues with upload through that
> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement?
>

For BBR, pacing is part of the design in order to make BBR more "gentle" in
terms of the rate at which it sends, in order to put less pressure on
buffers and keep packet loss lower. This is particularly important when a
BBR flow is restarting from idle. In this case BBR starts with a full cwnd,
and it counts on pacing to pace out the packets at the estimated bandwidth,
so that the queue can stay relatively short and yet the pipe can be filled
immediately.

Running BBR without pacing makes BBR more aggressive, particularly in
restarting from idle, but also in the steady state, where BBR tries to use
pacing to keep the queue short.

For bulk transfer tests with one flow, running BBR without pacing will
likely cause higher queues and loss rates at the bottleneck, which may
negatively impact other traffic sharing that bottleneck.


> 3. Could fq_codel on the physical host be the reason that it still works?
>

Nope, fq_codel does not implement pacing.


> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a
> replacement?
>

Nope, BBR needs pacing to work correctly, and currently fq is the only
Linux qdisc that implements pacing.


> 5. Is BBR perhaps modified to do the right thing without having to change
> the qdisc in the current kernel 4.9?
>

Nope. Linux 4.9 contains the initial public release of BBR from September
2016. And there have been no code changes since then (just expanded
comments).

Thanks for the test report!

neal

[-- Attachment #2: Type: text/html, Size: 5796 bytes --]