[Bloat] Initial tests with BBR in kernel 4.9

[Hans-Kristian Bakke]

Actually.. the 1-4 mbit/s results with fq sporadically appears again as I
keep testing but it is most likely caused by all the unknowns between me an
my testserver. But still, changing to pfifo_qdisc seems to normalize the
throughput again with BBR, could this be one of those times where BBR and
pacing actually is getting hurt for playing nice in some very variable
bottleneck on the way?

On 25 January 2017 at 23:01, Neal Cardwell <ncardwell at google.com> wrote:

> On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkbakke at gmail.com>
> wrote:
>
>> Hi
>>
>> Kernel 4.9 finally landed in Debian testing so I could finally test BBR
>> in a real life environment that I have struggled with getting any kind of
>> performance out of.
>>
>> The challenge at hand is UDP based OpenVPN through europe at around 35 ms
>> rtt to my VPN-provider with plenty of available bandwith available in both
>> ends and everything completely unknown in between. After tuning the
>> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms
>> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s,
>> while the upload was stuck at 30 to 60 mbit/s.
>>
>> Just by activating BBR the bandwith instantly shot up to around 150
>> mbit/s using a fat tcp test to a public iperf3 server located near my VPN
>> exit point in the Netherlands. Replace BBR with qubic again and the
>> performance is once again all over the place ranging from very bad to bad,
>> but never better than 1/3 of BBRs "steady state". In other words "instant
>> WIN!"
>>
>
> Glad to hear it. Thanks for the test report!
>
>
>> However, seeing the requirement of fq and pacing for BBR and noticing
>> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE
>> host with fq_codel on all physical interfaces, I was surprised to see that
>> it worked so well.
>> I then replaced pfifo_fast with fq and the performance went right down to
>> only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained the
>> performance at once.
>>
>> I have got some questions to you guys that know a lot more than me about
>> these things:
>>
> 1. Do fq (and fq_codel) even work reliably in a VM? What is the best
>> choice for default qdisc to use in a VM in general?
>>
>
> Eric covered this one. We are not aware of specific issues with fq in VM
> environments. And  we have tested that fq works sufficiently well on Google
> Cloud VMs.
>
>
>> 2. Why do BBR immediately "fix" all my issues with upload through that
>> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement?
>>
>
> For BBR, pacing is part of the design in order to make BBR more "gentle"
> in terms of the rate at which it sends, in order to put less pressure on
> buffers and keep packet loss lower. This is particularly important when a
> BBR flow is restarting from idle. In this case BBR starts with a full cwnd,
> and it counts on pacing to pace out the packets at the estimated bandwidth,
> so that the queue can stay relatively short and yet the pipe can be filled
> immediately.
>
> Running BBR without pacing makes BBR more aggressive, particularly in
> restarting from idle, but also in the steady state, where BBR tries to use
> pacing to keep the queue short.
>
> For bulk transfer tests with one flow, running BBR without pacing will
> likely cause higher queues and loss rates at the bottleneck, which may
> negatively impact other traffic sharing that bottleneck.
>
>
>> 3. Could fq_codel on the physical host be the reason that it still works?
>>
>
> Nope, fq_codel does not implement pacing.
>
>
>> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a
>> replacement?
>>
>
> Nope, BBR needs pacing to work correctly, and currently fq is the only
> Linux qdisc that implements pacing.
>
>
>> 5. Is BBR perhaps modified to do the right thing without having to change
>> the qdisc in the current kernel 4.9?
>>
>
> Nope. Linux 4.9 contains the initial public release of BBR from September
> 2016. And there have been no code changes since then (just expanded
> comments).
>
> Thanks for the test report!
>
> neal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20170125/52e75357/attachment-0002.html>