[Bloat] Initial tests with BBR in kernel 4.9

[Hans-Kristian Bakke <hkbakke@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

Hi

Kernel 4.9 finally landed in Debian testing so I could finally test BBR in
a real life environment that I have struggled with getting any kind of
performance out of.

The challenge at hand is UDP based OpenVPN through europe at around 35 ms
rtt to my VPN-provider with plenty of available bandwith available in both
ends and everything completely unknown in between. After tuning the
UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms
the download part seemed to work nicely at an unreliable 150 to 300 mbit/s,
while the upload was stuck at 30 to 60 mbit/s.

Just by activating BBR the bandwith instantly shot up to around 150 mbit/s
using a fat tcp test to a public iperf3 server located near my VPN exit
point in the Netherlands. Replace BBR with qubic again and the performance
is once again all over the place ranging from very bad to bad, but never
better than 1/3 of BBRs "steady state". In other words "instant WIN!"

However, seeing the requirement of fq and pacing for BBR and noticing that
I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE host
with fq_codel on all physical interfaces, I was surprised to see that it
worked so well.
I then replaced pfifo_fast with fq and the performance went right down to
only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained the
performance at once.

I have got some questions to you guys that know a lot more than me about
these things:
1. Do fq (and fq_codel) even work reliably in a VM? What is the best choice
for default qdisc to use in a VM in general?
2. Why do BBR immediately "fix" all my issues with upload through that
"unreliable" big BDP link with pfifo_fast when fq pacing is a requirement?
3. Could fq_codel on the physical host be the reason that it still works?
4. Do BBR _only_ work with fq pacing or could fq_codel be used as a
replacement?
5. Is BBR perhaps modified to do the right thing without having to change
the qdisc in the current kernel 4.9?

Sorry for long post, but this is an interesting topic!

Regards,
Hans-Kristian Bakke

[-- Attachment #2: Type: text/html, Size: 3723 bytes --]