[Bloat] Initial tests with BBR in kernel 4.9

[Hans-Kristian Bakke]

I can do that. I guess I should do the capture from tun1 as that is the
place that the tcp-traffic is visible? My non-virtual nic is only seeing
OpenVPN encapsulated UDP-traffic.

On 25 January 2017 at 23:48, Neal Cardwell <ncardwell at google.com> wrote:

> On Wed, Jan 25, 2017 at 5:38 PM, Hans-Kristian Bakke <hkbakke at gmail.com>
> wrote:
>
>> Actually.. the 1-4 mbit/s results with fq sporadically appears again as I
>> keep testing but it is most likely caused by all the unknowns between me an
>> my testserver. But still, changing to pfifo_qdisc seems to normalize the
>> throughput again with BBR, could this be one of those times where BBR and
>> pacing actually is getting hurt for playing nice in some very variable
>> bottleneck on the way?
>>
>
> Possibly. Would you be able to take a tcpdump trace of each trial (headers
> only would be ideal), and post on a web site somewhere a pcap trace for one
> of the slow trials?
>
> For example:
>
>    tcpdump -n -w /tmp/out.pcap -s 120 -i eth0 -c 1000000 &
>
> thanks,
> neal
>
>
>
>>
>> On 25 January 2017 at 23:01, Neal Cardwell <ncardwell at google.com> wrote:
>>
>>> On Wed, Jan 25, 2017 at 3:54 PM, Hans-Kristian Bakke <hkbakke at gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Kernel 4.9 finally landed in Debian testing so I could finally test BBR
>>>> in a real life environment that I have struggled with getting any kind of
>>>> performance out of.
>>>>
>>>> The challenge at hand is UDP based OpenVPN through europe at around 35
>>>> ms rtt to my VPN-provider with plenty of available bandwith available in
>>>> both ends and everything completely unknown in between. After tuning the
>>>> UDP-buffers up to make room for my 500 mbit/s symmetrical bandwith at 35 ms
>>>> the download part seemed to work nicely at an unreliable 150 to 300 mbit/s,
>>>> while the upload was stuck at 30 to 60 mbit/s.
>>>>
>>>> Just by activating BBR the bandwith instantly shot up to around 150
>>>> mbit/s using a fat tcp test to a public iperf3 server located near my VPN
>>>> exit point in the Netherlands. Replace BBR with qubic again and the
>>>> performance is once again all over the place ranging from very bad to bad,
>>>> but never better than 1/3 of BBRs "steady state". In other words "instant
>>>> WIN!"
>>>>
>>>
>>> Glad to hear it. Thanks for the test report!
>>>
>>>
>>>> However, seeing the requirement of fq and pacing for BBR and noticing
>>>> that I am running pfifo_fast within a VM with virtio NIC on a Proxmox VE
>>>> host with fq_codel on all physical interfaces, I was surprised to see that
>>>> it worked so well.
>>>> I then replaced pfifo_fast with fq and the performance went right down
>>>> to only 1-4 mbit/s from around 150 mbit/s. Removing the fq again regained
>>>> the performance at once.
>>>>
>>>> I have got some questions to you guys that know a lot more than me
>>>> about these things:
>>>>
>>> 1. Do fq (and fq_codel) even work reliably in a VM? What is the best
>>>> choice for default qdisc to use in a VM in general?
>>>>
>>>
>>> Eric covered this one. We are not aware of specific issues with fq in VM
>>> environments. And  we have tested that fq works sufficiently well on Google
>>> Cloud VMs.
>>>
>>>
>>>> 2. Why do BBR immediately "fix" all my issues with upload through that
>>>> "unreliable" big BDP link with pfifo_fast when fq pacing is a requirement?
>>>>
>>>
>>> For BBR, pacing is part of the design in order to make BBR more "gentle"
>>> in terms of the rate at which it sends, in order to put less pressure on
>>> buffers and keep packet loss lower. This is particularly important when a
>>> BBR flow is restarting from idle. In this case BBR starts with a full cwnd,
>>> and it counts on pacing to pace out the packets at the estimated bandwidth,
>>> so that the queue can stay relatively short and yet the pipe can be filled
>>> immediately.
>>>
>>> Running BBR without pacing makes BBR more aggressive, particularly in
>>> restarting from idle, but also in the steady state, where BBR tries to use
>>> pacing to keep the queue short.
>>>
>>> For bulk transfer tests with one flow, running BBR without pacing will
>>> likely cause higher queues and loss rates at the bottleneck, which may
>>> negatively impact other traffic sharing that bottleneck.
>>>
>>>
>>>> 3. Could fq_codel on the physical host be the reason that it still
>>>> works?
>>>>
>>>
>>> Nope, fq_codel does not implement pacing.
>>>
>>>
>>>> 4. Do BBR _only_ work with fq pacing or could fq_codel be used as a
>>>> replacement?
>>>>
>>>
>>> Nope, BBR needs pacing to work correctly, and currently fq is the only
>>> Linux qdisc that implements pacing.
>>>
>>>
>>>> 5. Is BBR perhaps modified to do the right thing without having to
>>>> change the qdisc in the current kernel 4.9?
>>>>
>>>
>>> Nope. Linux 4.9 contains the initial public release of BBR from
>>> September 2016. And there have been no code changes since then (just
>>> expanded comments).
>>>
>>> Thanks for the test report!
>>>
>>> neal
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20170126/5dd1dd1e/attachment-0002.html>