On 03/12/16 19:13, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote:

>>> I have one thing that I _wonder_ if could be BBR's fault: I run
>>> backup over SSH. (That would be tar + gzip + ssh.) The first full
>>> backup after I rolled out BBR on the server (the one sending the
>>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of
>>> free I/O, and neither tar nor gzip (well, pigz) used a full core.
>>> My only remaining explanation would be that somehow, BBR didn't
>>> deal well with the irregular stream of data coming from tar. (A
>>> wget between the same machines at the same time gave 6-700
>>> Mbit/sec.)

>> Thanks for the report, Steinar. This is the first report we've had
>>  like this, but it would be interesting to find out what's going
>> on.
>>
>> Even if you don't have time to apply the patches Eric mentions, it
>>  would be hugely useful if the next time you have a slow transfer
>> like that you could post a link to a tcpdump packet capture
>> (headers only is best, say -s 120). Ideally the trace would
>> capture a whole connection, so we can see the wscale on the SYN
>> exchange.
>
> I tried reproducing it now. I can't get as far down as 50 Mbit/sec,
> but it stopped around 100 Mbit/sec, still without any clear
> bottlenecks. cubic was just as bad, though.
>
> I've taken two tcpdumps as requested; I can't reboot this server
> easily right now, unfortunately. They are:
>
> http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg
> http://storage.sesse.net/bbr2.pcap -- wget between same hosts
>
> /* Steinar */

Since no-one's explicitly mentioned this: be aware that SSH is known for doing application-level windowing, limiting performance.

E.g. see https://www.psc.edu/index.php/hpn-ssh/638