On 03/12/16 19:13, Steinar H. Gunderson wrote:
 > On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote:

 >>> I have one thing that I _wonder_ if could be BBR's fault: I run
 >>> backup over SSH. (That would be tar + gzip + ssh.) The first full
 >>> backup after I rolled out BBR on the server (the one sending the
 >>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of
 >>> free I/O, and neither tar nor gzip (well, pigz) used a full core.
 >>> My only remaining explanation would be that somehow, BBR didn't
 >>> deal well with the irregular stream of data coming from tar. (A
 >>> wget between the same machines at the same time gave 6-700
 >>> Mbit/sec.)

 >> Thanks for the report, Steinar. This is the first report we've had
 >>  like this, but it would be interesting to find out what's going
 >> on.
 >>
 >> Even if you don't have time to apply the patches Eric mentions, it
 >>  would be hugely useful if the next time you have a slow transfer
 >> like that you could post a link to a tcpdump packet capture
 >> (headers only is best, say -s 120). Ideally the trace would
 >> capture a whole connection, so we can see the wscale on the SYN
 >> exchange.
 >
 > I tried reproducing it now. I can't get as far down as 50 Mbit/sec,
 > but it stopped around 100 Mbit/sec, still without any clear
 > bottlenecks. cubic was just as bad, though.
 >
 > I've taken two tcpdumps as requested; I can't reboot this server
 > easily right now, unfortunately. They are:
 >
 > http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg
 > http://storage.sesse.net/bbr2.pcap -- wget between same hosts
 >
 > /* Steinar */

Since no-one's explicitly mentioned this: be aware that SSH is known for 
doing application-level windowing, limiting performance.

E.g. see https://www.psc.edu/index.php/hpn-ssh/638