On 03/12/16 19:13, Steinar H. Gunderson wrote: > On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote: >>> I have one thing that I _wonder_ if could be BBR's fault: I run >>> backup over SSH. (That would be tar + gzip + ssh.) The first full >>> backup after I rolled out BBR on the server (the one sending the >>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of >>> free I/O, and neither tar nor gzip (well, pigz) used a full core. >>> My only remaining explanation would be that somehow, BBR didn't >>> deal well with the irregular stream of data coming from tar. (A >>> wget between the same machines at the same time gave 6-700 >>> Mbit/sec.) >> Thanks for the report, Steinar. This is the first report we've had >> like this, but it would be interesting to find out what's going >> on. >> >> Even if you don't have time to apply the patches Eric mentions, it >> would be hugely useful if the next time you have a slow transfer >> like that you could post a link to a tcpdump packet capture >> (headers only is best, say -s 120). Ideally the trace would >> capture a whole connection, so we can see the wscale on the SYN >> exchange. > > I tried reproducing it now. I can't get as far down as 50 Mbit/sec, > but it stopped around 100 Mbit/sec, still without any clear > bottlenecks. cubic was just as bad, though. > > I've taken two tcpdumps as requested; I can't reboot this server > easily right now, unfortunately. They are: > > http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg > http://storage.sesse.net/bbr2.pcap -- wget between same hosts > > /* Steinar */ Since no-one's explicitly mentioned this: be aware that SSH is known for doing application-level windowing, limiting performance. E.g. see https://www.psc.edu/index.php/hpn-ssh/638