From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x244.google.com (mail-wm0-x244.google.com [IPv6:2a00:1450:400c:c09::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 043803B260 for ; Wed, 7 Dec 2016 11:28:17 -0500 (EST) Received: by mail-wm0-x244.google.com with SMTP id u144so28778465wmu.0 for ; Wed, 07 Dec 2016 08:28:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=Vfrai/bcjoInkAH4oWz3L3IRd0JDk7/H+al2UsAGAXI=; b=SVlqJaAqVNa6t6NjBZr7dJpi69hROiSplEYNl/WPHwtN1Mw+c5f+i9/9gjhbcJBiog wGkZI5VNloshKVf1tF72xAc9lCUWgXC5scW/T+IcMsb129iQx/0VmaNlPlcJLfiSrhC3 fwb2VIRq21rEl4bcclrrMvS3eGU2A8urPgkyQvHH2u5u46+soB7OxkeMjRAg7KMMsC01 vkUbVKwQrnFj1YvdMQWEF7wchQITt4GquEMuxg8DSk/MEk/qBQOQIgnyvBoIMOyzAtJV KH7GFi9TZv/mn9v3BtN1TjRDDS1nUwO5ZlbG+3QZLtpCjbnsqUsmJ70lIA+6pvinMxwv J1rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=Vfrai/bcjoInkAH4oWz3L3IRd0JDk7/H+al2UsAGAXI=; b=a/Do5Q2/J+CxNGu0QwPPIirRHjM8orMl5zKpF1ufdvmsnAcQEc3ZJONU3yEyGCUSag 2VnIEVgMVX3OuDcorkHqJwAXTNA5bxb5ec8SWYIHlUUD0Txm4K6kk/SzsaK5KdZRWGfv lEnSclO2vIG+OnvJLmy8aAcg1eLopT4owsEaAbRtMFsZFr2uAbUL/U9aToQAS22dNgfg uIecO4qbcIeXSiyjjPbvObApjRjU7Ndx9ZEklTcWPxOv+xBsVSmT4GOkYKouqQGRr7ej s2L+SWbRhOM6Jz/6vzdcbI0LWA5cQweN5Z1iVWn1bSsABK8wLUpAbXwoF+eeffLB8x1D V67A== X-Gm-Message-State: AKaTC033PvLxxPeuTbClhoNkLmjJzT2t4XIaNA62IiyYabO4momcP3mx0insswFUvklzBg== X-Received: by 10.28.31.65 with SMTP id f62mr3667072wmf.108.1481128096920; Wed, 07 Dec 2016 08:28:16 -0800 (PST) Received: from volcano.localdomain (host-92-31-0-184.as13285.net. [92.31.0.184]) by smtp.googlemail.com with ESMTPSA id v10sm32248079wji.29.2016.12.07.08.28.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Dec 2016 08:28:16 -0800 (PST) To: "Steinar H. Gunderson" , Neal Cardwell References: <56F6A3AB-3A47-4178-BEFF-04E3DC23B039@gmail.com> <20161202224006.GA5065@sesse.net> <1480721486.18162.392.camel@edumazet-glaptop3.roam.corp.google.com> <20161203191353.GA972@sesse.net> Cc: Jonathan Morton , "aqm@ietf.org" , bloat From: Alan Jenkins Message-ID: <83609420-9ab8-666d-8282-e2f22b17d68d@gmail.com> Date: Wed, 7 Dec 2016 16:28:15 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20161203191353.GA972@sesse.net> Content-Type: multipart/alternative; boundary="------------B68F0D472A2E7516F9115738" Subject: Re: [Bloat] TCP BBR paper is now generally available X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2016 16:28:18 -0000 This is a multi-part message in MIME format. --------------B68F0D472A2E7516F9115738 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 03/12/16 19:13, Steinar H. Gunderson wrote: > On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote: >>> I have one thing that I _wonder_ if could be BBR's fault: I run >>> backup over SSH. (That would be tar + gzip + ssh.) The first full >>> backup after I rolled out BBR on the server (the one sending the >>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of >>> free I/O, and neither tar nor gzip (well, pigz) used a full core. >>> My only remaining explanation would be that somehow, BBR didn't >>> deal well with the irregular stream of data coming from tar. (A >>> wget between the same machines at the same time gave 6-700 >>> Mbit/sec.) >> Thanks for the report, Steinar. This is the first report we've had >> like this, but it would be interesting to find out what's going >> on. >> >> Even if you don't have time to apply the patches Eric mentions, it >> would be hugely useful if the next time you have a slow transfer >> like that you could post a link to a tcpdump packet capture >> (headers only is best, say -s 120). Ideally the trace would >> capture a whole connection, so we can see the wscale on the SYN >> exchange. > > I tried reproducing it now. I can't get as far down as 50 Mbit/sec, > but it stopped around 100 Mbit/sec, still without any clear > bottlenecks. cubic was just as bad, though. > > I've taken two tcpdumps as requested; I can't reboot this server > easily right now, unfortunately. They are: > > http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg > http://storage.sesse.net/bbr2.pcap -- wget between same hosts > > /* Steinar */ Since no-one's explicitly mentioned this: be aware that SSH is known for doing application-level windowing, limiting performance. E.g. see https://www.psc.edu/index.php/hpn-ssh/638 --------------B68F0D472A2E7516F9115738 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 03/12/16 19:13, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote:

>>> I have one thing that I _wonder_ if could be BBR's fault: I run
>>> backup over SSH. (That would be tar + gzip + ssh.) The first full
>>> backup after I rolled out BBR on the server (the one sending the
>>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of
>>> free I/O, and neither tar nor gzip (well, pigz) used a full core.
>>> My only remaining explanation would be that somehow, BBR didn't
>>> deal well with the irregular stream of data coming from tar. (A
>>> wget between the same machines at the same time gave 6-700
>>> Mbit/sec.)

>> Thanks for the report, Steinar. This is the first report we've had
>>  like this, but it would be interesting to find out what's going
>> on.
>>
>> Even if you don't have time to apply the patches Eric mentions, it
>>  would be hugely useful if the next time you have a slow transfer
>> like that you could post a link to a tcpdump packet capture
>> (headers only is best, say -s 120). Ideally the trace would
>> capture a whole connection, so we can see the wscale on the SYN
>> exchange.
>
> I tried reproducing it now. I can't get as far down as 50 Mbit/sec,
> but it stopped around 100 Mbit/sec, still without any clear
> bottlenecks. cubic was just as bad, though.
>
> I've taken two tcpdumps as requested; I can't reboot this server
> easily right now, unfortunately. They are:
>
> http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg
> http://storage.sesse.net/bbr2.pcap -- wget between same hosts
>
> /* Steinar */

Since no-one's explicitly mentioned this: be aware that SSH is known for doing application-level windowing, limiting performance.

E.g. see https://www.psc.edu/index.php/hpn-ssh/638

--------------B68F0D472A2E7516F9115738--