From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 74A083B2A0 for ; Sat, 3 Dec 2016 15:20:18 -0500 (EST) Received: by mail-pg0-x244.google.com with SMTP id p66so11576592pga.2 for ; Sat, 03 Dec 2016 12:20:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:to:cc:date:in-reply-to:references :mime-version:content-transfer-encoding; bh=c6QfGFOVBHCo3e4RZuSI5c7kpoz0imOeEi12I8mnYBU=; b=P83mRIVLm3XdG1uZ6z8X4aTmUN3idnWD3ZCdiEzCMXESLvObXGBIZ97rhAsk2FHcK9 LdzAWtvRYc8Oqm7ehOljvz9LR9wqSwBEyJAy7dPgUgnApWfe+O5kD5BV5xS6+d7Ue0q6 uC9jTeERO5w5PZTDTwdDRdpBDgz4OPCqhJzcxV+mJvAUfYBwg48rLV9EsXaIZj12XQZ4 F2kNvOHSWYvSPaDfosBcrWg9lTk7KR9BKPabKfwcHD4qxCMG6dmFEKTM1CNozdMsmyj7 C8j+yzdMomTFcJa1FSZu9+TeAhaeznFz01qllee5yyFU/kmuj7SZzIz+oVFKMDdTnAX+ jq6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=c6QfGFOVBHCo3e4RZuSI5c7kpoz0imOeEi12I8mnYBU=; b=TrxADt54xmOLJ2S413/PKnGyrf8jIxUvHX6YcwVwHlIXjaZzhNSLS8FNzawA0KCms6 kY3wOJfVCMvCTX6ybMhgtta3RUABRdnaghCARcpZA9rHv6cNxhz6M7O30FqyLBAAwx01 z+6equpCx2gdIG+5DeaR7p/U4rPb3WNdTBDKHSpr8i0saMe6Xf00Z/9bZUw1MYeosWDt fnyGK6/xkas8yAUDvhzzFvX2CHMADbbegqHNEBRQLgZ9Nn8TOkxauoERdbdLNYx0YfYB NeV2WgLNNBF8so5UOls8WlOR+a6tjWepK9x9Qt9qGu0TqST/08t9ocjt9acjTXq851iC A84Q== X-Gm-Message-State: AKaTC03GqnVHnbe0wGA4WTze60CiqZcxB3SPFftty+CW5fwmHRuCpVMJBMac+bktvC4O9g== X-Received: by 10.84.129.99 with SMTP id 90mr111575799plb.154.1480796417623; Sat, 03 Dec 2016 12:20:17 -0800 (PST) Received: from [172.19.250.57] ([172.19.250.57]) by smtp.googlemail.com with ESMTPSA id y6sm16806763pge.16.2016.12.03.12.20.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 03 Dec 2016 12:20:16 -0800 (PST) Message-ID: <1480796415.18162.408.camel@edumazet-glaptop3.roam.corp.google.com> From: Eric Dumazet To: "Steinar H. Gunderson" Cc: Neal Cardwell , Jonathan Morton , "aqm@ietf.org" , bloat Date: Sat, 03 Dec 2016 12:20:15 -0800 In-Reply-To: <20161203191353.GA972@sesse.net> References: <56F6A3AB-3A47-4178-BEFF-04E3DC23B039@gmail.com> <20161202224006.GA5065@sesse.net> <1480721486.18162.392.camel@edumazet-glaptop3.roam.corp.google.com> <20161203191353.GA972@sesse.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Bloat] TCP BBR paper is now generally available X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2016 20:20:18 -0000 On Sat, 2016-12-03 at 20:13 +0100, Steinar H. Gunderson wrote: > On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote: > > Thanks for the report, Steinar. This is the first report we've had > > like this, but it would be interesting to find out what's going on. > > > > Even if you don't have time to apply the patches Eric mentions, it > > would be hugely useful if the next time you have a slow transfer like > > that you could post a link to a tcpdump packet capture (headers only > > is best, say -s 120). Ideally the trace would capture a whole > > connection, so we can see the wscale on the SYN exchange. > > I tried reproducing it now. I can't get as far down as 50 Mbit/sec, > but it stopped around 100 Mbit/sec, still without any clear bottlenecks. > cubic was just as bad, though. > > I've taken two tcpdumps as requested; I can't reboot this server easily > right now, unfortunately. They are: > > http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg > http://storage.sesse.net/bbr2.pcap -- wget between same hosts > > /* Steinar */ Hi Steinar Huge ACK decimation it seems. 11:13:28.762772 IP6 S > C: Flags [.], seq 777672:779100, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428 11:13:28.763145 IP6 S > C: Flags [.], seq 779100:783384, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 4284 11:13:28.763190 IP6 S > C: Flags [.], seq 783384:789096, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 5712 11:13:28.763239 IP6 S > C: Flags [.], seq 789096:790524, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428 11:13:28.763334 IP6 S > C: Flags [.], seq 790524:791952, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428 11:13:28.764109 IP6 S > C: Flags [.], seq 791952:794808, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 2856 11:13:28.764138 IP6 S > C: Flags [.], seq 794808:800520, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 5712 11:13:28.764189 IP6 S > C: Flags [.], seq 800520:804804, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 4284 11:13:28.764980 IP6 S > C: Flags [.], seq 804804:806232, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 1428 11:13:28.765034 IP6 S > C: Flags [.], seq 806232:811944, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712 11:13:28.765086 IP6 S > C: Flags [.], seq 811944:817656, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712 11:13:28.765905 IP6 S > C: Flags [.], seq 817656:823368, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712 11:13:28.765956 IP6 S > C: Flags [.], seq 823368:829080, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712 11:13:28.766005 IP6 S > C: Flags [.], seq 829080:831936, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 2856 11:13:28.766869 IP6 S > C: Flags [.], seq 831936:834792, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 2856 11:13:28.766898 IP6 S > C: Flags [.], seq 834792:840504, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 5712 11:13:28.766947 IP6 S > C: Flags [.], seq 840504:841932, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 1428 11:13:28.766997 IP6 S > C: Flags [.], seq 841932:843360, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 1428 11:13:28.767532 IP6 C > S: Flags [.], ack 783384, win 4106, options [nop,nop,TS val 864447710 ecr 3498278784], length 0 11:13:28.767541 IP6 C > S: Flags [.], ack 789096, win 4151, options [nop,nop,TS val 864447710 ecr 3498278784], length 0 11:13:28.767543 IP6 C > S: Flags [.], ack 791952, win 4173, options [nop,nop,TS val 864447710 ecr 3498278784], length 0 11:13:28.767544 IP6 C > S: Flags [.], ack 794808, win 4195, options [nop,nop,TS val 864447710 ecr 3498278785], length 0 11:13:28.767546 IP6 C > S: Flags [.], ack 800520, win 4240, options [nop,nop,TS val 864447710 ecr 3498278785], length 0 11:13:28.767547 IP6 C > S: Flags [.], ack 804804, win 4273, options [nop,nop,TS val 864447710 ecr 3498278785], length 0 11:13:28.767549 IP6 C > S: Flags [.], ack 811944, win 4329, options [nop,nop,TS val 864447710 ecr 3498278787], length 0 11:13:28.767550 IP6 C > S: Flags [.], ack 817656, win 4374, options [nop,nop,TS val 864447710 ecr 3498278787], length 0 11:13:28.767552 IP6 C > S: Flags [.], ack 823368, win 4418, options [nop,nop,TS val 864447711 ecr 3498278787], length 0 11:13:28.767553 IP6 C > S: Flags [.], ack 829080, win 4463, options [nop,nop,TS val 864447711 ecr 3498278787], length 0 11:13:28.767554 IP6 C > S: Flags [.], ack 831936, win 4485, options [nop,nop,TS val 864447711 ecr 3498278787], length 0 11:13:28.767556 IP6 C > S: Flags [.], ack 834792, win 4508, options [nop,nop,TS val 864447711 ecr 3498278789], length 0 11:13:28.767557 IP6 C > S: Flags [.], ack 840504, win 4552, options [nop,nop,TS val 864447711 ecr 3498278789], length 0 11:13:28.767559 IP6 C > S: Flags [.], ack 843360, win 4575, options [nop,nop,TS val 864447711 ecr 3498278789], length 0 Just to be clear,what is the kernel version at the sender ? Thanks !