From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A6D1E3B2A0; Wed, 21 Sep 2016 09:49:20 -0400 (EDT) Received: by mail-lf0-x233.google.com with SMTP id g62so41526984lfe.3; Wed, 21 Sep 2016 06:49:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=Sa5JcEYVvXf4+rD61/k2pVXPayU9AHh6Sh2eQAyOmqs=; b=maNwLcaj1UUVTQJp7U1QHeVCFrMFkBnfSrGV8MLqHfKfjKgBHwoJBtMjML7TayhFKE N/Bl+mGeKkzLRxTH+Q9qIqN7rXsrbb1jmzEXG5axEyZo0pxeOf0uyR4HUcFEqwPCm6AZ 9XrxiwDVtTkAJ2+n54aAo8/v0g9dj8eVFAcxFeH2rH2T1ZMfYSFZoijNB6esGJytWMau 4N9cZDgENSVulB3xOgE/8pZ+z7rr4pXaHSSZJsvZjp5A3M88aWSnBKyMSiaPBcYSfD+X vsc7yOYPnm/YN40cwpljdhbSsb+x2tUAn4YGPY44WoiwQTJzbsDiDy+jx2ML6fnoSRPk HcgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=Sa5JcEYVvXf4+rD61/k2pVXPayU9AHh6Sh2eQAyOmqs=; b=cGJI9Hkr+eT3avQummn9TcR6OoRgOLtWLaAElkQpV+CLUFN6rUKyKnOw8S/aFntkfE G/OXyUohw10c3DKJqX9OxSK2qcgXG8sVKFCxS6Wm+NrFF7mBxGwL/jc1WP9BUKtVLJdC Bx2swaA87QFCs0neyGoxQMTSS8eXbXpIrCa1MISbM26HEllx8gYzW7lbL/ESRWwvGYig 4oKR7QJ8Pe/EiDLyOL82Rf87mVmxOMI5q2Z619yMvFYVBlTfXqf8tLTXfBeAQQRkuQZ8 XZpHjRm+TrR+cLJL8PQVZf0y1s5N1lb5rJ8lJni2z+kDzF5sSSiXsI6rofAAwSqGxRlx b41A== X-Gm-Message-State: AE9vXwNcibkpWzJUKal4nZj6ap+Q0d36h/j/3mTBq5z73rcpc4vgGs21V5h1U/YUgxtNLw== X-Received: by 10.194.85.18 with SMTP id d18mr32922841wjz.43.1474465759256; Wed, 21 Sep 2016 06:49:19 -0700 (PDT) Received: from localhost.localdomain (host-89-243-172-136.as13285.net. [89.243.172.136]) by smtp.googlemail.com with ESMTPSA id ml1sm33683985wjb.46.2016.09.21.06.49.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Sep 2016 06:49:18 -0700 (PDT) To: Mikael Abrahamsson , Dave Taht References: <92a6ae25-530f-1837-addd-8a9ef07dd022@gmail.com> Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat From: Alan Jenkins Message-ID: Date: Wed, 21 Sep 2016 14:49:15 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------63D2BE56861E04212C5C3AA7" Subject: Re: [Cerowrt-devel] [Bloat] BBR congestion control algorithm for TCP in net-next X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Sep 2016 13:49:21 -0000 This is a multi-part message in MIME format. --------------63D2BE56861E04212C5C3AA7 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 21/09/16 13:40, Mikael Abrahamsson wrote: > On Wed, 21 Sep 2016, Dave Taht wrote: > >> * It seriously outcompetes cubic, particularly on the single queue >> aqms. fq_codel is fine. I need to take apart the captures to see how >> well it is behaving in this case. My general hope was that with fq in >> place, anything that was delay based worked better as it was only >> competing against itself. > > I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or > just bfifo? Looks like there is no fq. the queue before the (simulated) _bottleneck_ is simple bfifo. sch_fq is used on the _endpoint_, because it's a requirement of tcp-bbr It's disambiguated by knowing you don't add child qdiscs to sch_fq, and no-one has written a linux qdisc called fq_bfifo. > > If someone doesn't have the correct Flent available, I posted two > screenshots here: http://imgur.com/a/cFtMd > > What I think I see: > > The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit > confusing, but according to your description). > > So it looks like BBR1 fills the pipe within half a second or so, nice > steady state. Then CUBIC2 starts, and slowly over a few seconds, > starts to starve BBR1 of BW, it looks like steady state here would be > that CUBIC2 would end up with around 65-70% of the BW, and BBR1 > getting 30-35%. Then BBR4 comes along (10 seconds in), and just KILLS > them both, smacks them over the head with a hammer, taking 90% of the > BW, wildly oscillating in BW way above 20 megabit/s down to 10. The > ping here goes up to around 150-160ms. CUBIC3 starts at 15 seconds and > get basically no bw at all. > > Then at around 22 seconds in, I guess pretty close to 12-13 seconds > after BBR4 was started, BBR4 starts to calm down, slowly letting the > other streams come back to life. At around 30 seconds, they all seem > to get at least a bit of the bw each and nobody is completely starved, > but BBR1 seems to not get much BW at all (very dotted line). > > When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 > has a 2/3 to 1/3 advantage. > > Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC > flows. Same with PIE. > > So it seems my intuition was wrong, at least for these scenarios. It > wasn't CUBIC that would kill BBR, it's the other way around. Great to > have testing tools! Thanks Flent! > wow. I was wondering how BBR ever relinquished bandwidth. "With some difficulty". Though not in the second screenshot where there's a strict AQM (cake flowblind). Then it shares with other BBRs while never letting CUBIC get established. (I think if you look at drops, they'd be higher than what 4xCUBIC converges to on that link. See https://groups.google.com/forum/#!msg/bbr-dev/EXgiWxHvaEU/wl3Q39svAAAJ ). --------------63D2BE56861E04212C5C3AA7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit On 21/09/16 13:40, Mikael Abrahamsson wrote:
On Wed, 21 Sep 2016, Dave Taht wrote:

* It seriously outcompetes cubic, particularly on the single queue aqms. fq_codel is fine. I need to take apart the captures to see how well it is behaving in this case. My general hope was that with fq in place, anything that was delay based worked better as it was only competing against itself.

I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or just bfifo? Looks like there is no fq.

the queue before the (simulated) _bottleneck_ is simple bfifo.

sch_fq is used on the _endpoint_, because it's a requirement of tcp-bbr

It's disambiguated by knowing you don't add child qdiscs to sch_fq, and no-one has written a linux qdisc called fq_bfifo. 


If someone doesn't have the correct Flent available, I posted two screenshots here: http://imgur.com/a/cFtMd

What I think I see:

The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit confusing, but according to your description).

So it looks like BBR1 fills the pipe within half a second or so, nice steady state. Then CUBIC2 starts, and slowly over a few seconds, starts to starve BBR1 of BW, it looks like steady state here would be that CUBIC2 would end up with around 65-70% of the BW, and BBR1 getting 30-35%. Then BBR4 comes along (10 seconds in), and just KILLS them both, smacks them over the head with a hammer, taking 90% of the BW, wildly oscillating in BW way above 20 megabit/s down to 10. The ping here goes up to around 150-160ms. CUBIC3 starts at 15 seconds and get basically no bw at all.

Then at around 22 seconds in, I guess pretty close to 12-13 seconds after BBR4 was started, BBR4 starts to calm down, slowly letting the other streams come back to life. At around 30 seconds, they all seem to get at least a bit of the bw each and nobody is completely starved, but BBR1 seems to not get much BW at all (very dotted line).

When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 has a 2/3 to 1/3 advantage.

Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC flows. Same with PIE.

So it seems my intuition was wrong, at least for these scenarios. It wasn't CUBIC that would kill BBR, it's the other way around. Great to have testing tools! Thanks Flent!


wow.  I was wondering how BBR ever relinquished bandwidth.  "With some difficulty".

Though not in the second screenshot where there's a strict AQM (cake flowblind).  Then it shares with other BBRs while never letting CUBIC get established.  (I think if you look at drops, they'd be higher than what 4xCUBIC  converges to on that link.  See https://groups.google.com/forum/#!msg/bbr-dev/EXgiWxHvaEU/wl3Q39svAAAJ ).
--------------63D2BE56861E04212C5C3AA7--