From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-x22b.google.com (mail-it0-x22b.google.com [IPv6:2607:f8b0:4001:c0b::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 1321C3B2A2 for ; Tue, 1 Nov 2016 19:14:38 -0400 (EDT) Received: by mail-it0-x22b.google.com with SMTP id q124so3577034itd.1 for ; Tue, 01 Nov 2016 16:14:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=fxQH5N29Y34BCvFbE1Uabf/kpotiM0m6GXOQuzjJmuU=; b=eWhyEbUT2vQqbl+Um/Pi9flILlRrjj7WCYlkw3X6V8jBvW37aMdWRuvIkFZ9sg0TPl sNFeEZjh6tNapek2fNaJpZcF5fVUisPrye5x5GUNa48pEOc7t/4eGBmyRmCpRXWVwtmD kDRx0V5KcIkv26/8wF8r6wkew/zb6swba6VtCZjeLb31Rxy8aFYDhitT/EZkXsbAoitB KHYDCkKhylYdSkvfQMwdCB7Tx2O9tcCEwVa47CjZyjMcys8jjkf5s39K9txXjhPBIujY a5L3CIOq5Q37odumA+1nS7EwtGlqCyLiZjFpBwhe+0XFs/k7QyvKcj6hN+W6UevYynZi DSrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=fxQH5N29Y34BCvFbE1Uabf/kpotiM0m6GXOQuzjJmuU=; b=Jwlh585YVqRySsmsVdCkdPgAoTQUFwiM1LABZBTI5XA/Kf2mYMyC3IeLtbQPrKGDCU K3rvquWdy5O/RZt0G9mbN4tE6PBemgM7/LmbjYNU2SXGi1deo8Ckzadrtkt1XHIbJaFS yrS+BoikDNpPm6gzs4Jv6wrzG/Z4HZvwreavZ7z2i10NDewPdaMC3BmEa7xfcuVGTodR Q7R0LxGMw/edi2mztYw8JNJv86En/fR0SiPCLaEYa3t+xXiXtbpzrPgjcvP4qYyt6L4w I1TYRxbwxKB0fuDghUJmYJB1ooGeOP1fg36nC/jag6T4MBsCh0IFtEAVc3MGN65LNdzh cgyg== X-Gm-Message-State: ABUngvddTPyyLvSTdAjoY1NLBgpuOCdjRTv2mosNzmPKHlXr9cb+v4alJmLtzqlctpb+FG33oKd1YAmZsYL8xVES X-Received: by 10.36.30.204 with SMTP id 195mr414699itt.90.1478042078298; Tue, 01 Nov 2016 16:14:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.245.163 with HTTP; Tue, 1 Nov 2016 16:13:57 -0700 (PDT) In-Reply-To: References: <20161021084726.GA1913@sesse.net> <20161027170447.GA28383@sesse.net> From: Yuchung Cheng Date: Tue, 1 Nov 2016 16:13:57 -0700 Message-ID: To: Dave Taht Cc: "Steinar H. Gunderson" , bloat , BBR Development Content-Type: multipart/alternative; boundary=001a114484740abf880540457dec Subject: Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2016 23:14:39 -0000 --001a114484740abf880540457dec Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Dave, We are able to reproduce the cubic starvation issue with codel (but not fq_codel), regardless of ECN. So clearly ECN isn't the issue but the single-queued AQM is. We didn't notice that earlier even though you've clearly indicated that in your report. But Cubic and BBR loss response functions are nonlinear so that's expected. We are curious why you choose the single-queued AQM. Is it just for the sake of testing? On Thu, Oct 27, 2016 at 2:30 PM, Yuchung Cheng wrote: > On Thu, Oct 27, 2016 at 11:14 AM, Dave Taht wrote: > > On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng > wrote: > >> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht > wrote: > >>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson > >>> wrote: > >>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote= : > >>>>> As a random data point, I tried a single flow from my main server i= n > .no > >>>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR > (naturally > >>>>> also in sch_fq) on the sender side. The results were quite > consistent across > >>>>> runs: > >>>> > >>>> Another datapoint: A friend of mine had a different, worse path (of > about 40 ms) > >>>> and tested with iperf. > >>>> > >>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485 > Mbit/sec. > >>> > >>> I mostly live in a world (wifi) where loss is uncommon, unless forced > >>> on it with a AQM. > >>> > >>> At the moment my biggest beef with BBR is that it ignores ECN entirel= y > >>> (and yet negotiates it). BBR is then so efficient at using up all the > >>> pipe that a single queued aqm "marks madly" and everything else > >>> eventually starves. Watch "ping" fade out here... > >>> > >>> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_ > eventually_starving_ping.png > >> > >> Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van > >> co-designs both :) > > > > It works pretty darn good with codel without ecn. I'm pretty darn happy > with it. > > > > fq_codel is even more lovely, especially when competing with cubic. > > > > There are issues with single queued aqms with BBR vs cubic > > > > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share- > creaming-cubic-flowblind-aqm.svg > > > >> We have tested BBR with CoDel before and it works. > > > > Well, against cubic on the same link in single queue mode, even > > without ecn, life looks like this: > > > > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share- > creaming-cubic-flowblind-aqm.svg > > > > but fq_codel is fine, so long as there is no ecn vs nonecn collission > > > > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png > > > >> Could you share your tcpdump traces with us (maybe > >> you already did but no sure) or suggest how to reproduce this. > >> > >> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption > >> in your graph) > > > > That's two BBRs with ecn enabled, going through cake in the single > > queue aqm mode "flowblind". I have similar plots with pie and codel > > with ecn enabled somewhere. > > > > The emulation is 48ms RTT, 20Mbit down, 5Mbit up. > > > > Regrettably I'm on a couple deadlines (a talk tomorrow, and next > > thursday), and can't look harder, I do have caps comparing ecn vs > > noecn here > > > > http://blog.cerowrt.org/flent/bbr-ecncaps/ > Thanks for the data (and sorry for ignoring that before). Neal and I > think the behaviors you are observing matches BBR's top issue we are > actively pursuing: with N>1 flows, BBR may build 1.5BDP of queue. But > let's separate that from ECN negotiation. Besidethe implementation > complication Eric pointed out, even if BBR refrains from ECN > negotiation, and the test result wouldn't change much I suspect. > > We'll get back soon. > > > > > > > > >> > >>> > >>> somewhat conversely in fq_codel, this means that it ignores codel's > >>> marking attempts entirely and BBR retains it's own dynamics, (while > >>> the non-BBR flows are fine) which is kind of neat to watch. > >>> > >>>> /* Steinar */ > >>>> -- > >>>> Homepage: https://www.sesse.net/ > >>>> _______________________________________________ > >>>> Bloat mailing list > >>>> Bloat@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/bloat > >>> > >>> > >>> > >>> -- > >>> Dave T=C3=A4ht > >>> Let's go make home routers and wifi faster! With better software! > >>> http://blog.cerowrt.org > >>> _______________________________________________ > >>> Bloat mailing list > >>> Bloat@lists.bufferbloat.net > >>> https://lists.bufferbloat.net/listinfo/bloat > > > > > > > > -- > > Dave T=C3=A4ht > > Let's go make home routers and wifi faster! With better software! > > http://blog.cerowrt.org > > > > -- > > You received this message because you are subscribed to the Google > Groups "BBR Development" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to bbr-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > --001a114484740abf880540457dec Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Dave,

We are able to reproduce the c= ubic starvation issue with codel (but not fq_codel), regardless of ECN. So = clearly ECN isn't the issue but the single-queued AQM is. We didn't= notice that earlier even though you've clearly indicated that in your = report. But Cubic and BBR loss response functions are nonlinear so that'= ;s expected.

We are curious why you choose the sin= gle-queued AQM. Is it just for the sake of testing?



On Thu, Oct 27, 2016 at 2:30 PM, Yuchung Cheng <ycheng@google.com>= wrote:
On Thu, Oct 27, 2016 at 11:14 AM, Dave Taht <dave.taht@gmail.com> wrote:
> On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ycheng@google.com> wrote:
>> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
>>> <sgunderson@bigfo= ot.com> wrote:
>>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunde= rson wrote:
>>>>> As a random data point, I tried a single flow from my = main server in .no
>>>>> to my backup server in .nl and compared CUBIC (with sc= h_fq) to BBR (naturally
>>>>> also in sch_fq) on the sender side. The results were q= uite consistent across
>>>>> runs:
>>>>
>>>> Another datapoint: A friend of mine had a different, worse= path (of about 40 ms)
>>>> and tested with iperf.
>>>>
>>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delive= red 485 Mbit/sec.
>>>
>>> I mostly live in a world (wifi) where loss is uncommon, unless= forced
>>> on it with a AQM.
>>>
>>> At the moment my biggest beef with BBR is that it ignores ECN = entirely
>>> (and yet negotiates it). BBR is then so efficient at using up = all the
>>> pipe that a single queued aqm "marks madly" and ever= ything else
>>> eventually starves. Watch "ping" fade out here... >>>
>>> htt= p://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_s= tarving_ping.png
>>
>> Thanks Dave for this issue. We design BBR with CoDel in mind b/c V= an
>> co-designs both :)
>
> It works pretty darn good with codel without ecn. I'm pretty darn = happy with it.
>
> fq_codel is even more lovely, especially when competing with cubic. >
> There are issues with single queued aqms with BBR vs cubic
>
> http:= //blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cub= ic-flowblind-aqm.svg
>
>> We have tested BBR with CoDel before and it works.
>
> Well, against cubic on the same link in single queue mode, even
> without ecn, life looks like this:
>
> http:= //blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-creaming-cub= ic-flowblind-aqm.svg
>
> but fq_codel is fine, so long as there is no ecn vs nonecn collission<= br> >
> http://blog.cerowrt.org/fle= nt/bbr-ecncaps/bandwidth-share-ecn-fq.png
>
>> Could you share your tcpdump traces with us (maybe
>> you already did but no sure) or suggest how to reproduce this.
>>
>> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on cap= tion
>> in your graph)
>
> That's two BBRs with ecn enabled, going through cake in the single=
> queue aqm mode "flowblind". I have similar plots with pie an= d codel
> with ecn enabled somewhere.
>
> The emulation is 48ms RTT, 20Mbit down, 5Mbit up.
>
> Regrettably I'm on a couple deadlines (a talk tomorrow, and next > thursday), and can't look harder, I do have caps comparing ecn vs<= br> > noecn here
>
> http://blog.cerowrt.org/flent/bbr-ecncaps/
Thanks for the data (and sorry for ignoring that before). Neal = and I
think the behaviors you are observing matches BBR's top issue we are actively pursuing: with N>1 flows, BBR may build 1.5BDP of queue. But let's separate that from ECN negotiation. Besidethe implementation
complication Eric pointed out, even if BBR refrains from ECN
negotiation, and the test result wouldn't change much I suspect.

We'll get back soon.

>
>
>
>>
>>>
>>> somewhat conversely in fq_codel, this means that it ignores co= del's
>>> marking attempts entirely and BBR retains it's own dynamic= s, (while
>>> the non-BBR flows are fine) which is kind of neat to watch. >>>
>>>> /* Steinar */
>>>> --
>>>> Homepage: https://www.sesse.net/
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists= .bufferbloat.net
>>>> https://lists.bufferbloat.net/list= info/bloat
>>>
>>>
>>>
>>> --
>>> Dave T=C3=A4ht
>>> Let's go make home routers and wifi faster! With better so= ftware!
>>> http://blog.cerowrt.org
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.buf= ferbloat.net
>>> https://lists.bufferbloat.net/listin= fo/bloat
>
>
>
> --
> Dave T=C3=A4ht
> Let's go make home routers and wifi faster! With better software!<= br> > http://blog.cerowrt.org
>
> --
> You received this message because you are subscribed to the Google Gro= ups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send= an email to bbr-= dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/opto= ut.

--001a114484740abf880540457dec--