Re: [Bloat] [bbr-dev] Re: "BBR" TCP patches submitted to linux kernel

[Yuchung Cheng <ycheng@google.com>]

[-- Attachment #1: Type: text/plain, Size: 5256 bytes --]

Hi Dave,

We are able to reproduce the cubic starvation issue with codel (but not
fq_codel), regardless of ECN. So clearly ECN isn't the issue but the
single-queued AQM is. We didn't notice that earlier even though you've
clearly indicated that in your report. But Cubic and BBR loss response
functions are nonlinear so that's expected.

We are curious why you choose the single-queued AQM. Is it just for the
sake of testing?



On Thu, Oct 27, 2016 at 2:30 PM, Yuchung Cheng <ycheng@google.com> wrote:

> On Thu, Oct 27, 2016 at 11:14 AM, Dave Taht <dave.taht@gmail.com> wrote:
> > On Thu, Oct 27, 2016 at 10:57 AM, Yuchung Cheng <ycheng@google.com>
> wrote:
> >> On Thu, Oct 27, 2016 at 10:33 AM, Dave Taht <dave.taht@gmail.com>
> wrote:
> >>> On Thu, Oct 27, 2016 at 10:04 AM, Steinar H. Gunderson
> >>> <sgunderson@bigfoot.com> wrote:
> >>>> On Fri, Oct 21, 2016 at 10:47:26AM +0200, Steinar H. Gunderson wrote:
> >>>>> As a random data point, I tried a single flow from my main server in
> .no
> >>>>> to my backup server in .nl and compared CUBIC (with sch_fq) to BBR
> (naturally
> >>>>> also in sch_fq) on the sender side. The results were quite
> consistent across
> >>>>> runs:
> >>>>
> >>>> Another datapoint: A friend of mine had a different, worse path (of
> about 40 ms)
> >>>> and tested with iperf.
> >>>>
> >>>> CUBIC delivered 20.1 Mbit/sec (highly varying). BBR delivered 485
> Mbit/sec.
> >>>
> >>> I mostly live in a world (wifi) where loss is uncommon, unless forced
> >>> on it with a AQM.
> >>>
> >>> At the moment my biggest beef with BBR is that it ignores ECN entirely
> >>> (and yet negotiates it). BBR is then so efficient at using up all the
> >>> pipe that a single queued aqm "marks madly" and everything else
> >>> eventually starves. Watch "ping" fade out here...
> >>>
> >>> http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_
> eventually_starving_ping.png
> >>
> >> Thanks Dave for this issue. We design BBR with CoDel in mind b/c Van
> >> co-designs both :)
> >
> > It works pretty darn good with codel without ecn. I'm pretty darn happy
> with it.
> >
> > fq_codel is even more lovely, especially when competing with cubic.
> >
> > There are issues with single queued aqms with BBR vs cubic
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-
> creaming-cubic-flowblind-aqm.svg
> >
> >> We have tested BBR with CoDel before and it works.
> >
> > Well, against cubic on the same link in single queue mode, even
> > without ecn, life looks like this:
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-
> creaming-cubic-flowblind-aqm.svg
> >
> > but fq_codel is fine, so long as there is no ecn vs nonecn collission
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/bandwidth-share-ecn-fq.png
> >
> >> Could you share your tcpdump traces with us (maybe
> >> you already did but no sure) or suggest how to reproduce this.
> >>
> >> This is 2 bbr flow or bbr + ecn-cubic? (I am guessing based on caption
> >> in your graph)
> >
> > That's two BBRs with ecn enabled, going through cake in the single
> > queue aqm mode "flowblind". I have similar plots with pie and codel
> > with ecn enabled somewhere.
> >
> > The emulation is 48ms RTT, 20Mbit down, 5Mbit up.
> >
> > Regrettably I'm on a couple deadlines (a talk tomorrow, and next
> > thursday), and can't look harder, I do have caps comparing ecn vs
> > noecn here
> >
> > http://blog.cerowrt.org/flent/bbr-ecncaps/
> Thanks for the data (and sorry for ignoring that before). Neal and I
> think the behaviors you are observing matches BBR's top issue we are
> actively pursuing: with N>1 flows, BBR may build 1.5BDP of queue. But
> let's separate that from ECN negotiation. Besidethe implementation
> complication Eric pointed out, even if BBR refrains from ECN
> negotiation, and the test result wouldn't change much I suspect.
>
> We'll get back soon.
>
> >
> >
> >
> >>
> >>>
> >>> somewhat conversely in fq_codel, this means that it ignores codel's
> >>> marking attempts entirely and BBR retains it's own dynamics, (while
> >>> the non-BBR flows are fine) which is kind of neat to watch.
> >>>
> >>>> /* Steinar */
> >>>> --
> >>>> Homepage: https://www.sesse.net/
> >>>> _______________________________________________
> >>>> Bloat mailing list
> >>>> Bloat@lists.bufferbloat.net
> >>>> https://lists.bufferbloat.net/listinfo/bloat
> >>>
> >>>
> >>>
> >>> --
> >>> Dave Täht
> >>> Let's go make home routers and wifi faster! With better software!
> >>> http://blog.cerowrt.org
> >>> _______________________________________________
> >>> Bloat mailing list
> >>> Bloat@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> > Dave Täht
> > Let's go make home routers and wifi faster! With better software!
> > http://blog.cerowrt.org
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "BBR Development" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to bbr-dev+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>

[-- Attachment #2: Type: text/html, Size: 8194 bytes --]