From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x235.google.com (mail-lf0-x235.google.com [IPv6:2a00:1450:4010:c07::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 88D223B2A3 for ; Thu, 4 May 2017 13:40:08 -0400 (EDT) Received: by mail-lf0-x235.google.com with SMTP id h4so12280130lfj.3 for ; Thu, 04 May 2017 10:40:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=cm2hBqZI6sWGFOhbHgiccwKOjDj/ZHDrLyKsXYFCEAw=; b=lMDHfCvCh89HggZsN4EgM35o6SLU7pXZvuMstzU3X1KUnKwAiRtlp+5+uqNiwMZ1PC w5lcqdq39QN3zomRwUFfcs5eMwg0Eh/ckzWpp9a9Efc8wK6sKy4CN6k9K9mrQieCfZoZ VHqJuZbuOoNtFt61ZjnSJIZXHpqEz391u8xEBDQypRtJYOn8lDe+JgJZhrDmwtf9s9ow M+yXwohVJg5QqRQb/sKDxEpl3gtkx7e+xV3CGY/gajS3vYA3GAlF1K5gBoB0x2ilKnQA EZEe2z1XalJIhkE0bxsof+WS0M2ER27tFXnRdS3nMAN6krcrn9weFb9O/7dv7mO/Dofw OmiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=cm2hBqZI6sWGFOhbHgiccwKOjDj/ZHDrLyKsXYFCEAw=; b=W1HqYXMY0iQiW/ST/9fElcASyakpsdcSMll9gS7RbPZmhHq/pBGr3tQNUQTrAUb5aM pKRFTge5X1fKYpTqsnVCEnXKxg8O5zVGOCHLl/Aj+vGNnyoMbXQ3jO76VtmTVXqwv/+T 0aHkpvkbJFSD7GNrVSpb9bIH3HHv7dwp54Hza61RO7sfkad8w6XRSJiSauHrcYwSZe4d 59LT39d2yZNJWI2rj0e3+isYTbkOeg0ix+1ZDHMdGx67pnijAdYLJoUl0WS0zsOv4CQU uLsFRHYwUOsd3+Qj16jhht/LYJqbjkZfPOhLLaqJ/ZQa/ad2MTEVbwGLn9W8qL5mZuTQ pddQ== X-Gm-Message-State: AN3rC/4GMtJxLXFp22uwhfYOW37CdR2oFbodjkD+FU00tDqf7D3c+VA0 p/wXkSLwYRC8aUdKv4dsSa+r0+WjtA== X-Received: by 10.46.33.11 with SMTP id h11mr17156482ljh.57.1493919607280; Thu, 04 May 2017 10:40:07 -0700 (PDT) MIME-Version: 1.0 Sender: gettysjim@gmail.com Received: by 10.25.77.79 with HTTP; Thu, 4 May 2017 10:40:06 -0700 (PDT) In-Reply-To: <5e76ed9f-7eae-f309-2d64-3ed34f19d3d4@gmail.com> References: <85386ca9-f0be-f60f-796a-e5aa3b8ee212@gmail.com> <5ca09d5c-e674-110a-72e4-b8fd434c7a5d@gmail.com> <5e76ed9f-7eae-f309-2d64-3ed34f19d3d4@gmail.com> From: Jim Gettys Date: Thu, 4 May 2017 13:40:06 -0400 X-Google-Sender-Auth: OcRU7uRgVcYbe1ov1lWJPLUdZT4 Message-ID: To: Andy Furniss Cc: xnor , David Lang , Cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=001a1142c25883da26054eb643b1 Subject: Re: [Cake] cake default target is too low for bbr? X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 May 2017 17:40:08 -0000 --001a1142c25883da26054eb643b1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, May 4, 2017 at 6:22 AM, Andy Furniss wrote: > Jim Gettys wrote: > >> On Wed, May 3, 2017 at 5:50 AM, Andy Furniss >> wrote: >> >> Andy Furniss wrote: >>> >>> Andy Furniss wrote: >>>> >>>> b) it reacts to increase in RTT. An experiment with 10 Mbps >>>> bottleneck, >>>> >>>>> 40 ms RTT and a typical 1000 packet buffer, increase in RTT >>>>>> with BBR is ~3 ms while with cubic it is over 1000 ms. >>>>>> >>>>>> >>>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I >>>>> tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I >>>>> deliberately test using ifb on my PC because I want to pretend >>>>> to be a router - IME (OK it was a while ago) testing on eth >>>>> directly gives different results - like the locally generated >>>>> tcp is backing off and giving different results. >>>>> >>>>> >>>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo >>>> on ifb. >>>> >>>> >>> So, as Jonathan pointed out to me in another thread bbr needs fq >>> and it seems fq only wotks on root of a real eth, which means thay >>> are invalid tests. >>> >>> >> =E2=80=8BSpecifically, BBR needs packet pacing to work properly: the >> algorithm depends on the packets being properly paced. >> >> Today, fq is the only qdisc supporting pacing. >> >> The right answer would be to add packet pacing to cake/fq_codel >> directly. Until that is done, we don't know how BBR will work in our >> world. - Jim=E2=80=8B >> > > I guess you mean so cake could be used on egress of sender (in place of > fq)? > =E2=80=8BYes. =E2=80=8B > > That's not really the test that I intend to do, which is more like - > > [boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress shape] > a bit lower than "line" rate and see how much "ISP" buffer gets filled. > > Also compare bbr, cubic and netem different rtts etc. =E2=80=8BOk. The usual warnings about netem being dangerous apply, though = netem can be useful if run on a separate machine. Netem is an attractive nuisance, but has caused lots of results to be ultimately useless.... Be careful. - Jim =E2=80=8B > > > >>> I will soon (need to find a crossover cable!) be able to see using >>> a third sender how cake varies shaping bbr in simulated ingress. >>> >>> I can test now how bbr fills buffers - some slightly strange >>> results, one netperf ends up being "good" =3D buffer only a few ms. >>> >>> 5 netperfs started together are not so good but nothing like >>> cubic. >>> >>> 5 netperfs started with a gap of a second or two are initially >>> terrible, filling the buffer for about 30 seconds, then eventually >>> falling back to lower occupancy. >>> >>> TODO - maybe this is a netperf artifact like bbr/fq thinks it is >>> app limited. >>> >>> The worse thing about bbr + longer RTT I see so far is that its >>> design seems to be to deliberately bork latency by 2x rtt during >>> initial bandwidth probe. It does drain afterwards, but for >>> something like dash generating a regular spike is not very game >>> friendly and the spec "boasts" that unlike cubic a loss in the >>> exponential phase is ignored, making ingress shaping somewhat less >>> effective. >>> >> --001a1142c25883da26054eb643b1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Thu, Ma= y 4, 2017 at 6:22 AM, Andy Furniss <adf.lists@gmail.com> w= rote:
Jim Gettys wrote:
On Wed, May 3, 2017 at 5:50 AM, Andy Furniss <adf.lists@gmail.com>
wrote:

Andy Furniss wrote:

Andy Furniss wrote:

b) it reacts to increase in RTT. An experiment with 10 Mbps
bottleneck,
40 ms RTT and a typical 1000 packet buffer, increase in RTT
with BBR is ~3 ms while with cubic it is over 1000 ms.


That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I
tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I
deliberately test using ifb on my PC because I want to pretend
to be a router - IME (OK it was a while ago) testing on eth
directly gives different results - like the locally generated
tcp is backing off and giving different results.


I retested this with 40ms latency (netem) with hfsc + 1000 pfifo
on ifb.


So, as Jonathan pointed out to me in another thread bbr needs fq
and it seems fq only wotks on root of a real eth, which means thay
are invalid tests.


=E2=80=8BSpecifically, BBR needs packet pacing to work properly: the
algorithm depends on the packets being properly paced.

Today, fq is the only qdisc supporting pacing.

The right answer would be to add packet pacing to cake/fq_codel
directly. Until that is done, we don't know how BBR will work in our world. - Jim=E2=80=8B

I guess you mean so cake could be used on egress of sender (in place of fq)= ?

=E2=80=8BYes.
=E2=80=8B

That's not really the test that I intend to do, which is more like -
[boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress sha= pe]
a bit lower than "line" rate and see how much "ISP" buf= fer gets filled.

Also compare bbr, cubic and netem different rtts etc.

=
=E2=80=8BOk.= =C2=A0 The usual warnings about netem being dangerous apply, though netem c= an be useful if run on a separate machine.=C2=A0 Netem is an attractive nui= sance, but has caused lots of results to be ultimately useless....=C2=A0 Be= careful.
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 - Jim
=E2=80=8B



I will soon (need to find a crossover cable!) be able to see using
a third sender how cake varies shaping bbr in simulated ingress.

I can test now how bbr fills buffers - some slightly strange
results, one netperf ends up being "good" =3D buffer only a few m= s.

5 netperfs started together are not so good but nothing like
cubic.

5 netperfs started with a gap of a second or two are initially
terrible, filling the buffer for about 30 seconds, then eventually
falling back to lower occupancy.

TODO - maybe this is a netperf artifact like bbr/fq thinks it is
app limited.

The worse thing about bbr + longer RTT I see so far is that its
design seems to be to deliberately bork latency by 2x rtt during
initial bandwidth probe. It does drain afterwards, but for
something like dash generating a regular spike is not very game
friendly and the spec "boasts" that unlike cubic a loss in the exponential phase is ignored, making ingress shaping somewhat less
effective.

--001a1142c25883da26054eb643b1--