[Cerowrt-devel] taking apart BBR's behaviors in flent

[Cerowrt-devel] taking apart BBR's behaviors in flent

[Dave Taht]

Adding in the BBR list. (I have no problem with this discussion moving
entirely to the BBR list...)

The flent dataset we're looking at is here:
http://blog.cerowrt.org/flent/bbr-comprehensive.tgz # this might get
bigger, I have pfifo tests running now

My description of what I was doing and what I observed, started here:

https://lists.bufferbloat.net/pipermail/cerowrt-devel/2016-September/010858.html

where I basically setup a 48ms 20Mbit internet emulation and hit it
with every combination of cubic,bbr, with and without ecn, against all
forms of queue management (bfifo, pie, fq-codel, cake (with and
without fq))

On Wed, Sep 21, 2016 at 5:40 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> * It seriously outcompetes cubic, particularly on the single queue aqms.
>> fq_codel is fine. I need to take apart the captures to see how well it is
>> behaving in this case. My general hope was that with fq in place, anything
>> that was delay based worked better as it was only competing against itself.
>
>
> I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or just
> bfifo? Looks like there is no fq.

The structure of the test naming is
srcqdisc-bottleneckqdisc-otherparameters, so this was coming from
sch_fq, going through bfifo set for 256k in either direction. Sorry
for the confusion.

> If someone doesn't have the correct Flent available, I posted two
> screenshots here: http://imgur.com/a/cFtMd

You can get a bit more detail by zooming in on the plot via the
controls. The spike at the end is an artifact we sometimes get from
the dataset, which messes up the auto-scaling. We also tend to kick
into a log scale automatically more often than I'd like, you can
disable log scaling via the menu.

Also, you can save plots in any format (no need for a screen shot),
and doing comparison plots is easy with the file browser or data->add
other open files.

I'm perpetually taking a slice at the problem with flent-gui
*noecn*.gz or flent-gui *{cake,pie)*-noecn*.gz before resorting to the
file browser to combine plots more intelligently.

Thx for taking a look and posting your pics!

> What I think I see:
>
> The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit
> confusing, but according to your description).

Your interpretation of the sequencing is correct. BBR, 3 seconds
later, cubic2, 3 seconds later BBR4, 3 seconds later cubic3, all tests
run for 60 seconds each.

I will change the labling (and add more plot types) after the coffee
kicks into my system. Other suggestions for structuring an A/B test
like this welcomed - prior to this I'd had something similar that
started things on 5 second intervals. That "beat" against BBR's 10
second interval, so I went with 3 second intervals alternating the cc
types. I could switch to primes?

I could use a better name - it's not a "square wave" - for the
"tcp_4up_squarewave" test itself, also.

> So it looks like BBR1 fills the pipe within half a second or so, nice steady
> state. Then CUBIC2 starts, and slowly over a few seconds, starts to starve
> BBR1 of BW, it looks like steady state here would be that CUBIC2 would end
> up with around 65-70% of the BW, and BBR1 getting 30-35%.

I can't draw that conclusion on 3 seconds worth of competing data on
drop tail, so a longer test of staggering the start for the two
different cc's is needed.

>Then BBR4 comes
> along (10 seconds in), and just KILLS them both, smacks them over the head
> with a hammer, taking 90% of the BW, wildly oscillating in BW way above 20b
> megabit/s down to 10.

Latecomer advantage. And it's not "oscillating above 20mbits" per se'
- that's impossible - there's packet loss, so what flent reports is
the "delivered" throughput, which oscillates as a function of losses
filled in..

I was sampling on a 50ms interval here, which is as low as flent can
go, and close to the RTT. Were I sampling at flent's higher default
(200ms) intervals, or running at a lower RTT, less "occilations" would
have been apparent.

I have been bit *badly* lately by sampling at rates above what nyquist
would recommend, and here I would have preferred to have sampled at
1/2 the baseline RTT.

This would be a lot easier to see from a capture, looking at sacks, and/or cwnd.

> The ping here goes up to around 150-160ms. CUBIC3

single queues suck

> starts at 15 seconds and get basically no bw at all.
>
> Then at around 22 seconds in, I guess pretty close to 12-13 seconds after
> BBR4 was started, BBR4 starts to calm down, slowly letting the other streams
> come back to life. At around 30 seconds, they all seem to get at least a bit
> of the bw each and nobody is completely starved, but BBR1 seems to not get
> much BW at all (very dotted line).

T+28 looks to me as though all flows have got close to their fair
share (this is quite a long path it would take ages to see the real
long term behavior), and after flows start to die off (60 seconds
each, offset by their starting delay), stuff seems to grab back what
they should more or less.

I can grow the test from 60 to 600 seconds and stagger the starts out that
way to grow it... or shorten the RTT... or buffer size. I did a string
of tests against pfifo_1000 while I was asleep, haven't looked yet.


> When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 has a
> 2/3 to 1/3 advantage.

Too little detail to tell. Possibly. But maybe not, after another 10sec.

> Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC flows.
> Same with PIE.

Yep. The single queue AQMs expect their induced drops to matter to all
flows. BBR disregards them as noise. I think there's hope though, if
BBR can treat ECN CE as a clear indication of of congestion and not
filter it as it does drops.

But cake/fq_codel is just fine with different cc's in the mix, and I'm
dying to look at the captures for what happens there.

> So it seems my intuition was wrong, at least for these scenarios. It wasn't
> CUBIC that would kill BBR, it's the other way around.

My intuition was that "delay based TCPs can't work on the internet!" -
and was wrong, also.

> Great to have testing
> tools! Thanks Flent!

Thx, toke! I try not to remember just how hard it was to do this sort
of analysis on complex network flows when we started.


>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cerowrt-devel] [bbr-dev] taking apart BBR's behaviors in flent

[Neal Cardwell]

On Wed, Sep 21, 2016 at 3:25 PM, Dave Taht <dave.taht@gmail.com> wrote:
> > So it seems my intuition was wrong, at least for these scenarios. It wasn't
> > CUBIC that would kill BBR, it's the other way around.
>
> My intuition was that "delay based TCPs can't work on the internet!" -
> and was wrong, also.

Keep in mind that BBR is not really "delay-based", at least in the
traditional sense. BBR is not based on backing off in response to a
single signal like loss or RTT increases. If BBR could be said to be
"based" on any one thing, it's "model-based": it has a model of the
network with two parameters: bottleneck bandwidth and round-trip
propagation time. So delay increases do not always lead to a slower
sending rate or lower volume of data in flight. For example, if the
round-trip propagation delay increases but the bandwidth stays
constant, BBR can actually increase the amount of data in flight in
order to achieve its fair share of the bandwidth available in the
longer pipe.

Thanks for all this testing!

neal

Re: [Cerowrt-devel] taking apart BBR's behaviors in flent

[Dave Taht]

I have pushed an updated set of bbr verses cubic tests (formerly
cake/fq_codel/pie/bfifo_256/cake_flowblind)  now including pfifo_100,
pfifo_1000, and a CMTS/cable modem emulation that dates back to the
network conditions we had in 2011: 20Mbit down, 5Mbit up, when the
bufferbloat project began.

http://blog.cerowrt.org/flent/bbr-comprehensive.tgz

I note that my conditions are reversed throughout: "up" is from the
server to the client (20Mbit). Mae Culpa: I dare not change anything
on the clients right now as other testing is going on. So what I'm
measuring and emulating here, is downstream bufferbloat (CMTS to
cablemodem to end user), rather than what Jim Gettys started with
(seeing uplink bloat measured in seconds).

Since 2011, download and upload speeds have improved (I think
100/20Mbit is most common nowadays(?), modem buffering on some models
has declined somewhat, CMTS configurations have got better - but that
emulation remains close to correct and in some cases, is much less
bloated than the results many cable users still get today. I also have
dsl, and other emulations, but I figure this dataset is enough to go
on for a while. (since 2011 Web pages have got "tighter", uTP has
deployed to a larger extent, BQL and other mods to the linux stack are
now common, etc, etc. also)

I did produce a couple graphs.

2011 in all it's cubic bloated glory:

http://blog.cerowrt.org/flent/bbr-comprehensive/bufferbloat_20Mbit_2011_glory_cubic.svg

vs BBR competing with itself:

http://blog.cerowrt.org/flent/bbr-comprehensive/bufferbloat_bbr_2016.svg

And one showing a full fledged simultaneous up/down test with bbr +
cubic on the increasingly misnamed squarewave test.

http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_vs_cubic_20Mbit_cablemodem_emulation.png

I do not have more time or energy to review this further today, nor
look at the captures. Probably. Wet Paint!

Enjoy the data. Happy graphing!

PS: one hysterical thing that happened: when I'd started pushing this
data to my git repository and website, I'd forgotten I'd left the
cable modem emulation on and rtt 48 stuff enabled - as well as cubic -
and I had to endure watching the darn thing slow, stop, resume in a
burst, and so on, for what felt like ages.

Re: [Cerowrt-devel] taking apart BBR's behaviors in flent

[Alan Jenkins]

[-- Attachment #1.1: Type: text/plain, Size: 2511 bytes --]

On Wednesday, 21 September 2016 20:25:32 UTC+1, Dave Taht wrote:
>
> > Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC 
> flows. 
> > Same with PIE. 
>
> Yep. The single queue AQMs expect their induced drops to matter to all 
> flows. BBR disregards them as noise. I think there's hope though, if 
> BBR can treat ECN CE as a clear indication of of congestion and not 
> filter it as it does drops.
>

Extra credit assignment: get the next version of DOCSIS PIE to turn on ECN?

https://tools.ietf.org/html/draft-ietf-aqm-docsis-pie-02#section-4.7
 

> But cake/fq_codel is just fine with different cc's in the mix, and I'm 
> dying to look at the captures for what happens there.
>

Very glad to see that, I can keep using fq_codel :).

> So it seems my intuition was wrong, at least for these scenarios. It 
> wasn't 
> > CUBIC that would kill BBR, it's the other way around.


So (from the other thread) BBR is designed to use the traditional 
recommendation of 1 BDP's worth of buffer.  In the absence of other CC's, 
it would limit itself to that.  Understandable for bottlenecks in end-site 
modems or wifi.

Shallower buffers cause somewhat increased packet loss (given multiple 
competing BBR streams).  BBR is designed to survive this without difficulty 
(incurring retransmit latency).  Competing loss-based CCs will suffer badly.

The patch says it's designed to improve throughput "on today's high-speed 
long-haul links using commodity switches with shallow buffers" by not 
"[over-reacting] to losses caused by transient traffic bursts".

If there is systemic congestion at those switches[1]...

[1] ex
https://www.ncta.com/sites/prod/files/MIT-Congestion-DC.pdf
http://groups.csail.mit.edu/ana/Measurement-and-Analysis-of-Internet-Interconnection-and-Congestion-September2014.pdf

...I wait with interest to see what the ACM article says.

My intuition was that "delay based TCPs can't work on the internet!" - 
> and was wrong, also. 
>

> > Great to have testing 
> > tools! Thanks Flent! 
>
> Thx, toke! I try not to remember just how hard it was to do this sort 
> of analysis on complex network flows when we started.
>

And thanks for the matrix of test results!

It shows how powerful a tool it is, to see points raised so quickly.

If I'm reading the drops graph right, I can see multi-second periods >= 10% 
packet loss when the buffer is limited to 25ms (bfifo_64k, 
bw=20Mbit-rtt=48ms-flows=2-noecn-bbr).  Clearly explains why normal CUBIC 
gets crushed :).

Alan

[-- Attachment #1.2: Type: text/html, Size: 3432 bytes --]

Re: [Cerowrt-devel] [bbr-dev] Re: taking apart BBR's behaviors in flent

[Neal Cardwell]

On Thu, Sep 22, 2016 at 8:09 AM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On Wednesday, 21 September 2016 20:25:32 UTC+1, Dave Taht wrote:
>>
>> > Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC
>> > flows.
>> > Same with PIE.
>>
>> Yep. The single queue AQMs expect their induced drops to matter to all
>> flows. BBR disregards them as noise. I think there's hope though, if
>> BBR can treat ECN CE as a clear indication of of congestion and not
>> filter it as it does drops.
>
>
> Extra credit assignment: get the next version of DOCSIS PIE to turn on ECN?
>
> https://tools.ietf.org/html/draft-ietf-aqm-docsis-pie-02#section-4.7
>
>>
>> But cake/fq_codel is just fine with different cc's in the mix, and I'm
>> dying to look at the captures for what happens there.
>
>
> Very glad to see that, I can keep using fq_codel :).
>
>> > So it seems my intuition was wrong, at least for these scenarios. It
>> > wasn't
>> > CUBIC that would kill BBR, it's the other way around.
>
>
> So (from the other thread) BBR is designed to use the traditional
> recommendation of 1 BDP's worth of buffer.  In the absence of other CC's, it
> would limit itself to that.  Understandable for bottlenecks in end-site
> modems or wifi.

I mentioned this in another thread this afternoon, but will mention it
here as well,
since it's an important point:

Yes, the current behavior where of creating a ~1*BDP queue when bulk BBR flows
share a bottleneck is something we are actively working on improving. The
current behavior is not the end goal, but rather the place where the current
code arrives due to some subtle trade-offs.  We want the number of packets in
flight to be allowed to potentially be ~1*BDP beyond the pipe BDP, in order to
handle delays that are due to link-layer or receiver effects that cause
delayed, stretched, or aggregated ACKs (these are very common for wifi,
cellular, and cable systems). But ideally we don't want more packets in flight
if the delays are simply due to competing flows, since that can lead to the
~1*BDP of queue you can see in these tests. So we are working on improving the
behavior (reducing the queue) in the case with competing BBR flows.

neal