* Re: [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next
[not found] ` <alpine.DEB.2.02.1609211210280.1477@uplift.swm.pp.se>
@ 2016-09-21 11:19 ` Dave Taht
2016-09-21 11:32 ` Mikael Abrahamsson
2016-09-21 12:40 ` Mikael Abrahamsson
0 siblings, 2 replies; 4+ messages in thread
From: Dave Taht @ 2016-09-21 11:19 UTC (permalink / raw)
To: Mikael Abrahamsson, bloat; +Cc: cerowrt-devel
On Wed, Sep 21, 2016 at 3:15 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> I dunno, I'm just reading tea leaves here!
>>
>> can't wait for the paper!
>
>
> +1.
>
> I would like to understand how BBR interacts with a window-fully-open
> classic TCP session and FIFO induced delay that is in steady-state before
> the BBR session starts.
I did a fairly comprehensive string of tests today, comparing it at
20Mbits, 48ms RTT, to cubic and competing with cubic, against a byte
fifo of 256k, pie, cake, cake flowblind, and fq_codel.
The flent datasets are now at:
http://blog.cerowrt.org/flent/bbr-comprehensive.tgz
You will need the latest flent from git to plot the new
tcp_4up_squarewave test. That test starts 1 BBR flow, then 3 seconds
later, cubic, 3 seconds after that, bbr again, 3 seconds after cubic
again.
It's 4am here, and we've also been busy with something else due today,
but some observations:
* respecting ecn appears unimplemented, so if you compare cubic with
ecn against bbr not respecting it, bad things happen. I bug reported
this to the bbr mailing list. To some extent cake's and pie's overload
on ecn drop mechanism helps. fq_codel/cake with fq is fine, so long as
there are no collisions.
* It seriously outcompetes cubic, particularly on the single queue
aqms. fq_codel is fine. I need to take apart the captures to see how
well it is behaving in this case. My general hope was that with fq in
place, anything that was delay based worked better as it was only
competing against itself.
* It does the right things against bfifo when multiple streams are
present, but in the presence of reverse traffic, cubic starves. (the
dataset for this is kind of wrong in that the legend of the
"smackdown" chart claims BBR is on on the download, but it was
unimplemented in that direction, I only had it running on the upload.)
Did I mention fq* was fine? :)
I'm going to bed.
> So let's say I have 100ms of lightspeed-in-fiber RTT, and I am then running
> a file transfer with some other TCP algorithm which is sitting there, window
> fully open, creating an additional 100ms of stupid-router-FIFO-buffering
> delay.
>
> So new BBR TCP session comes along, sees 200ms of RTT, and starts sending. I
> guess the classic TCP algorithm still keeps its window fully open, and
> doesn't care that RTT now increased to 210ms by the BBR flow packets.
I see evidence of the classic latecomer advantage but it subsides in 10-20 sec:
http://blog.cerowrt.org/flent/bbr-comprehensive/latecomer_advantage.svg
>
> Now what? BBR flow sees increased latency, and backs off, right? So how much
> of the bandwidth will each flow get? How do these interact?
Plot it in flent.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next
2016-09-21 11:19 ` [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next Dave Taht
@ 2016-09-21 11:32 ` Mikael Abrahamsson
2016-09-21 12:40 ` Mikael Abrahamsson
1 sibling, 0 replies; 4+ messages in thread
From: Mikael Abrahamsson @ 2016-09-21 11:32 UTC (permalink / raw)
To: Dave Taht; +Cc: bloat, cerowrt-devel
On Wed, 21 Sep 2016, Dave Taht wrote:
> I did a fairly comprehensive string of tests today, comparing it at
> 20Mbits, 48ms RTT, to cubic and competing with cubic, against a byte
> fifo of 256k, pie, cake, cake flowblind, and fq_codel.
20 megabit/s is 2.5 megabyte/s, so that 256k FIFO is only 100ms worth of
buffering. I guess you see packet drop in steady state here, ie buffer is
full?
I'd be interested in seeing same experiment start with 10MB FIFO, and
having CUBIC flow start first and give it proper head start.
My intuition and understanding of what's going to happen might very well
be completely off, but I think it'd be interesting to know.
I'll take a look at your flent data, thanks for posting them!
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next
2016-09-21 11:19 ` [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next Dave Taht
2016-09-21 11:32 ` Mikael Abrahamsson
@ 2016-09-21 12:40 ` Mikael Abrahamsson
2016-09-21 13:49 ` Alan Jenkins
1 sibling, 1 reply; 4+ messages in thread
From: Mikael Abrahamsson @ 2016-09-21 12:40 UTC (permalink / raw)
To: Dave Taht; +Cc: bloat, cerowrt-devel
On Wed, 21 Sep 2016, Dave Taht wrote:
> * It seriously outcompetes cubic, particularly on the single queue aqms.
> fq_codel is fine. I need to take apart the captures to see how well it
> is behaving in this case. My general hope was that with fq in place,
> anything that was delay based worked better as it was only competing
> against itself.
I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or just
bfifo? Looks like there is no fq.
If someone doesn't have the correct Flent available, I posted two
screenshots here: http://imgur.com/a/cFtMd
What I think I see:
The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit
confusing, but according to your description).
So it looks like BBR1 fills the pipe within half a second or so, nice
steady state. Then CUBIC2 starts, and slowly over a few seconds, starts to
starve BBR1 of BW, it looks like steady state here would be that CUBIC2
would end up with around 65-70% of the BW, and BBR1 getting 30-35%. Then
BBR4 comes along (10 seconds in), and just KILLS them both, smacks them
over the head with a hammer, taking 90% of the BW, wildly oscillating in
BW way above 20 megabit/s down to 10. The ping here goes up to around
150-160ms. CUBIC3 starts at 15 seconds and get basically no bw at all.
Then at around 22 seconds in, I guess pretty close to 12-13 seconds after
BBR4 was started, BBR4 starts to calm down, slowly letting the other
streams come back to life. At around 30 seconds, they all seem to get at
least a bit of the bw each and nobody is completely starved, but BBR1
seems to not get much BW at all (very dotted line).
When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4 has
a 2/3 to 1/3 advantage.
Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC
flows. Same with PIE.
So it seems my intuition was wrong, at least for these scenarios. It
wasn't CUBIC that would kill BBR, it's the other way around. Great to have
testing tools! Thanks Flent!
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next
2016-09-21 12:40 ` Mikael Abrahamsson
@ 2016-09-21 13:49 ` Alan Jenkins
0 siblings, 0 replies; 4+ messages in thread
From: Alan Jenkins @ 2016-09-21 13:49 UTC (permalink / raw)
To: Mikael Abrahamsson, Dave Taht; +Cc: cerowrt-devel, bloat
[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]
On 21/09/16 13:40, Mikael Abrahamsson wrote:
> On Wed, 21 Sep 2016, Dave Taht wrote:
>
>> * It seriously outcompetes cubic, particularly on the single queue
>> aqms. fq_codel is fine. I need to take apart the captures to see how
>> well it is behaving in this case. My general hope was that with fq in
>> place, anything that was delay based worked better as it was only
>> competing against itself.
>
> I'm looking at 4up-sqwave-fq_bfifo-256k. Is this really fq_bfifo, or
> just bfifo? Looks like there is no fq.
the queue before the (simulated) _bottleneck_ is simple bfifo.
sch_fq is used on the _endpoint_, because it's a requirement of tcp-bbr
It's disambiguated by knowing you don't add child qdiscs to sch_fq, and
no-one has written a linux qdisc called fq_bfifo.
>
> If someone doesn't have the correct Flent available, I posted two
> screenshots here: http://imgur.com/a/cFtMd
>
> What I think I see:
>
> The flows are started in order: "BBR1, CUBIC2, BBR4, CUBIC3" (a bit
> confusing, but according to your description).
>
> So it looks like BBR1 fills the pipe within half a second or so, nice
> steady state. Then CUBIC2 starts, and slowly over a few seconds,
> starts to starve BBR1 of BW, it looks like steady state here would be
> that CUBIC2 would end up with around 65-70% of the BW, and BBR1
> getting 30-35%. Then BBR4 comes along (10 seconds in), and just KILLS
> them both, smacks them over the head with a hammer, taking 90% of the
> BW, wildly oscillating in BW way above 20 megabit/s down to 10. The
> ping here goes up to around 150-160ms. CUBIC3 starts at 15 seconds and
> get basically no bw at all.
>
> Then at around 22 seconds in, I guess pretty close to 12-13 seconds
> after BBR4 was started, BBR4 starts to calm down, slowly letting the
> other streams come back to life. At around 30 seconds, they all seem
> to get at least a bit of the bw each and nobody is completely starved,
> but BBR1 seems to not get much BW at all (very dotted line).
>
> When at the end there is only CUBIC3 and BBR4 left, it looks like BBR4
> has a 2/3 to 1/3 advantage.
>
> Looking at cake_flowblind_noecn, BBR1 and BBR4 just kills both CUBIC
> flows. Same with PIE.
>
> So it seems my intuition was wrong, at least for these scenarios. It
> wasn't CUBIC that would kill BBR, it's the other way around. Great to
> have testing tools! Thanks Flent!
>
wow. I was wondering how BBR ever relinquished bandwidth. "With some
difficulty".
Though not in the second screenshot where there's a strict AQM (cake
flowblind). Then it shares with other BBRs while never letting CUBIC
get established. (I think if you look at drops, they'd be higher than
what 4xCUBIC converges to on that link. See
https://groups.google.com/forum/#!msg/bbr-dev/EXgiWxHvaEU/wl3Q39svAAAJ ).
[-- Attachment #2: Type: text/html, Size: 3923 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-09-21 13:49 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAMZR1YDj_QB3sE1VXbn0GCUYF072x-E2DFVXSAKSgW1uTja_OQ@mail.gmail.com>
[not found] ` <CAA93jw4+oTz00qvScSOyDmQyeEfhoYs__cB_1LJ-AZDHymUfkQ@mail.gmail.com>
[not found] ` <92a6ae25-530f-1837-addd-8a9ef07dd022@gmail.com>
[not found] ` <CAA93jw5hR7bjxYNgzWQOSAtKWG+0oUGLFk+dmvmSMgkx5YTzuA@mail.gmail.com>
[not found] ` <alpine.DEB.2.02.1609211210280.1477@uplift.swm.pp.se>
2016-09-21 11:19 ` [Bloat] [Cerowrt-devel] BBR congestion control algorithm for TCP in net-next Dave Taht
2016-09-21 11:32 ` Mikael Abrahamsson
2016-09-21 12:40 ` Mikael Abrahamsson
2016-09-21 13:49 ` Alan Jenkins
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox