<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi Matt,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 08.07.21 at 15:29 Matt Mathis wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAH56bmBT0jZ6+9TTnsaL9FW00LOUACPSSV432Rb3XkyMHq074Q@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">I think there is something missing from your
model. I just scanned your paper and noticed that you made no
mention of rounding errors, nor some details around the drain
phase timing, The implementation guarantees that the actual
average rate across the combined BW probe and drain is strictly
less than the measured maxBW and that the flight size comes back
down to minRTT*maxBW before returning to unity pacing gain. In
some sense these checks are redundant, but If you don't do them,
it is absolutely true that you are at risk of seeing divergent
behaviors.</div>
</blockquote>
Sure, most models abstract things away and so does our model leave
out<br>
some details, but it describes quite accurately what happens if
multiple <br>
BBRv1 flows are present. So the model was not only confirmed by our
<br>
own measurements, but also by many others who did BBRv1 experiments.
<blockquote type="cite"
cite="mid:CAH56bmBT0jZ6+9TTnsaL9FW00LOUACPSSV432Rb3XkyMHq074Q@mail.gmail.com">
<div dir="ltr">
<div>That said, it is also true that multi-stream BBR behavior
is quite complicated and needs more queue space than single
stream. This </div>
</div>
</blockquote>
Yes, mostly between 1bdp and 1.5bdp of queue space.<br>
<blockquote type="cite"
cite="mid:CAH56bmBT0jZ6+9TTnsaL9FW00LOUACPSSV432Rb3XkyMHq074Q@mail.gmail.com">
<div dir="ltr">
<div>complicates the story around the traditional workaround of
using multiple streams to compensate for Reno & CUBIC
lameness at larger scales (ordinary scales today).
Multi-stream does not help BBR throughput and raises the queue
occupancy, to the detriment of other users.</div>
<div><br>
</div>
<div>
<div>And yes, in my presentation, I described the core BBR
algorithms as a framework, which might be extended to
incorporate many additional algorithms if they provide
optimal control in some settings. And yes, several are
present in BBRv2.</div>
</div>
</div>
</blockquote>
<p>Ok, thanks for clarification.</p>
<p>Regards,<br>
Roland<br>
</p>
<blockquote type="cite"
cite="mid:CAH56bmBT0jZ6+9TTnsaL9FW00LOUACPSSV432Rb3XkyMHq074Q@mail.gmail.com">
<div dir="ltr">
<div>
<div> </div>
</div>
<div>
<div>Thanks,
<div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">--MM--<br>
The best way to predict the future is to create
it. - Alan Kay<br>
<br>
We must not tolerate intolerance;</div>
<div dir="ltr"> however our response must be
carefully measured: </div>
<div> too strong would be hypocritical
and risks spiraling out of control;</div>
<div> too weak risks being mistaken for
tacit approval.</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jul 8, 2021 at 4:24 AM
Bless, Roland (TM) <<a href="mailto:roland.bless@kit.edu"
moz-do-not-send="true">roland.bless@kit.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Matt,</div>
<div><br>
</div>
<div>On 08.07.21 at 00:38 Matt Mathis wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Actually BBR does have a window based
backup, which normally only comes into play during load
spikes and at very short RTTs. It defaults to
2*minRTT*maxBW, which is twice the steady state window
in it's normal paced mode.</div>
</blockquote>
<p>So yes, BBR follows option b), but I guess that you are
referring to BBRv1 here. <br>
We have shown in [1, Sec.III] that BBRv1 flows will <b>always</b>
run (conceptually) toward their above quoted inflight-cap
of<br>
2*minRTT*maxBW, if more than one BBR flow is present at
the bottleneck. So strictly speaking " which <b>normally
only</b> comes <br>
into play during load spikes and at very short RTTs" isn't
true for multiple BBRv1 flows.<br>
</p>
<p>It seems that in BBRv2 there are many more mechanisms
present <br>
that try to control the amount of inflight data more
tightly and the new "cap"<br>
is at 1.25 BDP.<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div>This is too large for short queue routers in the
Internet core, but it helps a lot with cross traffic
on large queue edge routers.<br>
</div>
</div>
</blockquote>
<p>Best regards,<br>
Roland<br>
</p>
<p>[1] <a
href="https://ieeexplore.ieee.org/document/8117540"
target="_blank" moz-do-not-send="true">https://ieeexplore.ieee.org/document/8117540</a></p>
<blockquote type="cite"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jul 7, 2021 at
3:19 PM Bless, Roland (TM) <<a
href="mailto:roland.bless@kit.edu" target="_blank"
moz-do-not-send="true">roland.bless@kit.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>Hi Matt,<br>
<br>
[sorry for the late reply, overlooked this one]</div>
<div><br>
</div>
<div>please, see comments inline.<br>
</div>
<div><br>
</div>
<div>On 02.07.21 at 21:46 Matt Mathis via Bloat
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">The argument is absolutely correct
for Reno, CUBIC and all other self-clocked
protocols. One of the core assumptions in
Jacobson88, was that the clock for the entire
system comes from packets draining through the
bottleneck queue. In this world, the clock is
intrinsically brittle if the buffers are too
small. The drain time needs to be a substantial
fraction of the RTT.</div>
</blockquote>
I'd like to separate the functions here a bit:<br>
<p>1) "automatic pacing" by ACK clocking</p>
<p>2) congestion-window-based operation</p>
<p>I agree that the automatic pacing generated by
the ACK clock (function 1) is increasingly <br>
distorted these days and may consequently cause
micro bursts.<br>
This can be mitigated by using paced sending,
which I consider very useful. <br>
However, I consider abandoning the (congestion)
window-based approaches <br>
with ACK feedback (function 2) as harmful:<br>
a congestion window has an automatic
self-stabilizing property since the ACK feedback
reflects<br>
also the queuing delay and the congestion window
limits the amount of inflight data.<br>
In contrast, rate-based senders risk instability:
two senders in an M/D/1 setting, each sender
sending with 50%<br>
bottleneck rate in average, both using paced
sending at 120% of the average rate, suffice to
cause<br>
instability (queue grows unlimited).<br>
<br>
IMHO, two approaches seem to be useful:<br>
a) congestion-window-based operation with paced
sending<br>
b) rate-based/paced sending with limiting the
amount of inflight data<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>However, we have reached the point where we
need to discard that requirement. One of the
side points of BBR is that in many
environments it is cheaper to burn serving CPU
to pace into short queue networks than it is
to "right size" the network queues.</div>
<div><br>
</div>
<div>The fundamental problem with the old way is
that in some contexts the buffer memory has to
beat Moore's law, because to maintain constant
drain time the memory size and BW both have to
scale with the link (laser) BW.</div>
<div><br>
</div>
<div>See the slides I gave at the Stanford
Buffer Sizing workshop december 2019: <a
href="https://docs.google.com/presentation/d/1VyBlYQJqWvPuGnQpxW4S46asHMmiA-OeMbewxo_r3Cc/edit#slide=id.g791555f04c_0_5"
target="_blank" moz-do-not-send="true">Buffer
Sizing: Position Paper</a> </div>
<div><br>
</div>
</div>
</blockquote>
<p>Thanks for the pointer. I don't quite get the
point that the buffer must have a certain size to
keep the ACK clock stable:<br>
in case of an non application-limited sender, a
very small buffer suffices to let the ACK clock <br>
run steady. The large buffers were mainly required
for loss-based CCs to let the standing queue <br>
build up that keeps the bottleneck busy during
CWnd reduction after packet loss, thereby <br>
keeping the (bottleneck link) utilization high.<br>
</p>
<p>Regards,</p>
<p> Roland<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div>Note that we are talking about DC and
Internet core. At the edge, BW is low enough
where memory is relatively cheap. In some
sense BB came about because memory is too
cheap in these environments.</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Thanks,</div>
--MM--<br>
The best way to predict the
future is to create it. - Alan
Kay<br>
<br>
We must not tolerate
intolerance;</div>
<div dir="ltr"> however our
response must be carefully
measured: </div>
<div> too strong would
be hypocritical and risks
spiraling out of control;</div>
<div> too weak risks
being mistaken for tacit
approval.</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 2,
2021 at 9:59 AM Stephen Hemminger <<a
href="mailto:stephen@networkplumber.org"
target="_blank" moz-do-not-send="true">stephen@networkplumber.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">On Fri, 2
Jul 2021 09:42:24 -0700<br>
Dave Taht <<a
href="mailto:dave.taht@gmail.com"
target="_blank" moz-do-not-send="true">dave.taht@gmail.com</a>>
wrote:<br>
<br>
> "Debunking Bechtolsheim credibly would
get a lot of attention to the<br>
> bufferbloat cause, I suspect." - dpreed<br>
> <br>
> "Why Big Data Needs Big Buffer Switches"
-<br>
> <a
href="http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf</a><br>
> <br>
<br>
Also, a lot depends on the TCP congestion
control algorithm being used.<br>
They are using NewReno which only researchers
use in real life.<br>
<br>
Even TCP Cubic has gone through several
revisions. In my experience, the<br>
NS-2 models don't correlate well to real world
behavior.<br>
<br>
In real world tests, TCP Cubic will consume
any buffer it sees at a<br>
congested link. Maybe that is what they mean
by capture effect.<br>
<br>
There is also a weird oscillation effect with
multiple streams, where one<br>
flow will take the buffer, then see a packet
loss and back off, the<br>
other flow will take over the buffer until it
sees loss.<br>
<br>
_______________________________________________</blockquote>
</div>
<pre>_______________________________________________
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>