Re: [Bloat] Animation

Re: [Bloat] Animation

[Daniel Brooks]

I saw the video that Richard posted, and I agree that it needs to be
accompanied by an explanation. Text is the simplest way to go, and
subtitles can be abused for this purpose, so I converted the video to a
WebM file and put it up on my webserver at
http://db48x.net/bufferbloat/. I then used universalsubtitles.org to add
subtitles to it; You can check them out at
http://universalsubtitles.org/en/videos/qstl1K7mdGuq/. The subtitles are
also supposed to show up on db48x.net but for the moment there's a snag.

Anyone can edit these subtitles right in the browser, so I encourage
anyone who has an idea for improvements to simply do so. Consider the
existing subtitles to be a first draft. Similarly, if there is any other
information we can sync to the video in order to produce a better
presentation then suggest it here in the list and I'll implement
it. Syncing text, images, whole webpages, and audio to the video is
practically trivial, so don't be shy. If someone wants to record a
spoken explanation that would be wonderful.

db48x

Re: [Bloat] Animation

[Jim Gettys]

On 03/01/2011 02:22 AM, Daniel Brooks wrote:
> I saw the video that Richard posted, and I agree that it needs to be
> accompanied by an explanation. Text is the simplest way to go, and
> subtitles can be abused for this purpose, so I converted the video to a
> WebM file and put it up on my webserver at
> http://db48x.net/bufferbloat/. I then used universalsubtitles.org to add
> subtitles to it; You can check them out at
> http://universalsubtitles.org/en/videos/qstl1K7mdGuq/. The subtitles are
> also supposed to show up on db48x.net but for the moment there's a snag.
>
> Anyone can edit these subtitles right in the browser, so I encourage
> anyone who has an idea for improvements to simply do so. Consider the
> existing subtitles to be a first draft. Similarly, if there is any other
> information we can sync to the video in order to produce a better
> presentation then suggest it here in the list and I'll implement
> it. Syncing text, images, whole webpages, and audio to the video is
> practically trivial, so don't be shy. If someone wants to record a
> spoken explanation that would be wonderful.
>
> db48x

The big problem I see (with the original simulation) is that it chooses 
a very unlikely situation to simulate; a number of hops with identical 
throughput, and the queues therefore growing at multiple hops.

Almost always, there will be (for a simple path such as that being 
simulated in this case) a single hop which is the bottleneck, and the 
queues grow on either side of that hop.

Given the original NS2 script, it should be easy to re-run with 
something much more "typical".

Another point we need to get across is that the bottleneck moves and 
therefore where the queues grow: e.g. when your 802.11 bandwidth drops 
below that of your broadband bandwidth and it shifts to your host and 
home router.

Anyone care to take the original ns2 script and regenerate video for the 
typical cases we know happen most often?
			- Jim

[Bloat] Overview modifications

[Eric Raymond]

I'm trawling through the mailing list archives for bits to improve the
overview document with.

jg wrote:
>Note that lossy wireless networks behave much better than "clean" 
>variable bandwidth wireless networks; packet drops due to such losses at 
>least cause TCP to back off sometimes....

This is an important enough point to be in the overview. 

Change in progress -- append to the "Hating" paragraph the following
sentence: "Lossy networks such as wireless actually show less chaotic
behavior under load than clean ones."  Is this correct and adequate?

Ah, now I've found jg's original reply on the overview.

jg wrote:
>1) the latency-speed connection needs to be emphasised; along with 
>bandwidth !=speed. You can at least start to attack the conflation of 
>"speed" and "bandwidth" that is in the market.

*Excellent* point.

New 'graph, after the one on DNS et al going kerflooey.

    The way these latency-sensitive services degrade illustrates a
    general point.  From a user point of view, perceived speed of the
    network is much more a function of latency (time to get a
    response) than of bandwidth (ability to bulk-ship). Thus,
    bufferbloat hammers the performance characteristic users care
    about most.

jg:
>To use the highway analogy, only a few areas have carpool lanes, and 
>most of the highway is still jammed.  Once the carpool lanes are gone 
>(you get to another part of the network), you're still stuck behind 
>miles of traffic.

Yup, I'll add this to the QoS section.

Jim criticized "How Hard Is It?"  Rewritten section looks like this:

----------------------------------------------------------------------
== How Hard Is It? ==

We started by asserting that bufferbloat is easy to fix.  First
we'll lay out the both the reasons for optimism, then the problems:

First, it's easy to detect once you understand it - and verifying
that you've fixed it is easy, too.

Second, the fixes are cheap and give direct benefits as soon as
they're applied.  You don't have to wait for other people to fix
bufferbloat in their devices to improve the performance of your own.

Third, you'll usually only have to fix it once per device; continual
tuning isn't necessary.

Fourth (and importantly!), trying to fix bufferbloat won't significantly
increase your risk of a network failure. If you fumble the first time,
it's reversible.

The hard problems are these:

First, we don't yet know how to write good buffer-management rules for
networks with variable bandwidth.  This is a research area, though not
an especially difficult one: good results seem likely in 12 to 36
months.

Second, a lot of routers (especially consumer-grade ones) are difficult
enough to upgrade firmware on that they will need to be replaced. Since
the IPv6 transition is bearing down on us, however, it may be possible
to fold our upgrade wave into that one.
----------------------------------------------------------------------

dtaht wrote:
>Both ARE useful, but can be addressed in order of reducing unmanaged
>buffers, applying AQM, and then QoS. 

That priority list needs to be made explicit in the overview.  I will now do so.
End of section:

    Explicit QoS may a place in our overall strategy for network performance,
    but the right order to do things is this: first reduce unmanaged buffers, 
    then get buffer queue management right, then implement QoS on top of that.

richard wrote:
>Might add a note that if equipment does need to be changed out, isn't it
>nice that we also have to change it out for the IPV4->IPV6 problem too
>and/or that we're hoping the device manufacturers will address the
>problem in the IPV6 equipment roll out.

Done, as you can see above.

richard:
>Here in Canada the metered billing is very high visibility at the
>moment. Would love to be able to point to this from some of the
>discussion areas ASAP :)

This *doesn't* go in the overview.  Two reasons:

Here, I don't want to be topical in a way likely to go stale rapidly.

Also, dtaht and jg are right that there's a political hazard in
getting too far into the whole tiered-billing/usage-metering issue.  I
don't think we can or should avoid it entirely; the fact that fixing
bufferbloat may eliminate the technical necessity is simply too
important a fact to elide.  On the other hand, we need to be very
careful and diplomatic and not pick any fights about this.

That completes my mining of the back mail.  

I'll expect another round of responses to this post, then I'll run
an edit for conciseness and length, then I'll start transplanting 
the overview onto the wiki. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Certainly one of the chief guarantees of freedom under any government,
no matter how popular and respected, is the right of the citizens to
keep and bear arms.  [...] the right of the citizens to bear arms is
just one guarantee against arbitrary government and one more safeguard
against a tyranny which now appears remote in America, but which
historically has proved to be always possible.
        -- Hubert H. Humphrey, 1960

Re: [Bloat] Overview modifications

[Jim Gettys]

On 02/06/2011 09:39 AM, Eric Raymond wrote:
> I'm trawling through the mailing list archives for bits to improve the
> overview document with.
>
> jg wrote:
>> Note that lossy wireless networks behave much better than "clean"
>> variable bandwidth wireless networks; packet drops due to such losses at
>> least cause TCP to back off sometimes....
>
> This is an important enough point to be in the overview.
>
> Change in progress -- append to the "Hating" paragraph the following
> sentence: "Lossy networks such as wireless actually show less chaotic
> behavior under load than clean ones."  Is this correct and adequate?

It's not chaotic behaviour.  In fact, it is much more worrying: it is 
periodic (oscillatory) behaviour.  Chaos is good, in this case.

My nightmare, is that as traffic shifts over more and more to saturated 
links as XP retires, we end up with self synchronising behaviour on a 
local, regional or global scale, and havoc ensues, and parts/all of the 
Internet stop working. Whether these fears are justified, I do not know.

Think: we may be a column of soldiers in cadence approaching a bridge...

>
> Ah, now I've found jg's original reply on the overview.
>
> jg wrote:
>> 1) the latency-speed connection needs to be emphasised; along with
>> bandwidth !=speed. You can at least start to attack the conflation of
>> "speed" and "bandwidth" that is in the market.
>
> *Excellent* point.
>
> New 'graph, after the one on DNS et al going kerflooey.
>
>      The way these latency-sensitive services degrade illustrates a
>      general point.  From a user point of view, perceived speed of the
>      network is much more a function of latency (time to get a
>      response) than of bandwidth (ability to bulk-ship). Thus,
>      bufferbloat hammers the performance characteristic users care
>      about most.
>
> jg:
>> To use the highway analogy, only a few areas have carpool lanes, and
>> most of the highway is still jammed.  Once the carpool lanes are gone
>> (you get to another part of the network), you're still stuck behind
>> miles of traffic.
>
> Yup, I'll add this to the QoS section.
>
> Jim criticized "How Hard Is It?"  Rewritten section looks like this:
>
> ----------------------------------------------------------------------
> == How Hard Is It? ==
>
> We started by asserting that bufferbloat is easy to fix.  First
> we'll lay out the both the reasons for optimism, then the problems:
>
> First, it's easy to detect once you understand it - and verifying
> that you've fixed it is easy, too.
>
> Second, the fixes are cheap and give direct benefits as soon as
> they're applied.  You don't have to wait for other people to fix
> bufferbloat in their devices to improve the performance of your own.
>
> Third, you'll usually only have to fix it once per device; continual
> tuning isn't necessary.
>
> Fourth (and importantly!), trying to fix bufferbloat won't significantly
> increase your risk of a network failure. If you fumble the first time,
> it's reversible.
>
> The hard problems are these:
>
> First, we don't yet know how to write good buffer-management rules for
> networks with variable bandwidth.  This is a research area, though not
> an especially difficult one: good results seem likely in 12 to 36
> months.
>
> Second, a lot of routers (especially consumer-grade ones) are difficult
> enough to upgrade firmware on that they will need to be replaced. Since
> the IPv6 transition is bearing down on us, however, it may be possible
> to fold our upgrade wave into that one.
> ----------------------------------------------------------------------

There is an additional point: understanding bufferbloat may allow you to 
avoid bufferbloat suffering immediately.  Example:

Upgrade your wireless access to 802.11N, so that the bufferbloat stays 
in the broadband hop where you may have already mitigated it, rather 
than fighting bufferbloat in the 802.11 hop, where we don't have as 
effective short term mitigations.

   or:

Wander to a part of your house where the 802.11 "goodput" is above that 
of the broadband link, for the same reasons.

Won't help you when doing local copy/backup to your file server; then 
you really do need to start working on your laptop/router.  Unless you 
plug into ethernet, of course, again, as a short term mitigation.


>
> dtaht wrote:
>> Both ARE useful, but can be addressed in order of reducing unmanaged
>> buffers, applying AQM, and then QoS.
>
> That priority list needs to be made explicit in the overview.  I will now do so.
> End of section:
>
>      Explicit QoS may a place in our overall strategy for network performance,
>      but the right order to do things is this: first reduce unmanaged buffers,
>      then get buffer queue management right, then implement QoS on top of that.
>
> richard wrote:
>> Might add a note that if equipment does need to be changed out, isn't it
>> nice that we also have to change it out for the IPV4->IPV6 problem too
>> and/or that we're hoping the device manufacturers will address the
>> problem in the IPV6 equipment roll out.
>
> Done, as you can see above.
>
> richard:
>> Here in Canada the metered billing is very high visibility at the
>> moment. Would love to be able to point to this from some of the
>> discussion areas ASAP :)
>
> This *doesn't* go in the overview.  Two reasons:
>
> Here, I don't want to be topical in a way likely to go stale rapidly.
>
> Also, dtaht and jg are right that there's a political hazard in
> getting too far into the whole tiered-billing/usage-metering issue.  I
> don't think we can or should avoid it entirely; the fact that fixing
> bufferbloat may eliminate the technical necessity is simply too
> important a fact to elide.  On the other hand, we need to be very
> careful and diplomatic and not pick any fights about this.
>
> That completes my mining of the back mail.

There is tons of mining in the replies to my blog to do.

>
> I'll expect another round of responses to this post, then I'll run
> an edit for conciseness and length, then I'll start transplanting
> the overview onto the wiki.

Re: [Bloat] Overview modifications

[Eric Raymond]

Jim Gettys <jg@freedesktop.org>:
> >Change in progress -- append to the "Hating" paragraph the following
> >sentence: "Lossy networks such as wireless actually show less chaotic
> >behavior under load than clean ones."  Is this correct and adequate?
> 
> It's not chaotic behaviour.  In fact, it is much more worrying: it
> is periodic (oscillatory) behaviour.  Chaos is good, in this case.

Dave also says my take is wrong and is promising to suggest a correction.
I have enough other stuff to do that I'll wait on that.
  
> My nightmare, is that as traffic shifts over more and more to
> saturated links as XP retires, we end up with self synchronising
> behaviour on a local, regional or global scale, and havoc ensues,
> and parts/all of the Internet stop working. Whether these fears are
> justified, I do not know.
> 
> Think: we may be a column of soldiers in cadence approaching a bridge...

New graphs at the end of "From Highway to Network":

    We also have some worries about the future.  For various reasons
    (including the gradual retirement of Windows XP) more and more
    Internet traffic is now running over saturated links.  In this new
    environment, we think there is a possibility that bufferbloat cascades
    and defects in management strategies might produce self-synchronising
    behaviour in network traffic - packet floods and network resonance on
    a local, regional or global scale that could be a greater threat to
    the Internet than the congestion-driven near-collapse of the NSF
    backbone in 1986.

    This is a classic "black swan" situation in Nassim Taleb's sense; in
    today's Internet-dependent economy there is a potential for nearly
    inacalculable havoc in the worst case, but we don't even know in
    principle how to estimate the overall risk.  Bufferbloat mitigation
    might keep us out of some very serious trouble, and is worth pursuing
    on those grounds alone.

> There is an additional point: understanding bufferbloat may allow
> you to avoid bufferbloat suffering immediately.  Example:

I think we're already noting that point fixes can be applied quickly
an effectively.

> There is tons of mining in the replies to my blog to do.

I'll look, but that sounds like it might be getting into too much detail 
for the overview.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: [Bloat] Overview modifications

[Jim Gettys]

On 02/06/2011 03:20 PM, Eric Raymond wrote:
> Jim Gettys<jg@freedesktop.org>:
>>> Change in progress -- append to the "Hating" paragraph the following
>>> sentence: "Lossy networks such as wireless actually show less chaotic
>>> behavior under load than clean ones."  Is this correct and adequate?
>>
>> It's not chaotic behaviour.  In fact, it is much more worrying: it
>> is periodic (oscillatory) behaviour.  Chaos is good, in this case.
>
> Dave also says my take is wrong and is promising to suggest a correction.
> I have enough other stuff to do that I'll wait on that.
>
>> My nightmare, is that as traffic shifts over more and more to
>> saturated links as XP retires, we end up with self synchronising
>> behaviour on a local, regional or global scale, and havoc ensues,
>> and parts/all of the Internet stop working. Whether these fears are
>> justified, I do not know.
>>
>> Think: we may be a column of soldiers in cadence approaching a bridge...
>
> New graphs at the end of "From Highway to Network":
>
>      We also have some worries about the future.  For various reasons
>      (including the gradual retirement of Windows XP) more and more
>      Internet traffic is now running over saturated links.  In this new
>      environment, we think there is a possibility that bufferbloat cascades
>      and defects in management strategies might produce self-synchronising
>      behaviour in network traffic - packet floods and network resonance on
>      a local, regional or global scale that could be a greater threat to
>      the Internet than the congestion-driven near-collapse of the NSF
>      backbone in 1986.

It's not just bufferbloat: a number of network technologies are bunching 
up packets and injecting them into the Internet with periodic bursts. 
Unfortunately, I don't have good references to this; I gather this is 
true of both wireless and wired technologies.

>
>      This is a classic "black swan" situation in Nassim Taleb's sense; in
>      today's Internet-dependent economy there is a potential for nearly
>      inacalculable havoc in the worst case, but we don't even know in
>      principle how to estimate the overall risk.  Bufferbloat mitigation
>      might keep us out of some very serious trouble, and is worth pursuing
>      on those grounds alone.

It's actually a general fear of any periodic behaviour; I'm just spooked 
to see it in such long period TCP traffic.

Van warned me about time based congestion phenomena in general.

[Bloat] Animation

[Richard Scheffenegger]

Hi,

I've created a simplistic simulation of bufferbloat with ns2-2.34.

http://www.bufferbloat.net/attachments/15/nam00000.avi


Since bandwidth x latency x time-per-frame are scale invariant (if each 
frame represents less time, the simulatied bandwidth goes up / latency goes 
down), I choose parameters which fit best with good graphical results.

Perhaps one should add some line-graphs next to the sender, showing the 
evolution of the congestion window, or the bandwidth utilized.

The simulated tcp stack was an older variant of Linux.

The animation is released under CC 3.0 - I explicitly invite you to ie. 
voice over the animation, use parts of it (ie. time-lapse for beverity) etc.

Best regards,
   Richard


----- Original Message ----- 
From: "Jim Gettys" <jg@freedesktop.org>
To: <esr@thyrsus.com>
Cc: "Eric Raymond" <esr@snark.thyrsus.com>; <bloat@lists.bufferbloat.net>
Sent: Monday, February 07, 2011 8:56 PM
Subject: Re: [Bloat] Overview modifications


> On 02/06/2011 03:20 PM, Eric Raymond wrote:
>> Jim Gettys<jg@freedesktop.org>:
>>>> Change in progress -- append to the "Hating" paragraph the following
>>>> sentence: "Lossy networks such as wireless actually show less chaotic
>>>> behavior under load than clean ones."  Is this correct and adequate?
>>>
>>> It's not chaotic behaviour.  In fact, it is much more worrying: it
>>> is periodic (oscillatory) behaviour.  Chaos is good, in this case.
>>
>> Dave also says my take is wrong and is promising to suggest a correction.
>> I have enough other stuff to do that I'll wait on that.
>>
>>> My nightmare, is that as traffic shifts over more and more to
>>> saturated links as XP retires, we end up with self synchronising
>>> behaviour on a local, regional or global scale, and havoc ensues,
>>> and parts/all of the Internet stop working. Whether these fears are
>>> justified, I do not know.
>>>
>>> Think: we may be a column of soldiers in cadence approaching a bridge...
>>
>> New graphs at the end of "From Highway to Network":
>>
>>      We also have some worries about the future.  For various reasons
>>      (including the gradual retirement of Windows XP) more and more
>>      Internet traffic is now running over saturated links.  In this new
>>      environment, we think there is a possibility that bufferbloat 
>> cascades
>>      and defects in management strategies might produce 
>> self-synchronising
>>      behaviour in network traffic - packet floods and network resonance 
>> on
>>      a local, regional or global scale that could be a greater threat to
>>      the Internet than the congestion-driven near-collapse of the NSF
>>      backbone in 1986.
>
> It's not just bufferbloat: a number of network technologies are bunching 
> up packets and injecting them into the Internet with periodic bursts. 
> Unfortunately, I don't have good references to this; I gather this is true 
> of both wireless and wired technologies.
>
>>
>>      This is a classic "black swan" situation in Nassim Taleb's sense; in
>>      today's Internet-dependent economy there is a potential for nearly
>>      inacalculable havoc in the worst case, but we don't even know in
>>      principle how to estimate the overall risk.  Bufferbloat mitigation
>>      might keep us out of some very serious trouble, and is worth 
>> pursuing
>>      on those grounds alone.
>
> It's actually a general fear of any periodic behaviour; I'm just spooked 
> to see it in such long period TCP traffic.
>
> Van warned me about time based congestion phenomena in general.
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat 

Re: [Bloat] Animation

[John W. Linville]

On Tue, Feb 08, 2011 at 07:54:57AM +0100, Richard Scheffenegger wrote:

> I've created a simplistic simulation of bufferbloat with ns2-2.34.
> 
> http://www.bufferbloat.net/attachments/15/nam00000.avi

Looks good -- I _think_ I almost understand what I'm seeing... :-)

> The animation is released under CC 3.0 - I explicitly invite you to
> ie. voice over the animation, use parts of it (ie. time-lapse for
> beverity) etc.

I hope someone does this.  If nothing else, maybe even just an
accompanying script or some text overlay during the animation would
be nice.

Hopefully you/we can clarify the issue for knuckle-dragging "layer 2"
guys like me that have been trained into the "don't drop packets"
mentality.  Some of the recent writings have been good for this too.
But a lot of the posts on this list and the blogs have amounted to a
URL with a comment like "the problem here is obvious" -- leaving me
to squint and scratch my head...

Given my associations with Linux and wireless, I am interested in
working on this problem.  Still, I (and presumably others) may need
some hand-holding!

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.

Re: [Bloat] Animation

[Eric Raymond]

John W. Linville <linville@tuxdriver.com>:
> On Tue, Feb 08, 2011 at 07:54:57AM +0100, Richard Scheffenegger wrote:
> 
> > I've created a simplistic simulation of bufferbloat with ns2-2.34.
> > 
> > http://www.bufferbloat.net/attachments/15/nam00000.avi
> 
> Looks good -- I _think_ I almost understand what I'm seeing... :-)

That's my reaction, too.  (a) it needs a narrative voicover, badly, and (b) 
the graphics need to be less abstract.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>