Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Richard Brown]

Folks,

I gave the talk to the local Linux User Group on Thursday, and it went really well. Two people came up to me after the talk and said, in effect, "You know, I think I've seen this. But I've always blamed something else." Their experience:

- Attempting to Skype with a bunch of web browser tabs open gives bad results. Closing the tabs made things better. (They been blaming the browser for "using too much memory". Now it's possible to think that it's a network problem.)

- Another person reported that his network connection (wireless ISP, two hops to a wired network) seemed to work OK as long as his household was mostly downloading. But uploading much of anything really made things bad.

I posted the slides at http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf

Rich

PS I've updated the CeroWrt site to include links to a bunch of relevant videos. (http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos) Please let me know if there are others that we should point to.

On Nov 25, 2012, at 8:11 PM, Richard E. Brown <rbrown@intermapper.com> wrote:

> Folks,
> 
> I am planning to give a talk about Bufferbloat to the local Linux User Group next week (http://dlslug.org). All this traffic on the list is fantastic, because it gives me a lot of background on the current state of bufferbloat. I've pulled together a bunch of general questions about CeroWrt that I would like to be able to cover if they come up:
> 
> - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from mid-September? (My router is using this build - r33460.) 
> 
> - I see the "QoS" item in the Network tab of the web GUI. Is this important for Sugarland? Or does some other router configuration take care of this now?
> 
> - What's the relationship between the QoS GUI item above and the debloat.sh and simple_qos.sh scripts that have been mentioned on this list? What's the best practice here for getting a router up and running?
> 
> - I can see how the CeroWrt de-bloating algorithms help protect against bad latency when I'm *uploading* big files. I'm not sure whether using CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files, though. What can I say about this?
> 
> - I believe the default DNS server in Sugarland is dnsmasq, not bind. Is DNSSEC enabled by default? Also: there's a report (Bug #411) that says that DNS is leaking internal names to the outside world. What's the best advice for closing this? ("list notinterface 'ge00'" is one recommendation…)
> 
> - I've been assembling information about the various de-bloating techniques implemented in CeroWrt. It seems that Infoblox has recently reorganized their blogs, and the links published earlier this week have all broken. Here are updates:
> 
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> http://www.infoblox.com/community/blog/router-buffer-tuning
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports
> 
> My plan is to give a little of the science behind bufferbloat mitigation and also put in a plug for CeroWrt. Any topics I haven't already mentioned that I should? Thanks!
> 
> Rich Brown
> Hanover, NH USA

Re: [Bloat] [Cerowrt-devel] Bufferbloat at LUG talk - Meeting Report

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 4723 bytes --]

Excellent job Richard! Those slides are very clean and informative and you
got fantastic real life user reports!
Point #1 is very common: lots of behind-the-scenes javascript, buffering,
asynchronous requests, facebook chat box and updates, etc.

I was trying to make a mock conversation for the purpose of providing a
story backing up the debloating efforts so that end users realize better
what's going on. Please, guys, have a look and comment:
https://soltysiak.com/wiki/index.php/BB_dialog

Part 1 is an intro, also touching on tiered ISP services. Part 2 would be
what bufferbloat is all about. Part 3 is an outro to have the users have a
take-home message, also touching on DPI and other evil stuff ISPs do trying
to workaround the issues.

You can edit that wiki.

I couldn't post it on bufferbloat.net wiki because I don't seem to have
privilege to create new pages so I setup my own.

Regards,
Maciej


On Sun, Dec 9, 2012 at 5:56 PM, Richard Brown
<richard.e.brown@dartware.com>wrote:

> Folks,
>
> I gave the talk to the local Linux User Group on Thursday, and it went
> really well. Two people came up to me after the talk and said, in effect,
> "You know, I think I've seen this. But I've always blamed something else."
> Their experience:
>
> - Attempting to Skype with a bunch of web browser tabs open gives bad
> results. Closing the tabs made things better. (They been blaming the
> browser for "using too much memory". Now it's possible to think that it's a
> network problem.)
>
> - Another person reported that his network connection (wireless ISP, two
> hops to a wired network) seemed to work OK as long as his household was
> mostly downloading. But uploading much of anything really made things bad.
>
> I posted the slides at
> http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf
>
> Rich
>
> PS I've updated the CeroWrt site to include links to a bunch of relevant
> videos. (http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos)
> Please let me know if there are others that we should point to.
>
> On Nov 25, 2012, at 8:11 PM, Richard E. Brown <rbrown@intermapper.com>
> wrote:
>
> > Folks,
> >
> > I am planning to give a talk about Bufferbloat to the local Linux User
> Group next week (http://dlslug.org). All this traffic on the list is
> fantastic, because it gives me a lot of background on the current state of
> bufferbloat. I've pulled together a bunch of general questions about
> CeroWrt that I would like to be able to cover if they come up:
> >
> > - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from
> mid-September? (My router is using this build - r33460.)
> >
> > - I see the "QoS" item in the Network tab of the web GUI. Is this
> important for Sugarland? Or does some other router configuration take care
> of this now?
> >
> > - What's the relationship between the QoS GUI item above and the
> debloat.sh and simple_qos.sh scripts that have been mentioned on this list?
> What's the best practice here for getting a router up and running?
> >
> > - I can see how the CeroWrt de-bloating algorithms help protect against
> bad latency when I'm *uploading* big files. I'm not sure whether using
> CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files,
> though. What can I say about this?
> >
> > - I believe the default DNS server in Sugarland is dnsmasq, not bind. Is
> DNSSEC enabled by default? Also: there's a report (Bug #411) that says that
> DNS is leaking internal names to the outside world. What's the best advice
> for closing this? ("list notinterface 'ge00'" is one recommendation…)
> >
> > - I've been assembling information about the various de-bloating
> techniques implemented in CeroWrt. It seems that Infoblox has recently
> reorganized their blogs, and the links published earlier this week have all
> broken. Here are updates:
> >
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> > http://www.infoblox.com/community/blog/router-buffer-tuning
> >
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports
> >
> > My plan is to give a little of the science behind bufferbloat mitigation
> and also put in a plug for CeroWrt. Any topics I haven't already mentioned
> that I should? Thanks!
> >
> > Rich Brown
> > Hanover, NH USA
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 6200 bytes --]

Re: [Bloat] [Cerowrt-devel] Bufferbloat at LUG talk - Meeting Report

[Richard Brown]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

Hello Maciej,

Thanks for the kind words about my presentation.

Your imagined dialog is spot on. It's a good way to let people recognize this common situation, and what the fix might be.

Best regards,

Rich

On Dec 9, 2012, at 12:32 PM, Maciej Soltysiak <maciej@soltysiak.com<mailto:maciej@soltysiak.com>> wrote:

Excellent job Richard! Those slides are very clean and informative and you got fantastic real life user reports!
Point #1 is very common: lots of behind-the-scenes javascript, buffering, asynchronous requests, facebook chat box and updates, etc.

I was trying to make a mock conversation for the purpose of providing a story backing up the debloating efforts so that end users realize better what's going on. Please, guys, have a look and comment:
https://soltysiak.com/wiki/index.php/BB_dialog

Part 1 is an intro, also touching on tiered ISP services. Part 2 would be what bufferbloat is all about. Part 3 is an outro to have the users have a take-home message, also touching on DPI and other evil stuff ISPs do trying to workaround the issues.

You can edit that wiki.

I couldn't post it on bufferbloat.net<http://bufferbloat.net/> wiki because I don't seem to have privilege to create new pages so I setup my own.

Regards,
Maciej


[-- Attachment #2: Type: text/html, Size: 1906 bytes --]

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Oliver Hohlfeld]

> - Attempting to Skype with a bunch of web browser tabs open gives bad results. Closing the tabs made things better. (They been blaming the browser for "using too much memory". Now it's possible to think that it's a network problem.)

Sorry to be nitpicking on this, but what evidence / indications tells
this is a networking problem? Why would the pages in the open tabs
constantly sent large bulks of data that fill up the buffers? While
there is certainly asynchronous communication over HTTP (Ajax etc.), I'm
talking about traffic demands that can interfere with VoIP. This seems
highly unlikely. Correlating every possible problem with bufferbloat
without further investigation doesn't make any sense to me.

> - Another person reported that his network connection (wireless ISP, two hops to a wired network) seemed to work OK as long as his household was mostly downloading. But uploading much of anything really made things bad.

This is a known fact. Even Gettys reported this issue in his first paper.

> I posted the slides at http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf

Thanks for sharing this!

A few comments:

- Slide 12 reads wired to me. It assumes that the entire main memory is
allocated to packet memory. How will the home router then run an OS and
any sort of application on top of it? The argument is flawed.

- Codel and SFQ are not the only solutions to improve bad FIFO queues.
Simple QoS mechanisms will do the trick as well (briefly mentioned on
slide 12).

- Slide 14 presents a long list of devices that lack of implementations.
While the problem has been demonstrated for the first half of the list,
I'd be interested to see if Codel could improve DSLAMs and cable
headends, that, to the best of my knowledge, hardly queue traffic and do
not induce significant queuing delays.

Oliver

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Neil Davies]

Oliver 

Every TCP transfer (which may be collection of web page accesses) that reaches an instantaneous packet rate in-excess of the capacity of the most constrained network element on the path causes such problems. 

This is a measurable phenomena, it is equipment/configuration dependant - but the non-stationarity introduced is of the order of several 100ms to seconds - we've got measurements that cover that range.

Such phenomena are a natural emergent consequence of the statistical packet sharing of *any* resource.

Neil

On 12 Dec 2012, at 14:26, Oliver Hohlfeld <oliver@net.t-labs.tu-berlin.de> wrote:

>> - Attempting to Skype with a bunch of web browser tabs open gives bad results. Closing the tabs made things better. (They been blaming the browser for "using too much memory". Now it's possible to think that it's a network problem.)
> 
> Sorry to be nitpicking on this, but what evidence / indications tells
> this is a networking problem? Why would the pages in the open tabs
> constantly sent large bulks of data that fill up the buffers? While
> there is certainly asynchronous communication over HTTP (Ajax etc.), I'm
> talking about traffic demands that can interfere with VoIP. This seems
> highly unlikely. Correlating every possible problem with bufferbloat
> without further investigation doesn't make any sense to me.
> 
>> - Another person reported that his network connection (wireless ISP, two hops to a wired network) seemed to work OK as long as his household was mostly downloading. But uploading much of anything really made things bad.
> 
> This is a known fact. Even Gettys reported this issue in his first paper.
> 
>> I posted the slides at http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf
> 
> Thanks for sharing this!
> 
> A few comments:
> 
> - Slide 12 reads wired to me. It assumes that the entire main memory is
> allocated to packet memory. How will the home router then run an OS and
> any sort of application on top of it? The argument is flawed.
> 
> - Codel and SFQ are not the only solutions to improve bad FIFO queues.
> Simple QoS mechanisms will do the trick as well (briefly mentioned on
> slide 12).
> 
> - Slide 14 presents a long list of devices that lack of implementations.
> While the problem has been demonstrated for the first half of the list,
> I'd be interested to see if Codel could improve DSLAMs and cable
> headends, that, to the best of my knowledge, hardly queue traffic and do
> not induce significant queuing delays.
> 
> Oliver
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Oliver Hohlfeld]

> Every TCP transfer (which may be collection of web page accesses) that reaches an instantaneous packet rate in-excess of the capacity of the most constrained network element on the path causes such problems. 

Which problems? I'm missing the relation to my original post.

Oliver

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

I think we're talking about the "many tabs open" problem.

But this bothers me as well, actually. Lots of tabs shouldn't affect the network unless some of the pages in them are actively transferring data for some reason. I've seen lots of pages that have heavy animation or Javascript in them, but unless Facebook is even nastier than I was previously aware of, that doesn't really translate to high network traffic.

On the other hand, actively browsing the Web does produce traffic. That is entirely feasible and even useful to do during a Skype call.

- Jonathan Morton
On Dec 19, 2012 2:06 PM, "Oliver Hohlfeld" <oliver@net.t-labs.tu-berlin.de> wrote:
> Every TCP transfer (which may be collection of web page accesses) that reaches an instantaneous packet rate in-excess of the capacity of the most constrained network element on the path causes such problems.

Which problems? I'm missing the relation to my original post.

Oliver
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: Type: text/html, Size: 1639 bytes --]

Re: [Bloat] Bufferbloat at LUG talk - Meeting Report

[Oliver Hohlfeld]

On 12/19/2012 01:20 PM, Jonathan Morton wrote:
> I think we're talking about the "many tabs open" problem.
> 
> But this bothers me as well, actually. Lots of tabs shouldn't affect the
> network unless some of the pages in them are actively transferring data
> for some reason. I've seen lots of pages that have heavy animation or
> Javascript in them, but unless Facebook is even nastier than I was
> previously aware of, that doesn't really translate to high network traffic.

That was exactly the point I tried to stress in the post quoted by Neil;
An open tab does not automatically translate to network traffic, i.e.,
there is not necessarily causality between "open tab" and "Skype
problem". The pure existence of open tabs doesn't say anything; they
must trigger data transfers, there is no network impact otherwise. If
at all and how web transfers impair voice flows depends on a whole lot
of other aspects.

I often observe Javascript rendering in Firefox to fully load one CPU
core, which sometimes interferes with Skype calls on my systems. But
that's not a networking problem and not related to bufferbloat.

Oliver