[Bloat] bufferbloat and the web service providers

[Bloat] bufferbloat and the web service providers

[Dave Taht]

I am not into politics. Really. A problem, though is getting the NN
people on both sides to stand down, because the real problem was
bufferbloat - and then work together to move forward, on an
engineering, rather than political basis.

So, incidentally, while I'm suddenly paying attention to politics,
while trying to find some way to get this engineering effort funded
for next year... I noticed a new lobbying group has come out of
stealth mode, representing the web apps people, google, facebook,
salesforce.com etc

http://internetassociation.org/

Anybody know these guys? I have long hoped to get these services aware
that they needed to help fix bufferbloat if they wanted their cloud
based businesses to work better.

IF I felt like expending money on a marketing campaign to explain our
issue, and get some funds towards fixing wifi in particular -

the script to the video would look like this (leveraging the old
"brain on drugs" meme)

* This is your cloud based business - showing a fast, normal load time
of some major website

* This is your cloud based business on bufferbloat - showing a loaded
webload go to hell of that site

* This is your cloud based business with bufferbloat fixed -

* Any questions?  - point back to campaign site

Then create a copy of that 30 second piece for every one of the web
service providers listed as supporting the above org. And as many
others as possible. And get it on youtube. And buy some ads.

I have no idea how much that would cost to produce professionally (?)
(at a cut above what jg has already done), but I'm certain it would do
some good, both in raising awareness and maybe gaining funding for
next year.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Bloat] bufferbloat and the web service providers

[Oliver Hohlfeld]

> (...) because the real problem was
> bufferbloat - and then work together to move forward, on an
> engineering, rather than political basis.
(...)
> I have long hoped to get these services aware
> that they needed to help fix bufferbloat if they wanted their cloud
> based businesses to work better.

Is there any evidence that they do suffer from bufferbloat? And if
so, how many of their customers are affected? 0.001%? 0.1%? 10%?
So, what is the economical extend of the problem for these services
providers?

Before thinking about scripts and massive lobbying campaigns, why don't
we answer the most basic questions first?

--Oliver

Re: [Bloat] bufferbloat and the web service providers

[Jonathan Morton]

On 19 Nov, 2012, at 2:32 pm, Oliver Hohlfeld wrote:

>> (...) because the real problem was
>> bufferbloat - and then work together to move forward, on an
>> engineering, rather than political basis.
> (...)
>> I have long hoped to get these services aware
>> that they needed to help fix bufferbloat if they wanted their cloud
>> based businesses to work better.
> 
> Is there any evidence that they do suffer from bufferbloat? And if
> so, how many of their customers are affected? 0.001%? 0.1%? 10%?
> So, what is the economical extend of the problem for these services
> providers?

I see many websites whose bottleneck is the database containing their content, not the physical uplink.  They can easily be identified by the way they serve up database connection errors when under unusually heavy load.  For example, http://www.robertsspaceindustries.com/ about 12 hours ago, as they pushed through about $1M of pledges within 24 hours.  The Raspberry Pi site deliberately switched to a lightweight static page for their launch event back in February, thereby staying up while two major suppliers' sites (both operating very heavy e-commerce designs) crashed hard under the load.

Databases get used by a lot of websites these days, either as part of a forum or e-commerce backend (entirely reasonable) or for serving up standard but frequently updated content - where "frequently updated" means several times a day, rather than several times a second.  In this latter case, it would make far more sense to periodically bake the content into static pages that can be served without hitting the database.  This in turn would free up DB capacity for things that really need it, such as the forum and e-commerce transactions.

Only *then* does bufferbloat start to be relevant.

Even so, I am also constantly amazed by the lack of performance of most databases under a heavy read-mostly load.  A lot of it could doubtless be solved by tuning, but very few organisations seem to have the competence to do this.  A lot of it probably has to do with using general-purpose databases for unsuitable purposes - sure, you can store 19KB of text in a varchar field, and a quarter-megabyte image too, but that doesn't mean you should do s in a high-performance application.

There are, of course, many databases that are highly optimised, out of the box, for storing large text and image data on a read-mostly basis, even employing extensive caching to avoid unnecessary disk access.  They are commonly known as "filesystems".  :-)

 - Jonathan Morton

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

What are the heaviest (amount of elements, css, images, scripts, js bugs,
ad trackers, all that filth) websites out there?
I wonder if it'd be worthwhile to demo loading that in a standard bloated
environment and compare with debloated setup?

Maybe that could be a variant of your idea below but geared towards end
users?

Maciej

On Mon, Nov 19, 2012 at 10:28 AM, Dave Taht <dave.taht@gmail.com> wrote:

> I am not into politics. Really. A problem, though is getting the NN
> people on both sides to stand down, because the real problem was
> bufferbloat - and then work together to move forward, on an
> engineering, rather than political basis.
>
> So, incidentally, while I'm suddenly paying attention to politics,
> while trying to find some way to get this engineering effort funded
> for next year... I noticed a new lobbying group has come out of
> stealth mode, representing the web apps people, google, facebook,
> salesforce.com etc
>
> http://internetassociation.org/
>
> Anybody know these guys? I have long hoped to get these services aware
> that they needed to help fix bufferbloat if they wanted their cloud
> based businesses to work better.
>
> IF I felt like expending money on a marketing campaign to explain our
> issue, and get some funds towards fixing wifi in particular -
>
> the script to the video would look like this (leveraging the old
> "brain on drugs" meme)
>
> * This is your cloud based business - showing a fast, normal load time
> of some major website
>
> * This is your cloud based business on bufferbloat - showing a loaded
> webload go to hell of that site
>
> * This is your cloud based business with bufferbloat fixed -
>
> * Any questions?  - point back to campaign site
>
> Then create a copy of that 30 second piece for every one of the web
> service providers listed as supporting the above org. And as many
> others as possible. And get it on youtube. And buy some ads.
>
> I have no idea how much that would cost to produce professionally (?)
> (at a cut above what jg has already done), but I'm certain it would do
> some good, both in raising awareness and maybe gaining funding for
> next year.
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 3264 bytes --]

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Jonathan Morton]

On 9 Dec, 2012, at 6:14 pm, Maciej Soltysiak wrote:

> What are the heaviest (amount of elements, css, images, scripts, js bugs, ad trackers, all that filth) websites out there?

Amazon, RS Components, ICanHazCheezburger, AOL (shudder) - those spring to mind immediately.

 - Jonathan

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Steinar H. Gunderson]

On Sun, Dec 09, 2012 at 05:14:48PM +0100, Maciej Soltysiak wrote:
> What are the heaviest (amount of elements, css, images, scripts, js bugs,
> ad trackers, all that filth) websites out there?

www.arngren.net

Enjoy. :-)

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Jonathan Morton]

On 9 Dec, 2012, at 6:28 pm, Steinar H. Gunderson wrote:

> On Sun, Dec 09, 2012 at 05:14:48PM +0100, Maciej Soltysiak wrote:
>> What are the heaviest (amount of elements, css, images, scripts, js bugs,
>> ad trackers, all that filth) websites out there?
> 
> www.arngren.net
> 
> Enjoy. :-)

Yikes.  Is that their *entire* catalogue on the front page, or just the edited highlights?

 - Jonathan

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Steinar H. Gunderson]

On Sun, Dec 09, 2012 at 06:30:26PM +0200, Jonathan Morton wrote:
> Yikes.  Is that their *entire* catalogue on the front page, or just the
> edited highlights?

Just the edited highlights. The paper version is, like, 200 pages of the same.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] [Cerowrt-devel] bufferbloat and the web service providers

[Dave Taht]

The results of the rrul benchmark in conjunction with the chrome web
page benchmark tests in combination with the debloat and simple_qos.sh
script in combination with linux 3.6.9 are so frigging spectacular at
4Mbit uplinks and above, that I'm reluctant to publish them.

I've tested dozens of web sites ranging from google to xfinity at this
point with improvements in web page network load time ranging from 4
to 10x under otherwise saturating workloads. Another way to look at it
is that most web page loads I've tried degrade by less than 20% while
under the extreme stress of the rrul benchmark. (vs 10x worse or not
completing at all under pfifo_fast or short queues)

Others have had great difficulty in reproducing these results, so I'm
gradually trying to reduce the sources of experimental error and
improve the scripts and documentation and so on... Recently I got bit,
bad, by the recent default of the GRO offload to a couple network
drivers.... my current simple_qos script only uses ecn on ingress...

So I'm setting up 3 labs to nail all that down and also work on
artificially induced longer RTTs.... which is going to take some time.
I hope to make available a few netperf servers for public use fairly
soon (or line up some volunteers to do so?)

In the meantime I'd urge folk to track the development of the rrul and
related benchmarks, install their own netperf (netserver) servers, and
follow along...

https://github.com/tohojo/netperf-wrapper

Upload speeds below 4Mbit remain a bit problematic and wifi a nightmare.

The 3.6.9-5 release of cerowrt can run enough netperf servers to keep
100Mbit links busy but only just barely, and in some tests, we overrun
the switch's buffers, not fq_codel, so htb is the way to go, to
another box, through the router....

On Sun, Dec 9, 2012 at 5:14 PM, Maciej Soltysiak <maciej@soltysiak.com> wrote:
> What are the heaviest (amount of elements, css, images, scripts, js bugs, ad
> trackers, all that filth) websites out there?
> I wonder if it'd be worthwhile to demo loading that in a standard bloated
> environment and compare with debloated setup?
>
> Maybe that could be a variant of your idea below but geared towards end
> users?
>
> Maciej
>
> On Mon, Nov 19, 2012 at 10:28 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> I am not into politics. Really. A problem, though is getting the NN
>> people on both sides to stand down, because the real problem was
>> bufferbloat - and then work together to move forward, on an
>> engineering, rather than political basis.
>>
>> So, incidentally, while I'm suddenly paying attention to politics,
>> while trying to find some way to get this engineering effort funded
>> for next year... I noticed a new lobbying group has come out of
>> stealth mode, representing the web apps people, google, facebook,
>> salesforce.com etc
>>
>> http://internetassociation.org/
>>
>> Anybody know these guys? I have long hoped to get these services aware
>> that they needed to help fix bufferbloat if they wanted their cloud
>> based businesses to work better.
>>
>> IF I felt like expending money on a marketing campaign to explain our
>> issue, and get some funds towards fixing wifi in particular -
>>
>> the script to the video would look like this (leveraging the old
>> "brain on drugs" meme)
>>
>> * This is your cloud based business - showing a fast, normal load time
>> of some major website
>>
>> * This is your cloud based business on bufferbloat - showing a loaded
>> webload go to hell of that site
>>
>> * This is your cloud based business with bufferbloat fixed -
>>
>> * Any questions?  - point back to campaign site
>>
>> Then create a copy of that 30 second piece for every one of the web
>> service providers listed as supporting the above org. And as many
>> others as possible. And get it on youtube. And buy some ads.
>>
>> I have no idea how much that would cost to produce professionally (?)
>> (at a cut above what jg has already done), but I'm certain it would do
>> some good, both in raising awareness and maybe gaining funding for
>> next year.
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt:
>> http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html