[Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[William Chan (陈智昌)]

Speaking as the primary Chromium developer in charge of this relevant code,
I would like to caution putting too much trust in the numbers generated.
Any statistical claims about the numbers are probably unreasonable to make.

Reasons:
* We don't actively maintain this code. It's behind the command line flags.
They are broken. The fact that it still results in numbers on the benchmark
extension is an example of where unmaintained code doesn't have the UI
disabled, even though the internal workings of the code fail to guarantee
correct operation. We haven't disabled it because, well, it's unmaintained.
* We don't make sure to flush all the network state in between runs, so if
you're using that option, don't trust it to work.
* If you have an advanced Chromium setup, this definitely does not work. I
advise using the benchmark extension only with a separate Chromium profile
for testing purposes. Our flushing of sockets, caches, etc does not
actually work correctly when you use the Chromium multiprofile feature and
also fails to flush lots of our other network caches.
* No one on Chromium really believes the time to paint numbers that we
output :) It's complicated. Our graphics stack is complicated. The time
from when Blink thinks it painted to when the GPU actually blits to the
screen cannot currently be corroborated with any high degree of accuracy
from within our code.
* It has not been maintained since 2010. It is quite likely there are many
other subtle inaccuracies here.

In short, while you can expect it to give you a very high level
understanding of performance issues, I advise against placing non-trivial
confidence in the accuracy of the numbers generated by the benchmark
extension. The fact that numbers are produced by the extension should not
be treated as evidence that the extension actually functions correctly.

Cheers.


On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht at gmail.com> wrote:

> Getting a grip on real web page load time behavior in an age of
> sharded websites,
> dozens of dns lookups, javascript, and fairly random behavior in ad
> services
> and cdns against how a modern browsers behaves is very, very hard.
>
> it turns out if you run
>
> google-chrome --enable-benchmarking --enable-net-benchmarking
>
> (Mac users have to embed these options in their startup script - see
>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags )
>
> enable developer options and install and run the chrome web page
> benchmarker,
> (
> https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?hl=en
> )
>
> that it works (at least for me, on a brief test of the latest chrome, on
> linux.
> Can someone try windows and mac?)
>
> You can then feed in a list of urls to test against, and post process
> the resulting .csv file to your hearts content. We used to use this
> benchmark a lot while trying to characterise typical web behaviors
> under aqm and packet scheduling systems under load. Running
> it simultaneously with a rrul test or one of the simpler tcp upload or
> download
> tests in the rrul suite was often quite interesting.
>
> It turned out the doc has been wrong a while as to the name of the second
> command lnie option. I was gearing up mentally for having to look at
> the source....
>
> http://code.google.com/p/chromium/issues/detail?id=338705
>
> /me happy
>
> --
> Dave Täht
>
> Heartbleed POC on wifi campus networks with EAP auth:
> http://www.eduroam.edu.au/advisory.html
>
> _______________________________________________
> aqm mailing list
> aqm at ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cerowrt-devel/attachments/20140417/b8e72ffd/attachment-0002.html>