From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ve0-x236.google.com (mail-ve0-x236.google.com [IPv6:2607:f8b0:400c:c01::236]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id A558A21F22D for ; Thu, 17 Apr 2014 12:01:48 -0700 (PDT) Received: by mail-ve0-f182.google.com with SMTP id jw12so1014362veb.27 for ; Thu, 17 Apr 2014 12:01:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hpOAtw6Lf/Xe/Nw61+NHIHmUNyZi4n0Xlh7MI76J/vU=; b=XN0ZzAsUWp2JBOuKBvwWDAdRRCkJ4j0OddABvibJapwB3QVXEw+LP0IJuFIgUbj3IZ 6vxOLH0xa5dyphEkBYMuUCw9GIHEx3KzzQKsOBW7JzwkAYyZwZixvTg60ixr99mTykLq Cw9Y9WdUbzuK5lV3JgxBFnLNVrRjqXd5nhBDweLaLooB+N3AysROs5XlcoCBmtDdJI7v z9f2VhvnXjmWg47uJEzE0xM1Ia4y2M6CpYuReEB3knTog60QlJJus0WK6Hicc1YxxvYj lP1IkuYuDlL1GqEDU99TxUzSnP5BvSIyfQoNfK+XIQAyTmb/cWBhOfih8arBFUkowvCX 20Gw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=hpOAtw6Lf/Xe/Nw61+NHIHmUNyZi4n0Xlh7MI76J/vU=; b=M5E0Mdsc+rgmiO3+67gtJi0RTKJ5Lp+zt1F+VF5UQSN8FZRhvznRDKIFj5Jr9in33Z s2/f9mhNqH/aFyvKQE3nfivd8pXdqmE1kI4QlSfDwlkhLfEQCUOk2HW4wsngTgzX5god ZM+AsB30brxUfI29U9rJVJ7sV6v8Gl7A5jvG4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hpOAtw6Lf/Xe/Nw61+NHIHmUNyZi4n0Xlh7MI76J/vU=; b=aCeP+3ZSTymoEuTpZZjx8zT72k3h3VBSd3Mug+6qnnwi0rkC7xOz5GF5sxGI9zSjSh 1VfpIOhLFFjoTaRQyBPbiRjWUQFG8W79d3YKAxjU+8Z5qImxJk5/bGtSdDMY2Pw8SCVj aZHoKBqKB2VNPAQhQh7O8srycFqO6WObrDnrzfdRqhUYTyznQ7SGyq4T2M1l2M2SD2fl /G2bdVrOm9cDCf4uRgsFsuUh37twHPTIetXFY/RKdMbCZhdMn+YeFo+YpyGZ1Dt6YYo9 oqB/udVXRP2cTz7RS+r8x0ZzanEn+bBILfntxqtaICP2R6QARKcxSU43wi2Vlxn3HWPu mAQg== X-Gm-Message-State: ALoCoQnraQoMmH/vLwlmlbXwFIfUFkPbkWdM1OZNdyVN4nep9zzSTSZmbrA2IN78h+oIhy5BD40EH+L9U1VGw65TOnzJ76UAWzadWNYhSh5Kwooxgl4Vqob9y8UKLqV0X2DF5mX7VLQc4Ea++VVQxR1l1/lgvpPCbrrTymnidhaVR5oQNSpOnSke7RgonOeX7Fbuv/It+G+E2Kp1ZjwxMhHeWKMJz7jGlQ== MIME-Version: 1.0 X-Received: by 10.221.63.1 with SMTP id xc1mr1529112vcb.35.1397761307070; Thu, 17 Apr 2014 12:01:47 -0700 (PDT) Sender: willchan@google.com Received: by 10.52.38.101 with HTTP; Thu, 17 Apr 2014 12:01:46 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 Apr 2014 12:01:46 -0700 X-Google-Sender-Auth: rXpkrkO22H4rGzSjw1F59Jpg2fQ Message-ID: From: =?UTF-8?B?V2lsbGlhbSBDaGFuICjpmYjmmbrmmIwp?= To: Dave Taht Content-Type: multipart/alternative; boundary=001a1133499e3075d304f741aaec X-Mailman-Approved-At: Thu, 17 Apr 2014 12:50:55 -0700 Cc: "aqm@ietf.org" , "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2014 19:01:49 -0000 --001a1133499e3075d304f741aaec Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Speaking as the primary Chromium developer in charge of this relevant code, I would like to caution putting too much trust in the numbers generated. Any statistical claims about the numbers are probably unreasonable to make. Reasons: * We don't actively maintain this code. It's behind the command line flags. They are broken. The fact that it still results in numbers on the benchmark extension is an example of where unmaintained code doesn't have the UI disabled, even though the internal workings of the code fail to guarantee correct operation. We haven't disabled it because, well, it's unmaintained. * We don't make sure to flush all the network state in between runs, so if you're using that option, don't trust it to work. * If you have an advanced Chromium setup, this definitely does not work. I advise using the benchmark extension only with a separate Chromium profile for testing purposes. Our flushing of sockets, caches, etc does not actually work correctly when you use the Chromium multiprofile feature and also fails to flush lots of our other network caches. * No one on Chromium really believes the time to paint numbers that we output :) It's complicated. Our graphics stack is complicated. The time from when Blink thinks it painted to when the GPU actually blits to the screen cannot currently be corroborated with any high degree of accuracy from within our code. * It has not been maintained since 2010. It is quite likely there are many other subtle inaccuracies here. In short, while you can expect it to give you a very high level understanding of performance issues, I advise against placing non-trivial confidence in the accuracy of the numbers generated by the benchmark extension. The fact that numbers are produced by the extension should not be treated as evidence that the extension actually functions correctly. Cheers. On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht wrote: > Getting a grip on real web page load time behavior in an age of > sharded websites, > dozens of dns lookups, javascript, and fairly random behavior in ad > services > and cdns against how a modern browsers behaves is very, very hard. > > it turns out if you run > > google-chrome --enable-benchmarking --enable-net-benchmarking > > (Mac users have to embed these options in their startup script - see > http://www.chromium.org/developers/how-tos/run-chromium-with-flags ) > > enable developer options and install and run the chrome web page > benchmarker, > ( > https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahe= kjcahlbpccbgaopjll?hl=3Den > ) > > that it works (at least for me, on a brief test of the latest chrome, on > linux. > Can someone try windows and mac?) > > You can then feed in a list of urls to test against, and post process > the resulting .csv file to your hearts content. We used to use this > benchmark a lot while trying to characterise typical web behaviors > under aqm and packet scheduling systems under load. Running > it simultaneously with a rrul test or one of the simpler tcp upload or > download > tests in the rrul suite was often quite interesting. > > It turned out the doc has been wrong a while as to the name of the second > command lnie option. I was gearing up mentally for having to look at > the source.... > > http://code.google.com/p/chromium/issues/detail?id=3D338705 > > /me happy > > -- > Dave T=E4ht > > Heartbleed POC on wifi campus networks with EAP auth: > http://www.eduroam.edu.au/advisory.html > > _______________________________________________ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm > --001a1133499e3075d304f741aaec Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Speaking as the primary Chromium developer in charge of th= is relevant code, I would like to caution putting too much trust in the num= bers generated. Any statistical claims about the numbers are probably unrea= sonable to make.

Reasons:
* We don't actively maintain this cod= e. It's behind the command line flags. They are broken. The fact that i= t still results in numbers on the benchmark extension is an example of wher= e unmaintained code doesn't have the UI disabled, even though the inter= nal workings of the code fail to guarantee correct operation. We haven'= t disabled it because, well, it's unmaintained.
* We don't make sure to flush all the network state in between run= s, so if you're using that option, don't trust it to work.
* If you have an advanced Chromium setup, this definitely does not work. = I advise using the benchmark extension only with a separate Chromium profil= e for testing purposes. Our flushing of sockets, caches, etc does not actua= lly work correctly when you use the Chromium multiprofile feature and also = fails to flush lots of our other network caches.
* No one on Chromium really believes the time to paint numbers that we= output :) It's complicated. Our graphics stack is complicated. The tim= e from when Blink thinks it painted to when the GPU actually blits to the s= creen cannot currently be corroborated with any high degree of accuracy fro= m within our code.
* It has not been maintained since 2010. It is quite likely there are = many other subtle inaccuracies here.

In short, whi= le you can expect it to give you a very high level understanding of perform= ance issues, I advise against placing non-trivial confidence in the accurac= y of the numbers generated by the benchmark extension. The fact that number= s are produced by the extension should not be treated as evidence that the = extension actually functions correctly.

Cheers.


<= div class=3D"gmail_quote">On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <d= ave.taht@gmail.com> wrote:
Getting a grip on real web page load time be= havior in an age of
sharded websites,
dozens of dns lookups, javascript, and fairly random behavior in ad service= s
and cdns against how a modern browsers behaves is very, very hard.

it turns out if you run

google-chrome --enable-benchmarking --enable-net-benchmarking

(Mac users have to embed these options in their startup script - see
=A0http://www.chromium.org/developers/how-tos/run-chr= omium-with-flags )

enable developer options and install and run the chrome web page benchmarke= r,
( https://chrome.goo= gle.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?h= l=3Den
)

that it works (at least for me, on a brief test of the latest chrome, on li= nux.
Can someone try windows and mac?)

You can then feed in a list of urls to test against, and post process
the resulting .csv file to your hearts content. We used to use this
benchmark a lot while trying to characterise typical web behaviors
under aqm and packet scheduling systems under load. Running
it simultaneously with a rrul test or one of the simpler tcp upload or down= load
tests in the rrul suite was often quite interesting.

It turned out the doc has been wrong a while as to the name of the second command lnie option. I was gearing up mentally for having to look at
the source....

http://code.google.com/p/chromium/issues/detail?id=3D338705<= /a>

/me happy

--
Dave T=E4ht

Heartbleed POC on wifi campus networks with EAP auth:
http:= //www.eduroam.edu.au/advisory.html

_______________________________________________
aqm mailing list
aqm@ietf.org
htt= ps://www.ietf.org/mailman/listinfo/aqm

--001a1133499e3075d304f741aaec--