[Cerowrt-devel] chrome web page benchmarker fixed

[Cerowrt-devel] chrome web page benchmarker fixed

[Dave Taht]

Getting a grip on real web page load time behavior in an age of
sharded websites,
dozens of dns lookups, javascript, and fairly random behavior in ad services
and cdns against how a modern browsers behaves is very, very hard.

it turns out if you run

google-chrome --enable-benchmarking --enable-net-benchmarking

(Mac users have to embed these options in their startup script - see
 http://www.chromium.org/developers/how-tos/run-chromium-with-flags )

enable developer options and install and run the chrome web page benchmarker,
( https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?hl=en
)

that it works (at least for me, on a brief test of the latest chrome, on linux.
Can someone try windows and mac?)

You can then feed in a list of urls to test against, and post process
the resulting .csv file to your hearts content. We used to use this
benchmark a lot while trying to characterise typical web behaviors
under aqm and packet scheduling systems under load. Running
it simultaneously with a rrul test or one of the simpler tcp upload or download
tests in the rrul suite was often quite interesting.

It turned out the doc has been wrong a while as to the name of the second
command lnie option. I was gearing up mentally for having to look at
the source....

http://code.google.com/p/chromium/issues/detail?id=338705

/me happy

-- 
Dave Täht

Heartbleed POC on wifi campus networks with EAP auth:
http://www.eduroam.edu.au/advisory.html

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[William Chan (陈智昌)]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

Speaking as the primary Chromium developer in charge of this relevant code,
I would like to caution putting too much trust in the numbers generated.
Any statistical claims about the numbers are probably unreasonable to make.

Reasons:
* We don't actively maintain this code. It's behind the command line flags.
They are broken. The fact that it still results in numbers on the benchmark
extension is an example of where unmaintained code doesn't have the UI
disabled, even though the internal workings of the code fail to guarantee
correct operation. We haven't disabled it because, well, it's unmaintained.
* We don't make sure to flush all the network state in between runs, so if
you're using that option, don't trust it to work.
* If you have an advanced Chromium setup, this definitely does not work. I
advise using the benchmark extension only with a separate Chromium profile
for testing purposes. Our flushing of sockets, caches, etc does not
actually work correctly when you use the Chromium multiprofile feature and
also fails to flush lots of our other network caches.
* No one on Chromium really believes the time to paint numbers that we
output :) It's complicated. Our graphics stack is complicated. The time
from when Blink thinks it painted to when the GPU actually blits to the
screen cannot currently be corroborated with any high degree of accuracy
from within our code.
* It has not been maintained since 2010. It is quite likely there are many
other subtle inaccuracies here.

In short, while you can expect it to give you a very high level
understanding of performance issues, I advise against placing non-trivial
confidence in the accuracy of the numbers generated by the benchmark
extension. The fact that numbers are produced by the extension should not
be treated as evidence that the extension actually functions correctly.

Cheers.


On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com> wrote:

> Getting a grip on real web page load time behavior in an age of
> sharded websites,
> dozens of dns lookups, javascript, and fairly random behavior in ad
> services
> and cdns against how a modern browsers behaves is very, very hard.
>
> it turns out if you run
>
> google-chrome --enable-benchmarking --enable-net-benchmarking
>
> (Mac users have to embed these options in their startup script - see
>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags )
>
> enable developer options and install and run the chrome web page
> benchmarker,
> (
> https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?hl=en
> )
>
> that it works (at least for me, on a brief test of the latest chrome, on
> linux.
> Can someone try windows and mac?)
>
> You can then feed in a list of urls to test against, and post process
> the resulting .csv file to your hearts content. We used to use this
> benchmark a lot while trying to characterise typical web behaviors
> under aqm and packet scheduling systems under load. Running
> it simultaneously with a rrul test or one of the simpler tcp upload or
> download
> tests in the rrul suite was often quite interesting.
>
> It turned out the doc has been wrong a while as to the name of the second
> command lnie option. I was gearing up mentally for having to look at
> the source....
>
> http://code.google.com/p/chromium/issues/detail?id=338705
>
> /me happy
>
> --
> Dave Täht
>
> Heartbleed POC on wifi campus networks with EAP auth:
> http://www.eduroam.edu.au/advisory.html
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>

[-- Attachment #2: Type: text/html, Size: 4779 bytes --]

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Dave Taht]

On Thu, Apr 17, 2014 at 12:01 PM, William Chan (陈智昌)
<willchan@chromium.org> wrote:
> Speaking as the primary Chromium developer in charge of this relevant code,
> I would like to caution putting too much trust in the numbers generated. Any
> statistical claims about the numbers are probably unreasonable to make.

Sigh. Other benchmarks such as the apache ("ab") benchmark
are primarily designed as stress testers for web servers, not as realistic
traffic. Modern web traffic has such a high level of dynamicism in it,
that static web page loads along any distribution, seem insufficient,
passive analysis of aggregated traffic "feels" incorrect relative to the
sorts of home and small business traffic I've seen, and so on.

Famous papers, such as this one:

http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-leland.pdf

Seem possibly irrelevant to draw conclusions from given the kind
of data they analysed and proceeding from an incorrect model or
gut feel for how the network behaves today seems to be foolish.

Even the most basic of tools, such as httping, had three basic bugs
that I found in a few minutes of trying to come up with some basic
behaviors yesterday:

https://lists.bufferbloat.net/pipermail/bloat/2014-April/001890.html

Those are going to be a lot easier to fix than diving into the chromium
codebase!

There are very few tools worth trusting, and I am always dubious
of papers that publish results with unavailable tools and data. The only
tools I have any faith in for network analysis are netperf, netperf-wrapper,
tcpdump and xplot.org, and to a large extent wireshark. Toke and I have
been tearing apart d-itg and I hope to one day be able to add that to
my trustable list... but better tools are needed!

Tools that I don't have a lot of faith in include that, iperf, anything written
in java or other high level languages, speedtest.net, and things like
shaperprobe.

Have very little faith in ns2, slightly more in ns3, and I've been meaning
to look over the mininet and other simulators whenever I got some spare
time; the mininet results stanford gets seem pretty reasonable and I
adore their reproducing results effort. Haven't explored ndt, keep meaning
to...

> Reasons:
> * We don't actively maintain this code. It's behind the command line flags.
> They are broken. The fact that it still results in numbers on the benchmark
> extension is an example of where unmaintained code doesn't have the UI
> disabled, even though the internal workings of the code fail to guarantee
> correct operation. We haven't disabled it because, well, it's unmaintained.

As I mentioned I was gearing up for a hacking run...

The vast majority of results I look at are actually obtained via
looking at packet captures. I mostly use benchmarks as abstractions
to see if they make some sense relative to the captures and tend
to compare different benchmarks against each other.

I realize others don't go into that level of detail, so you have given
fair warning! In our case we used the web page benchmarker as
a means to try and rapidly acquire some typical distributions of
get and tcp stream requests from things like the alexa top 1000,
and as a way to A/B different aqm/packet scheduling setups.

... but the only easily publishable results were from the benchmark itself,
and we (reluctantly) only published one graph from all the work that
went into it 2+ years back and used it as a test driver for the famous
ietf video, comparing two identical boxes running it at the same time
under different network conditions:

https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#IETF-demo-side-by-side-of-a-normal-cable-modem-vs-fq_codel

from what I fiddled with today, it is at least still useful for that?

moving on...

The web model in the cablelabs work doesn't look much like my captures,
in addition to not modeling dns at all, and using a smaller IW than google
it looks like this:

>> Model single user web page download as follows:

>> - Web page modeled as single HTML page + 100 objects spread evenly
>> across 4 servers. Web object sizes are currently fixed at 25 kB each,
>> whereas the initial HTML page is 100 kB. Appendix A provides an
>> alternative page model that may be explored in future work.

Where what I see is a huge number of stuff that fits into a single
iw10 slow start episode and some level of pipelining on larger stuff, so that a
large number of object sizes of less than 7k with a lightly tailed distribution
outside of that makes more sense.

(I'm not staring at appendix A right now, I'm under the impression
 it was better)

I certainly would like more suggestions for models and types
of web traffic, as well as simulation of https + pfs traffic,
spdy, quic, etc....

>> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).

Server RTTs from my own web history tend to be lower than 50ms.

>> - Initial HTTP GET to retrieve a moderately sized object (100 kB HTML
>> page) from server 1.

An initial GET to google fits into iw10 - it's about 7k.

>> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP GETs
>> (via separate TCP connections), 6 connections each to 4 different
>> server nodes

I usually don't see more than 15. and certainly not 25k sized objects.

 > - Once each individual HTTP GET completes, initiate a subsequent GET
> to the same server, until 25 objects have been retrieved from each
> server.


> * We don't make sure to flush all the network state in between runs, so if
> you're using that option, don't trust it to work.

The typical scenario we used was a run against dozens or hundreds of urls,
capturing traffic, while varying network conditions.

Regarded the first run as the most interesting.

Can exit the browser and restart after a run like that.

At moment, merely plan to use the tool primarily to survey various
web sites and load times while doing packet captures. Hope was
to get valid data from the network portion of the load, tho...

> * If you have an advanced Chromium setup, this definitely does not work. I
> advise using the benchmark extension only with a separate Chromium profile
> for testing purposes. Our flushing of sockets, caches, etc does not actually
> work correctly when you use the Chromium multiprofile feature and also fails
> to flush lots of our other network caches.

noted.


> * No one on Chromium really believes the time to paint numbers that we
> output :) It's complicated. Our graphics stack is complicated. The time from

I actually care only about time-to-full layout as that's a core network
effect...

> when Blink thinks it painted to when the GPU actually blits to the screen
> cannot currently be corroborated with any high degree of accuracy from
> within our code.

> * It has not been maintained since 2010. It is quite likely there are many
> other subtle inaccuracies here.

Grok.

> In short, while you can expect it to give you a very high level
> understanding of performance issues, I advise against placing non-trivial
> confidence in the accuracy of the numbers generated by the benchmark
> extension. The fact that numbers are produced by the extension should not be
> treated as evidence that the extension actually functions correctly.

OK, noted. Still delighted to be able to have a simple load generator
that exercises the browsers and generates some results, however
dubious.

>
> Cheers.
>
>
> On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Getting a grip on real web page load time behavior in an age of
>> sharded websites,
>> dozens of dns lookups, javascript, and fairly random behavior in ad
>> services
>> and cdns against how a modern browsers behaves is very, very hard.
>>
>> it turns out if you run
>>
>> google-chrome --enable-benchmarking --enable-net-benchmarking
>>
>> (Mac users have to embed these options in their startup script - see
>>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags )
>>
>> enable developer options and install and run the chrome web page
>> benchmarker,
>> (
>> https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?hl=en
>> )
>>
>> that it works (at least for me, on a brief test of the latest chrome, on
>> linux.
>> Can someone try windows and mac?)
>>
>> You can then feed in a list of urls to test against, and post process
>> the resulting .csv file to your hearts content. We used to use this
>> benchmark a lot while trying to characterise typical web behaviors
>> under aqm and packet scheduling systems under load. Running
>> it simultaneously with a rrul test or one of the simpler tcp upload or
>> download
>> tests in the rrul suite was often quite interesting.
>>
>> It turned out the doc has been wrong a while as to the name of the second
>> command lnie option. I was gearing up mentally for having to look at
>> the source....
>>
>> http://code.google.com/p/chromium/issues/detail?id=338705
>>
>> /me happy
>>
>> --
>> Dave Täht
>>
>> Heartbleed POC on wifi campus networks with EAP auth:
>> http://www.eduroam.edu.au/advisory.html
>>
>> _______________________________________________
>> aqm mailing list
>> aqm@ietf.org
>> https://www.ietf.org/mailman/listinfo/aqm
>
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[William Chan (陈智昌)]

[-- Attachment #1: Type: text/plain, Size: 10404 bytes --]

I understand the desire for this code to work, and if I had more bandwidth,
I would definitely fix it sooner. For now, all I can say is, be aware of
the potential issues with the code and use at your own risk. Sorry that we
don't have more bandwidth now to get this working sooner.


On Thu, Apr 17, 2014 at 2:07 PM, Dave Taht <dave.taht@gmail.com> wrote:

> On Thu, Apr 17, 2014 at 12:01 PM, William Chan (陈智昌)
> <willchan@chromium.org> wrote:
> > Speaking as the primary Chromium developer in charge of this relevant
> code,
> > I would like to caution putting too much trust in the numbers generated.
> Any
> > statistical claims about the numbers are probably unreasonable to make.
>
> Sigh. Other benchmarks such as the apache ("ab") benchmark
> are primarily designed as stress testers for web servers, not as realistic
> traffic. Modern web traffic has such a high level of dynamicism in it,
> that static web page loads along any distribution, seem insufficient,
> passive analysis of aggregated traffic "feels" incorrect relative to the
> sorts of home and small business traffic I've seen, and so on.
>
> Famous papers, such as this one:
>
> http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-leland.pdf
>
> Seem possibly irrelevant to draw conclusions from given the kind
> of data they analysed and proceeding from an incorrect model or
> gut feel for how the network behaves today seems to be foolish.
>
> Even the most basic of tools, such as httping, had three basic bugs
> that I found in a few minutes of trying to come up with some basic
> behaviors yesterday:
>
> https://lists.bufferbloat.net/pipermail/bloat/2014-April/001890.html
>
> Those are going to be a lot easier to fix than diving into the chromium
> codebase!
>
> There are very few tools worth trusting, and I am always dubious
> of papers that publish results with unavailable tools and data. The only
> tools I have any faith in for network analysis are netperf,
> netperf-wrapper,
> tcpdump and xplot.org, and to a large extent wireshark. Toke and I have
> been tearing apart d-itg and I hope to one day be able to add that to
> my trustable list... but better tools are needed!
>
> Tools that I don't have a lot of faith in include that, iperf, anything
> written
> in java or other high level languages, speedtest.net, and things like
> shaperprobe.
>
> Have very little faith in ns2, slightly more in ns3, and I've been meaning
> to look over the mininet and other simulators whenever I got some spare
> time; the mininet results stanford gets seem pretty reasonable and I
> adore their reproducing results effort. Haven't explored ndt, keep meaning
> to...
>
> > Reasons:
> > * We don't actively maintain this code. It's behind the command line
> flags.
> > They are broken. The fact that it still results in numbers on the
> benchmark
> > extension is an example of where unmaintained code doesn't have the UI
> > disabled, even though the internal workings of the code fail to guarantee
> > correct operation. We haven't disabled it because, well, it's
> unmaintained.
>
> As I mentioned I was gearing up for a hacking run...
>
> The vast majority of results I look at are actually obtained via
> looking at packet captures. I mostly use benchmarks as abstractions
> to see if they make some sense relative to the captures and tend
> to compare different benchmarks against each other.
>
> I realize others don't go into that level of detail, so you have given
> fair warning! In our case we used the web page benchmarker as
> a means to try and rapidly acquire some typical distributions of
> get and tcp stream requests from things like the alexa top 1000,
> and as a way to A/B different aqm/packet scheduling setups.
>
> ... but the only easily publishable results were from the benchmark itself,
> and we (reluctantly) only published one graph from all the work that
> went into it 2+ years back and used it as a test driver for the famous
> ietf video, comparing two identical boxes running it at the same time
> under different network conditions:
>
>
> https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#IETF-demo-side-by-side-of-a-normal-cable-modem-vs-fq_codel
>
> from what I fiddled with today, it is at least still useful for that?
>
> moving on...
>
> The web model in the cablelabs work doesn't look much like my captures,
> in addition to not modeling dns at all, and using a smaller IW than google
> it looks like this:
>
> >> Model single user web page download as follows:
>
> >> - Web page modeled as single HTML page + 100 objects spread evenly
> >> across 4 servers. Web object sizes are currently fixed at 25 kB each,
> >> whereas the initial HTML page is 100 kB. Appendix A provides an
> >> alternative page model that may be explored in future work.
>
> Where what I see is a huge number of stuff that fits into a single
> iw10 slow start episode and some level of pipelining on larger stuff, so
> that a
> large number of object sizes of less than 7k with a lightly tailed
> distribution
> outside of that makes more sense.
>
> (I'm not staring at appendix A right now, I'm under the impression
>  it was better)
>
> I certainly would like more suggestions for models and types
> of web traffic, as well as simulation of https + pfs traffic,
> spdy, quic, etc....
>
> >> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).
>
> Server RTTs from my own web history tend to be lower than 50ms.
>
> >> - Initial HTTP GET to retrieve a moderately sized object (100 kB HTML
> >> page) from server 1.
>
> An initial GET to google fits into iw10 - it's about 7k.
>
> >> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP GETs
> >> (via separate TCP connections), 6 connections each to 4 different
> >> server nodes
>
> I usually don't see more than 15. and certainly not 25k sized objects.
>
>  > - Once each individual HTTP GET completes, initiate a subsequent GET
> > to the same server, until 25 objects have been retrieved from each
> > server.
>
>
> > * We don't make sure to flush all the network state in between runs, so
> if
> > you're using that option, don't trust it to work.
>
> The typical scenario we used was a run against dozens or hundreds of urls,
> capturing traffic, while varying network conditions.
>
> Regarded the first run as the most interesting.
>
> Can exit the browser and restart after a run like that.
>
> At moment, merely plan to use the tool primarily to survey various
> web sites and load times while doing packet captures. Hope was
> to get valid data from the network portion of the load, tho...
>
> > * If you have an advanced Chromium setup, this definitely does not work.
> I
> > advise using the benchmark extension only with a separate Chromium
> profile
> > for testing purposes. Our flushing of sockets, caches, etc does not
> actually
> > work correctly when you use the Chromium multiprofile feature and also
> fails
> > to flush lots of our other network caches.
>
> noted.
>
>
> > * No one on Chromium really believes the time to paint numbers that we
> > output :) It's complicated. Our graphics stack is complicated. The time
> from
>
> I actually care only about time-to-full layout as that's a core network
> effect...
>
> > when Blink thinks it painted to when the GPU actually blits to the screen
> > cannot currently be corroborated with any high degree of accuracy from
> > within our code.
>
> > * It has not been maintained since 2010. It is quite likely there are
> many
> > other subtle inaccuracies here.
>
> Grok.
>
> > In short, while you can expect it to give you a very high level
> > understanding of performance issues, I advise against placing non-trivial
> > confidence in the accuracy of the numbers generated by the benchmark
> > extension. The fact that numbers are produced by the extension should
> not be
> > treated as evidence that the extension actually functions correctly.
>
> OK, noted. Still delighted to be able to have a simple load generator
> that exercises the browsers and generates some results, however
> dubious.
>
> >
> > Cheers.
> >
> >
> > On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> Getting a grip on real web page load time behavior in an age of
> >> sharded websites,
> >> dozens of dns lookups, javascript, and fairly random behavior in ad
> >> services
> >> and cdns against how a modern browsers behaves is very, very hard.
> >>
> >> it turns out if you run
> >>
> >> google-chrome --enable-benchmarking --enable-net-benchmarking
> >>
> >> (Mac users have to embed these options in their startup script - see
> >>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags )
> >>
> >> enable developer options and install and run the chrome web page
> >> benchmarker,
> >> (
> >>
> https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomahekjcahlbpccbgaopjll?hl=en
> >> )
> >>
> >> that it works (at least for me, on a brief test of the latest chrome, on
> >> linux.
> >> Can someone try windows and mac?)
> >>
> >> You can then feed in a list of urls to test against, and post process
> >> the resulting .csv file to your hearts content. We used to use this
> >> benchmark a lot while trying to characterise typical web behaviors
> >> under aqm and packet scheduling systems under load. Running
> >> it simultaneously with a rrul test or one of the simpler tcp upload or
> >> download
> >> tests in the rrul suite was often quite interesting.
> >>
> >> It turned out the doc has been wrong a while as to the name of the
> second
> >> command lnie option. I was gearing up mentally for having to look at
> >> the source....
> >>
> >> http://code.google.com/p/chromium/issues/detail?id=338705
> >>
> >> /me happy
> >>
> >> --
> >> Dave Täht
> >>
> >> Heartbleed POC on wifi campus networks with EAP auth:
> >> http://www.eduroam.edu.au/advisory.html
> >>
> >> _______________________________________________
> >> aqm mailing list
> >> aqm@ietf.org
> >> https://www.ietf.org/mailman/listinfo/aqm
> >
> >
>
>
>
> --
> Dave Täht
>
> NSFW:
> https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article
>

[-- Attachment #2: Type: text/html, Size: 13140 bytes --]

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Greg White]

[-- Attachment #1: Type: text/plain, Size: 11693 bytes --]

Dave, 

We used the 25k object size for a short time back in 2012 until we had
resources to build a more advanced model (appendix A).  I did a bunch of
captures of real web pages back in 2011 and compared the object size
statistics to models that I'd seen published.  Lognormal didn't seem to be
*exactly* right, but it wasn't a bad fit to what I saw.  I've attached a
CDF.

The choice of 4 servers was based somewhat on logistics, and also on a
finding that across our data set, the average web page retrieved 81% of
its resources from the top 4 servers.  Increasing to 5 servers only
increased that percentage to 84%.

The choice of RTTs also came from the web traffic captures. I saw
RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.

Much of this can be found in
https://tools.ietf.org/html/draft-white-httpbis-spdy-analysis-00

In many of the cases that we've simulated, the packet drop probability is
less than 1% for DNS packets.  In our web model, there are a total of 4
servers, so 4 DNS lookups assuming none of the addresses are cached. If
PLR = 1%, there would be a 3.9% chance of losing one or more DNS packets
(with a resulting ~5 second additional delay on load time).  I've probably
oversimplified this, but Kathie N. and I made the call that it would be
significantly easier to just do this math than to build a dns
implementation in ns2.  We've open sourced the web model (it's on Kathie's
web page and will be part of ns2.36) with an encouragement to the
community to improve on it.  If you'd like to port it to ns3 and add a dns
model, that would be fantastic.

-Greg


On 4/17/14, 3:07 PM, "Dave Taht" <dave.taht@gmail.com> wrote:

>On Thu, Apr 17, 2014 at 12:01 PM, William Chan (陈智昌)
><willchan@chromium.org> wrote:
>> Speaking as the primary Chromium developer in charge of this relevant
>>code,
>> I would like to caution putting too much trust in the numbers
>>generated. Any
>> statistical claims about the numbers are probably unreasonable to make.
>
>Sigh. Other benchmarks such as the apache ("ab") benchmark
>are primarily designed as stress testers for web servers, not as realistic
>traffic. Modern web traffic has such a high level of dynamicism in it,
>that static web page loads along any distribution, seem insufficient,
>passive analysis of aggregated traffic "feels" incorrect relative to the
>sorts of home and small business traffic I've seen, and so on.
>
>Famous papers, such as this one:
>
>http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-leland.pdf
>
>Seem possibly irrelevant to draw conclusions from given the kind
>of data they analysed and proceeding from an incorrect model or
>gut feel for how the network behaves today seems to be foolish.
>
>Even the most basic of tools, such as httping, had three basic bugs
>that I found in a few minutes of trying to come up with some basic
>behaviors yesterday:
>
>https://lists.bufferbloat.net/pipermail/bloat/2014-April/001890.html
>
>Those are going to be a lot easier to fix than diving into the chromium
>codebase!
>
>There are very few tools worth trusting, and I am always dubious
>of papers that publish results with unavailable tools and data. The only
>tools I have any faith in for network analysis are netperf,
>netperf-wrapper,
>tcpdump and xplot.org, and to a large extent wireshark. Toke and I have
>been tearing apart d-itg and I hope to one day be able to add that to
>my trustable list... but better tools are needed!
>
>Tools that I don't have a lot of faith in include that, iperf, anything
>written
>in java or other high level languages, speedtest.net, and things like
>shaperprobe.
>
>Have very little faith in ns2, slightly more in ns3, and I've been meaning
>to look over the mininet and other simulators whenever I got some spare
>time; the mininet results stanford gets seem pretty reasonable and I
>adore their reproducing results effort. Haven't explored ndt, keep meaning
>to...
>
>> Reasons:
>> * We don't actively maintain this code. It's behind the command line
>>flags.
>> They are broken. The fact that it still results in numbers on the
>>benchmark
>> extension is an example of where unmaintained code doesn't have the UI
>> disabled, even though the internal workings of the code fail to
>>guarantee
>> correct operation. We haven't disabled it because, well, it's
>>unmaintained.
>
>As I mentioned I was gearing up for a hacking run...
>
>The vast majority of results I look at are actually obtained via
>looking at packet captures. I mostly use benchmarks as abstractions
>to see if they make some sense relative to the captures and tend
>to compare different benchmarks against each other.
>
>I realize others don't go into that level of detail, so you have given
>fair warning! In our case we used the web page benchmarker as
>a means to try and rapidly acquire some typical distributions of
>get and tcp stream requests from things like the alexa top 1000,
>and as a way to A/B different aqm/packet scheduling setups.
>
>... but the only easily publishable results were from the benchmark
>itself,
>and we (reluctantly) only published one graph from all the work that
>went into it 2+ years back and used it as a test driver for the famous
>ietf video, comparing two identical boxes running it at the same time
>under different network conditions:
>
>https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#IETF-demo-s
>ide-by-side-of-a-normal-cable-modem-vs-fq_codel
>
>from what I fiddled with today, it is at least still useful for that?
>
>moving on...
>
>The web model in the cablelabs work doesn't look much like my captures,
>in addition to not modeling dns at all, and using a smaller IW than google
>it looks like this:
>
>>> Model single user web page download as follows:
>
>>> - Web page modeled as single HTML page + 100 objects spread evenly
>>> across 4 servers. Web object sizes are currently fixed at 25 kB each,
>>> whereas the initial HTML page is 100 kB. Appendix A provides an
>>> alternative page model that may be explored in future work.
>
>Where what I see is a huge number of stuff that fits into a single
>iw10 slow start episode and some level of pipelining on larger stuff, so
>that a
>large number of object sizes of less than 7k with a lightly tailed
>distribution
>outside of that makes more sense.
>
>(I'm not staring at appendix A right now, I'm under the impression
> it was better)
>
>I certainly would like more suggestions for models and types
>of web traffic, as well as simulation of https + pfs traffic,
>spdy, quic, etc....
>
>>> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).
>
>Server RTTs from my own web history tend to be lower than 50ms.
>
>>> - Initial HTTP GET to retrieve a moderately sized object (100 kB HTML
>>> page) from server 1.
>
>An initial GET to google fits into iw10 - it's about 7k.
>
>>> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP GETs
>>> (via separate TCP connections), 6 connections each to 4 different
>>> server nodes
>
>I usually don't see more than 15. and certainly not 25k sized objects.
>
> > - Once each individual HTTP GET completes, initiate a subsequent GET
>> to the same server, until 25 objects have been retrieved from each
>> server.
>
>
>> * We don't make sure to flush all the network state in between runs, so
>>if
>> you're using that option, don't trust it to work.
>
>The typical scenario we used was a run against dozens or hundreds of urls,
>capturing traffic, while varying network conditions.
>
>Regarded the first run as the most interesting.
>
>Can exit the browser and restart after a run like that.
>
>At moment, merely plan to use the tool primarily to survey various
>web sites and load times while doing packet captures. Hope was
>to get valid data from the network portion of the load, tho...
>
>> * If you have an advanced Chromium setup, this definitely does not
>>work. I
>> advise using the benchmark extension only with a separate Chromium
>>profile
>> for testing purposes. Our flushing of sockets, caches, etc does not
>>actually
>> work correctly when you use the Chromium multiprofile feature and also
>>fails
>> to flush lots of our other network caches.
>
>noted.
>
>
>> * No one on Chromium really believes the time to paint numbers that we
>> output :) It's complicated. Our graphics stack is complicated. The time
>>from
>
>I actually care only about time-to-full layout as that's a core network
>effect...
>
>> when Blink thinks it painted to when the GPU actually blits to the
>>screen
>> cannot currently be corroborated with any high degree of accuracy from
>> within our code.
>
>> * It has not been maintained since 2010. It is quite likely there are
>>many
>> other subtle inaccuracies here.
>
>Grok.
>
>> In short, while you can expect it to give you a very high level
>> understanding of performance issues, I advise against placing
>>non-trivial
>> confidence in the accuracy of the numbers generated by the benchmark
>> extension. The fact that numbers are produced by the extension should
>>not be
>> treated as evidence that the extension actually functions correctly.
>
>OK, noted. Still delighted to be able to have a simple load generator
>that exercises the browsers and generates some results, however
>dubious.
>
>>
>> Cheers.
>>
>>
>> On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> Getting a grip on real web page load time behavior in an age of
>>> sharded websites,
>>> dozens of dns lookups, javascript, and fairly random behavior in ad
>>> services
>>> and cdns against how a modern browsers behaves is very, very hard.
>>>
>>> it turns out if you run
>>>
>>> google-chrome --enable-benchmarking --enable-net-benchmarking
>>>
>>> (Mac users have to embed these options in their startup script - see
>>>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags )
>>>
>>> enable developer options and install and run the chrome web page
>>> benchmarker,
>>> (
>>> 
>>>https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomah
>>>ekjcahlbpccbgaopjll?hl=en
>>> )
>>>
>>> that it works (at least for me, on a brief test of the latest chrome,
>>>on
>>> linux.
>>> Can someone try windows and mac?)
>>>
>>> You can then feed in a list of urls to test against, and post process
>>> the resulting .csv file to your hearts content. We used to use this
>>> benchmark a lot while trying to characterise typical web behaviors
>>> under aqm and packet scheduling systems under load. Running
>>> it simultaneously with a rrul test or one of the simpler tcp upload or
>>> download
>>> tests in the rrul suite was often quite interesting.
>>>
>>> It turned out the doc has been wrong a while as to the name of the
>>>second
>>> command lnie option. I was gearing up mentally for having to look at
>>> the source....
>>>
>>> http://code.google.com/p/chromium/issues/detail?id=338705
>>>
>>> /me happy
>>>
>>> --
>>> Dave Täht
>>>
>>> Heartbleed POC on wifi campus networks with EAP auth:
>>> http://www.eduroam.edu.au/advisory.html
>>>
>>> _______________________________________________
>>> aqm mailing list
>>> aqm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/aqm
>>
>>
>
>
>
>-- 
>Dave Täht
>
>NSFW: 
>https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indec
>ent.article
>
>_______________________________________________
>aqm mailing list
>aqm@ietf.org
>https://www.ietf.org/mailman/listinfo/aqm


[-- Attachment #2: content_length_cdf.eps --]
[-- Type: application/octet-stream, Size: 17824 bytes --]

%!PS-Adobe-3.0 EPSF-3.0
%%Creator: MATLAB, The MathWorks, Inc. Version 7.14.0.739 (R2012a). Operating System: Darwin 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64.
%%Title: /Users/gwhite/Documents/FasterInternet/SPDY/Website traces/content_length_cdf.eps
%%CreationDate: 05/07/2013  14:35:35
%%DocumentNeededFonts: Helvetica
%%DocumentProcessColors: Cyan Magenta Yellow Black
%%LanguageLevel: 2
%%Pages: 1
%%BoundingBox:  -104   125   717   667
%%EndComments

%%BeginProlog
% MathWorks dictionary
/MathWorks 160 dict begin
% definition operators
/bdef {bind def} bind def
/ldef {load def} bind def
/xdef {exch def} bdef
/xstore {exch store} bdef
% operator abbreviations
/c  /clip ldef
/cc /concat ldef
/cp /closepath ldef
/gr /grestore ldef
/gs /gsave ldef
/mt /moveto ldef
/np /newpath ldef
/cm /currentmatrix ldef
/sm /setmatrix ldef
/rm /rmoveto ldef
/rl /rlineto ldef
/s {show newpath} bdef
/sc {setcmykcolor} bdef
/sr /setrgbcolor ldef
/sg /setgray ldef
/w /setlinewidth ldef
/j /setlinejoin ldef
/cap /setlinecap ldef
/rc {rectclip} bdef
/rf {rectfill} bdef
% page state control
/pgsv () def
/bpage {/pgsv save def} bdef
/epage {pgsv restore} bdef
/bplot /gsave ldef
/eplot {stroke grestore} bdef
% orientation switch
/portraitMode 0 def /landscapeMode 1 def /rotateMode 2 def
% coordinate system mappings
/dpi2point 0 def
% font control
/FontSize 0 def
/FMS {/FontSize xstore findfont [FontSize 0 0 FontSize neg 0 0]
  makefont setfont} bdef
/reencode {exch dup where {pop load} {pop StandardEncoding} ifelse
  exch dup 3 1 roll findfont dup length dict begin
  { 1 index /FID ne {def}{pop pop} ifelse } forall
  /Encoding exch def currentdict end definefont pop} bdef
/isroman {findfont /CharStrings get /Agrave known} bdef
/FMSR {3 1 roll 1 index dup isroman {reencode} {pop pop} ifelse
  exch FMS} bdef
/csm {1 dpi2point div -1 dpi2point div scale neg translate
 dup landscapeMode eq {pop -90 rotate}
  {rotateMode eq {90 rotate} if} ifelse} bdef
% line types: solid, dotted, dashed, dotdash
/SO { [] 0 setdash } bdef
/DO { [.5 dpi2point mul 4 dpi2point mul] 0 setdash } bdef
/DA { [6 dpi2point mul] 0 setdash } bdef
/DD { [.5 dpi2point mul 4 dpi2point mul 6 dpi2point mul 4
  dpi2point mul] 0 setdash } bdef
% macros for lines and objects
/L {lineto stroke} bdef
/MP {3 1 roll moveto 1 sub {rlineto} repeat} bdef
/AP {{rlineto} repeat} bdef
/PDlw -1 def
/W {/PDlw currentlinewidth def setlinewidth} def
/PP {closepath eofill} bdef
/DP {closepath stroke} bdef
/MR {4 -2 roll moveto dup  0 exch rlineto exch 0 rlineto
  neg 0 exch rlineto closepath} bdef
/FR {MR stroke} bdef
/PR {MR fill} bdef
/L1i {{currentfile picstr readhexstring pop} image} bdef
/tMatrix matrix def
/MakeOval {newpath tMatrix currentmatrix pop translate scale
0 0 1 0 360 arc tMatrix setmatrix} bdef
/FO {MakeOval stroke} bdef
/PO {MakeOval fill} bdef
/PD {currentlinewidth 2 div 0 360 arc fill
   PDlw -1 eq not {PDlw w /PDlw -1 def} if} def
/FA {newpath tMatrix currentmatrix pop translate scale
  0 0 1 5 -2 roll arc tMatrix setmatrix stroke} bdef
/PA {newpath tMatrix currentmatrix pop	translate 0 0 moveto scale
  0 0 1 5 -2 roll arc closepath tMatrix setmatrix fill} bdef
/FAn {newpath tMatrix currentmatrix pop translate scale
  0 0 1 5 -2 roll arcn tMatrix setmatrix stroke} bdef
/PAn {newpath tMatrix currentmatrix pop translate 0 0 moveto scale
  0 0 1 5 -2 roll arcn closepath tMatrix setmatrix fill} bdef
/vradius 0 def /hradius 0 def /lry 0 def
/lrx 0 def /uly 0 def /ulx 0 def /rad 0 def
/MRR {/vradius xdef /hradius xdef /lry xdef /lrx xdef /uly xdef
  /ulx xdef newpath tMatrix currentmatrix pop ulx hradius add uly
  vradius add translate hradius vradius scale 0 0 1 180 270 arc 
  tMatrix setmatrix lrx hradius sub uly vradius add translate
  hradius vradius scale 0 0 1 270 360 arc tMatrix setmatrix
  lrx hradius sub lry vradius sub translate hradius vradius scale
  0 0 1 0 90 arc tMatrix setmatrix ulx hradius add lry vradius sub
  translate hradius vradius scale 0 0 1 90 180 arc tMatrix setmatrix
  closepath} bdef
/FRR {MRR stroke } bdef
/PRR {MRR fill } bdef
/MlrRR {/lry xdef /lrx xdef /uly xdef /ulx xdef /rad lry uly sub 2 div def
  newpath tMatrix currentmatrix pop ulx rad add uly rad add translate
  rad rad scale 0 0 1 90 270 arc tMatrix setmatrix lrx rad sub lry rad
  sub translate rad rad scale 0 0 1 270 90 arc tMatrix setmatrix
  closepath} bdef
/FlrRR {MlrRR stroke } bdef
/PlrRR {MlrRR fill } bdef
/MtbRR {/lry xdef /lrx xdef /uly xdef /ulx xdef /rad lrx ulx sub 2 div def
  newpath tMatrix currentmatrix pop ulx rad add uly rad add translate
  rad rad scale 0 0 1 180 360 arc tMatrix setmatrix lrx rad sub lry rad
  sub translate rad rad scale 0 0 1 0 180 arc tMatrix setmatrix
  closepath} bdef
/FtbRR {MtbRR stroke } bdef
/PtbRR {MtbRR fill } bdef
/stri 6 array def /dtri 6 array def
/smat 6 array def /dmat 6 array def
/tmat1 6 array def /tmat2 6 array def /dif 3 array def
/asub {/ind2 exch def /ind1 exch def dup dup
  ind1 get exch ind2 get sub exch } bdef
/tri_to_matrix {
  2 0 asub 3 1 asub 4 0 asub 5 1 asub
  dup 0 get exch 1 get 7 -1 roll astore } bdef
/compute_transform {
  dmat dtri tri_to_matrix tmat1 invertmatrix 
  smat stri tri_to_matrix tmat2 concatmatrix } bdef
/ds {stri astore pop} bdef
/dt {dtri astore pop} bdef
/db {2 copy /cols xdef /rows xdef mul dup 3 mul string
  currentfile 
  3 index 0 eq {/ASCIIHexDecode filter}
  {/ASCII85Decode filter 3 index 2 eq {/RunLengthDecode filter} if }
  ifelse exch readstring pop
  dup 0 3 index getinterval /rbmap xdef
  dup 2 index dup getinterval /gbmap xdef
  1 index dup 2 mul exch getinterval /bbmap xdef pop pop}bdef
/it {gs np dtri aload pop moveto lineto lineto cp c
  cols rows 8 compute_transform 
  rbmap gbmap bbmap true 3 colorimage gr}bdef
/il {newpath moveto lineto stroke}bdef
currentdict end def
%%EndProlog

%%BeginSetup
MathWorks begin

0 cap

end
%%EndSetup

%%Page: 1 1
%%BeginPageSetup
%%PageBoundingBox:  -104   125   717   667
MathWorks begin
bpage
%%EndPageSetup

%%BeginObject: obj1
bplot

/dpi2point 12 def
portraitMode -1248 8004 csm

    0     0  9852  6495 rc
85 dict begin %Colortable dictionary
/c0 { 0.000000 0.000000 0.000000 sr} bdef
/c1 { 1.000000 1.000000 1.000000 sr} bdef
/c2 { 0.900000 0.000000 0.000000 sr} bdef
/c3 { 0.000000 0.820000 0.000000 sr} bdef
/c4 { 0.000000 0.000000 0.800000 sr} bdef
/c5 { 0.910000 0.820000 0.320000 sr} bdef
/c6 { 1.000000 0.260000 0.820000 sr} bdef
/c7 { 0.000000 0.820000 0.820000 sr} bdef
c0
1 j
1 sg
   0    0 9853 6496 rf
6 w
0 5294 7635 0 0 -5294 1281 5781 4 MP
PP
-7635 0 0 5294 7635 0 0 -5294 1281 5781 5 MP stroke
4 w
DO
0 sg
1281 5781 mt 1281  487 L
1281  487 mt 1281  487 L
2553 5781 mt 2553  487 L
2553  487 mt 2553  487 L
3826 5781 mt 3826  487 L
3826  487 mt 3826  487 L
5098 5781 mt 5098  487 L
5098  487 mt 5098  487 L
6371 5781 mt 6371  487 L
6371  487 mt 6371  487 L
7643 5781 mt 7643  487 L
7643  487 mt 7643  487 L
8916 5781 mt 8916  487 L
8916  487 mt 8916  487 L
1281 5781 mt 8916 5781 L
8916 5781 mt 8916 5781 L
1281 5251 mt 8916 5251 L
8916 5251 mt 8916 5251 L
1281 4722 mt 8916 4722 L
8916 4722 mt 8916 4722 L
1281 4192 mt 8916 4192 L
8916 4192 mt 8916 4192 L
1281 3663 mt 8916 3663 L
8916 3663 mt 8916 3663 L
1281 3134 mt 8916 3134 L
8916 3134 mt 8916 3134 L
1281 2604 mt 8916 2604 L
8916 2604 mt 8916 2604 L
1281 2075 mt 8916 2075 L
8916 2075 mt 8916 2075 L
1281 1545 mt 8916 1545 L
8916 1545 mt 8916 1545 L
1281 1016 mt 8916 1016 L
8916 1016 mt 8916 1016 L
1281  487 mt 8916  487 L
8916  487 mt 8916  487 L
SO
6 w
1281 5781 mt 8916 5781 L
1281  487 mt 8916  487 L
1281 5781 mt 1281  487 L
8916 5781 mt 8916  487 L
1281 5781 mt 8916 5781 L
1281 5781 mt 1281  487 L
1281 5781 mt 1281 5742 L
1281  487 mt 1281  525 L
DO
1281 5781 mt 1281  487 L
1281  487 mt 1281  487 L
SO
1281 5781 mt 1281 5704 L
1281  487 mt 1281  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

1193 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

1326 5889 mt 
(0) s
1664 5781 mt 1664 5742 L
1664  487 mt 1664  525 L
DO
1664 5781 mt 1664  487 L
1664  487 mt 1664  487 L
SO
1888 5781 mt 1888 5742 L
1888  487 mt 1888  525 L
DO
1888 5781 mt 1888  487 L
1888  487 mt 1888  487 L
SO
2047 5781 mt 2047 5742 L
2047  487 mt 2047  525 L
DO
2047 5781 mt 2047  487 L
2047  487 mt 2047  487 L
SO
2170 5781 mt 2170 5742 L
2170  487 mt 2170  525 L
DO
2170 5781 mt 2170  487 L
2170  487 mt 2170  487 L
SO
2271 5781 mt 2271 5742 L
2271  487 mt 2271  525 L
DO
2271 5781 mt 2271  487 L
2271  487 mt 2271  487 L
SO
2356 5781 mt 2356 5742 L
2356  487 mt 2356  525 L
DO
2356 5781 mt 2356  487 L
2356  487 mt 2356  487 L
SO
2430 5781 mt 2430 5742 L
2430  487 mt 2430  525 L
DO
2430 5781 mt 2430  487 L
2430  487 mt 2430  487 L
SO
2495 5781 mt 2495 5742 L
2495  487 mt 2495  525 L
DO
2495 5781 mt 2495  487 L
2495  487 mt 2495  487 L
SO
2553 5781 mt 2553 5742 L
2553  487 mt 2553  525 L
DO
2553 5781 mt 2553  487 L
2553  487 mt 2553  487 L
SO
2553 5781 mt 2553 5704 L
2553  487 mt 2553  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

2465 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

2598 5889 mt 
(1) s
2936 5781 mt 2936 5742 L
2936  487 mt 2936  525 L
DO
2936 5781 mt 2936  487 L
2936  487 mt 2936  487 L
SO
3160 5781 mt 3160 5742 L
3160  487 mt 3160  525 L
DO
3160 5781 mt 3160  487 L
3160  487 mt 3160  487 L
SO
3319 5781 mt 3319 5742 L
3319  487 mt 3319  525 L
DO
3319 5781 mt 3319  487 L
3319  487 mt 3319  487 L
SO
3442 5781 mt 3442 5742 L
3442  487 mt 3442  525 L
DO
3442 5781 mt 3442  487 L
3442  487 mt 3442  487 L
SO
3543 5781 mt 3543 5742 L
3543  487 mt 3543  525 L
DO
3543 5781 mt 3543  487 L
3543  487 mt 3543  487 L
SO
3628 5781 mt 3628 5742 L
3628  487 mt 3628  525 L
DO
3628 5781 mt 3628  487 L
3628  487 mt 3628  487 L
SO
3702 5781 mt 3702 5742 L
3702  487 mt 3702  525 L
DO
3702 5781 mt 3702  487 L
3702  487 mt 3702  487 L
SO
3767 5781 mt 3767 5742 L
3767  487 mt 3767  525 L
DO
3767 5781 mt 3767  487 L
3767  487 mt 3767  487 L
SO
3826 5781 mt 3826 5742 L
3826  487 mt 3826  525 L
DO
3826 5781 mt 3826  487 L
3826  487 mt 3826  487 L
SO
3826 5781 mt 3826 5704 L
3826  487 mt 3826  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

3738 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

3871 5889 mt 
(2) s
4209 5781 mt 4209 5742 L
4209  487 mt 4209  525 L
DO
4209 5781 mt 4209  487 L
4209  487 mt 4209  487 L
SO
4433 5781 mt 4433 5742 L
4433  487 mt 4433  525 L
DO
4433 5781 mt 4433  487 L
4433  487 mt 4433  487 L
SO
4592 5781 mt 4592 5742 L
4592  487 mt 4592  525 L
DO
4592 5781 mt 4592  487 L
4592  487 mt 4592  487 L
SO
4715 5781 mt 4715 5742 L
4715  487 mt 4715  525 L
DO
4715 5781 mt 4715  487 L
4715  487 mt 4715  487 L
SO
4816 5781 mt 4816 5742 L
4816  487 mt 4816  525 L
DO
4816 5781 mt 4816  487 L
4816  487 mt 4816  487 L
SO
4901 5781 mt 4901 5742 L
4901  487 mt 4901  525 L
DO
4901 5781 mt 4901  487 L
4901  487 mt 4901  487 L
SO
4975 5781 mt 4975 5742 L
4975  487 mt 4975  525 L
DO
4975 5781 mt 4975  487 L
4975  487 mt 4975  487 L
SO
5040 5781 mt 5040 5742 L
5040  487 mt 5040  525 L
DO
5040 5781 mt 5040  487 L
5040  487 mt 5040  487 L
SO
5098 5781 mt 5098 5742 L
5098  487 mt 5098  525 L
DO
5098 5781 mt 5098  487 L
5098  487 mt 5098  487 L
SO
5098 5781 mt 5098 5704 L
5098  487 mt 5098  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

5010 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

5143 5889 mt 
(3) s
5481 5781 mt 5481 5742 L
5481  487 mt 5481  525 L
DO
5481 5781 mt 5481  487 L
5481  487 mt 5481  487 L
SO
5705 5781 mt 5705 5742 L
5705  487 mt 5705  525 L
DO
5705 5781 mt 5705  487 L
5705  487 mt 5705  487 L
SO
5864 5781 mt 5864 5742 L
5864  487 mt 5864  525 L
DO
5864 5781 mt 5864  487 L
5864  487 mt 5864  487 L
SO
5987 5781 mt 5987 5742 L
5987  487 mt 5987  525 L
DO
5987 5781 mt 5987  487 L
5987  487 mt 5987  487 L
SO
6088 5781 mt 6088 5742 L
6088  487 mt 6088  525 L
DO
6088 5781 mt 6088  487 L
6088  487 mt 6088  487 L
SO
6173 5781 mt 6173 5742 L
6173  487 mt 6173  525 L
DO
6173 5781 mt 6173  487 L
6173  487 mt 6173  487 L
SO
6247 5781 mt 6247 5742 L
6247  487 mt 6247  525 L
DO
6247 5781 mt 6247  487 L
6247  487 mt 6247  487 L
SO
6312 5781 mt 6312 5742 L
6312  487 mt 6312  525 L
DO
6312 5781 mt 6312  487 L
6312  487 mt 6312  487 L
SO
6371 5781 mt 6371 5742 L
6371  487 mt 6371  525 L
DO
6371 5781 mt 6371  487 L
6371  487 mt 6371  487 L
SO
6371 5781 mt 6371 5704 L
6371  487 mt 6371  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

6283 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

6416 5889 mt 
(4) s
6754 5781 mt 6754 5742 L
6754  487 mt 6754  525 L
DO
6754 5781 mt 6754  487 L
6754  487 mt 6754  487 L
SO
6978 5781 mt 6978 5742 L
6978  487 mt 6978  525 L
DO
6978 5781 mt 6978  487 L
6978  487 mt 6978  487 L
SO
7137 5781 mt 7137 5742 L
7137  487 mt 7137  525 L
DO
7137 5781 mt 7137  487 L
7137  487 mt 7137  487 L
SO
7260 5781 mt 7260 5742 L
7260  487 mt 7260  525 L
DO
7260 5781 mt 7260  487 L
7260  487 mt 7260  487 L
SO
7361 5781 mt 7361 5742 L
7361  487 mt 7361  525 L
DO
7361 5781 mt 7361  487 L
7361  487 mt 7361  487 L
SO
7446 5781 mt 7446 5742 L
7446  487 mt 7446  525 L
DO
7446 5781 mt 7446  487 L
7446  487 mt 7446  487 L
SO
7520 5781 mt 7520 5742 L
7520  487 mt 7520  525 L
DO
7520 5781 mt 7520  487 L
7520  487 mt 7520  487 L
SO
7585 5781 mt 7585 5742 L
7585  487 mt 7585  525 L
DO
7585 5781 mt 7585  487 L
7585  487 mt 7585  487 L
SO
7643 5781 mt 7643 5742 L
7643  487 mt 7643  525 L
DO
7643 5781 mt 7643  487 L
7643  487 mt 7643  487 L
SO
7643 5781 mt 7643 5704 L
7643  487 mt 7643  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

7555 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

7688 5889 mt 
(5) s
8026 5781 mt 8026 5742 L
8026  487 mt 8026  525 L
DO
8026 5781 mt 8026  487 L
8026  487 mt 8026  487 L
SO
8250 5781 mt 8250 5742 L
8250  487 mt 8250  525 L
DO
8250 5781 mt 8250  487 L
8250  487 mt 8250  487 L
SO
8409 5781 mt 8409 5742 L
8409  487 mt 8409  525 L
DO
8409 5781 mt 8409  487 L
8409  487 mt 8409  487 L
SO
8532 5781 mt 8532 5742 L
8532  487 mt 8532  525 L
DO
8532 5781 mt 8532  487 L
8532  487 mt 8532  487 L
SO
8633 5781 mt 8633 5742 L
8633  487 mt 8633  525 L
DO
8633 5781 mt 8633  487 L
8633  487 mt 8633  487 L
SO
8718 5781 mt 8718 5742 L
8718  487 mt 8718  525 L
DO
8718 5781 mt 8718  487 L
8718  487 mt 8718  487 L
SO
8792 5781 mt 8792 5742 L
8792  487 mt 8792  525 L
DO
8792 5781 mt 8792  487 L
8792  487 mt 8792  487 L
SO
8857 5781 mt 8857 5742 L
8857  487 mt 8857  525 L
DO
8857 5781 mt 8857  487 L
8857  487 mt 8857  487 L
SO
8916 5781 mt 8916 5742 L
8916  487 mt 8916  525 L
DO
8916 5781 mt 8916  487 L
8916  487 mt 8916  487 L
SO
8916 5781 mt 8916 5704 L
8916  487 mt 8916  563 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

8828 5963 mt 
(10) s
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 80 FMSR

8961 5889 mt 
(6) s
1281 5781 mt 1357 5781 L
8916 5781 mt 8839 5781 L
%%IncludeResource: font Helvetica
/Helvetica /ISOLatin1Encoding 120 FMSR

1180 5825 mt 
(0) s
1281 5251 mt 1357 5251 L
8916 5251 mt 8839 5251 L
1080 5295 mt 
(0.1) s
1281 4722 mt 1357 4722 L
8916 4722 mt 8839 4722 L
1080 4766 mt 
(0.2) s
1281 4192 mt 1357 4192 L
8916 4192 mt 8839 4192 L
1080 4236 mt 
(0.3) s
1281 3663 mt 1357 3663 L
8916 3663 mt 8839 3663 L
1080 3707 mt 
(0.4) s
1281 3134 mt 1357 3134 L
8916 3134 mt 8839 3134 L
1080 3178 mt 
(0.5) s
1281 2604 mt 1357 2604 L
8916 2604 mt 8839 2604 L
1080 2648 mt 
(0.6) s
1281 2075 mt 1357 2075 L
8916 2075 mt 8839 2075 L
1080 2119 mt 
(0.7) s
1281 1545 mt 1357 1545 L
8916 1545 mt 8839 1545 L
1080 1589 mt 
(0.8) s
1281 1016 mt 1357 1016 L
8916 1016 mt 8839 1016 L
1080 1060 mt 
(0.9) s
1281  487 mt 1357  487 L
8916  487 mt 8839  487 L
1180  531 mt 
(1) s
1281 5781 mt 8916 5781 L
1281  487 mt 8916  487 L
1281 5781 mt 1281  487 L
8916 5781 mt 8916  487 L
gs 1281 487 7636 5295 rc
/c8 { 0.000000 0.000000 1.000000 sr} bdef
c8
35 -7 34 0 35 -3 35 -7 35 -7 35 -4 35 0 35 0 
35 -3 35 -4 34 -4 35 0 35 0 35 -7 35 -3 35 -7 
35 -4 35 -7 34 -7 35 -7 35 -7 35 -15 35 0 35 -10 
35 -7 35 -7 35 0 34 -18 35 -14 35 -39 35 -15 35 -42 
35 -43 35 -25 35 -39 34 -50 35 -28 35 -36 35 -39 35 -35 
35 -39 35 -36 35 -82 34 -49 35 -36 35 -32 35 -50 35 -49 
35 -25 35 -29 35 -24 35 -22 34 -60 35 -43 35 -25 35 -32 
35 -17 35 -29 35 -35 35 -57 34 -46 35 -43 35 -53 35 -25 
35 -68 35 -81 35 -96 35 -71 35 -89 34 -86 35 -46 35 -92 
35 -100 35 -78 35 -71 35 -60 35 -89 34 -82 35 -71 35 -50 
35 -74 35 -61 35 -74 35 -53 35 -82 34 -61 35 -67 35 -71 
35 -36 35 -53 35 -57 35 -39 35 -35 35 -50 34 -36 35 -39 
35 -25 35 -64 35 -60 35 -57 35 -32 35 -25 34 -35 35 -50 
35 -32 35 -53 35 -46 35 -15 35 -57 35 -17 35 -32 34 -25 
35 -36 35 -49 35 -32 35 -47 35 -28 35 -28 35 -25 34 -46 
35 -15 35 -14 35 -18 35 -7 35 -21 35 -36 35 -28 35 -25 
34 -28 35 -7 35 -18 35 -11 35 -21 35 -32 35 -4 35 -7 
34 -7 35 -4 35 -71 35 -71 35 -227 35 -4 35 0 35 -21 
34 0 35 0 35 -3 35 0 35 0 35 0 35 0 35 0 
35 -18 34 0 35 0 35 -7 35 0 35 0 35 0 35 -4 
35 -3 34 0 35 -11 35 0 35 0 35 0 35 0 35 0 
35 0 35 0 34 0 35 0 35 0 35 0 35 0 35 0 
35 0 35 0 34 0 35 0 35 0 35 0 35 0 35 0 
35 0 35 0 34 0 35 0 35 0 35 -11 35 0 35 0 
35 0 35 0 35 0 34 0 35 0 35 0 35 0 1298 5763 200 MP stroke
gr

c8
0 sg
3447  392 mt 
(CDF of object size for 15 popular websites \(1490 total obje\
cts\)) s
 998 3729 mt  -90 rotate
(Cumulative Distibution) s
90 rotate
4592 6106 mt 
(Object Size \(bytes\)) s

end %%Color Dict

eplot
%%EndObject

epage
end

showpage

%%Trailer
%%EOF

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[dpreed]

[-- Attachment #1: Type: text/plain, Size: 12813 bytes --]


Why is the DNS PLR so high?  1% is pretty depressing.
 
Also, it seems odd to eliminate 19% of the content retrieval because the tail is fat and long rather than short.  Wouldn't it be better to have 1000 servers?
 
 


On Friday, April 18, 2014 2:15pm, "Greg White" <g.white@CableLabs.com> said:



> Dave,
> 
> We used the 25k object size for a short time back in 2012 until we had
> resources to build a more advanced model (appendix A).  I did a bunch of
> captures of real web pages back in 2011 and compared the object size
> statistics to models that I'd seen published.  Lognormal didn't seem to be
> *exactly* right, but it wasn't a bad fit to what I saw.  I've attached a
> CDF.
> 
> The choice of 4 servers was based somewhat on logistics, and also on a
> finding that across our data set, the average web page retrieved 81% of
> its resources from the top 4 servers.  Increasing to 5 servers only
> increased that percentage to 84%.
> 
> The choice of RTTs also came from the web traffic captures. I saw
> RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.
> 
> Much of this can be found in
> https://tools.ietf.org/html/draft-white-httpbis-spdy-analysis-00
> 
> In many of the cases that we've simulated, the packet drop probability is
> less than 1% for DNS packets.  In our web model, there are a total of 4
> servers, so 4 DNS lookups assuming none of the addresses are cached. If
> PLR = 1%, there would be a 3.9% chance of losing one or more DNS packets
> (with a resulting ~5 second additional delay on load time).  I've probably
> oversimplified this, but Kathie N. and I made the call that it would be
> significantly easier to just do this math than to build a dns
> implementation in ns2.  We've open sourced the web model (it's on Kathie's
> web page and will be part of ns2.36) with an encouragement to the
> community to improve on it.  If you'd like to port it to ns3 and add a dns
> model, that would be fantastic.
> 
> -Greg
> 
> 
> On 4/17/14, 3:07 PM, "Dave Taht" <dave.taht@gmail.com> wrote:
> 
> >On Thu, Apr 17, 2014 at 12:01 PM, William Chan (陈智昌)
> ><willchan@chromium.org> wrote:
> >> Speaking as the primary Chromium developer in charge of this relevant
> >>code,
> >> I would like to caution putting too much trust in the numbers
> >>generated. Any
> >> statistical claims about the numbers are probably unreasonable to make.
> >
> >Sigh. Other benchmarks such as the apache ("ab") benchmark
> >are primarily designed as stress testers for web servers, not as realistic
> >traffic. Modern web traffic has such a high level of dynamicism in it,
> >that static web page loads along any distribution, seem insufficient,
> >passive analysis of aggregated traffic "feels" incorrect relative to the
> >sorts of home and small business traffic I've seen, and so on.
> >
> >Famous papers, such as this one:
> >
> >http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-leland.pdf
> >
> >Seem possibly irrelevant to draw conclusions from given the kind
> >of data they analysed and proceeding from an incorrect model or
> >gut feel for how the network behaves today seems to be foolish.
> >
> >Even the most basic of tools, such as httping, had three basic bugs
> >that I found in a few minutes of trying to come up with some basic
> >behaviors yesterday:
> >
> >https://lists.bufferbloat.net/pipermail/bloat/2014-April/001890.html
> >
> >Those are going to be a lot easier to fix than diving into the chromium
> >codebase!
> >
> >There are very few tools worth trusting, and I am always dubious
> >of papers that publish results with unavailable tools and data. The only
> >tools I have any faith in for network analysis are netperf,
> >netperf-wrapper,
> >tcpdump and xplot.org, and to a large extent wireshark. Toke and I have
> >been tearing apart d-itg and I hope to one day be able to add that to
> >my trustable list... but better tools are needed!
> >
> >Tools that I don't have a lot of faith in include that, iperf, anything
> >written
> >in java or other high level languages, speedtest.net, and things like
> >shaperprobe.
> >
> >Have very little faith in ns2, slightly more in ns3, and I've been meaning
> >to look over the mininet and other simulators whenever I got some spare
> >time; the mininet results stanford gets seem pretty reasonable and I
> >adore their reproducing results effort. Haven't explored ndt, keep meaning
> >to...
> >
> >> Reasons:
> >> * We don't actively maintain this code. It's behind the command line
> >>flags.
> >> They are broken. The fact that it still results in numbers on the
> >>benchmark
> >> extension is an example of where unmaintained code doesn't have the UI
> >> disabled, even though the internal workings of the code fail to
> >>guarantee
> >> correct operation. We haven't disabled it because, well, it's
> >>unmaintained.
> >
> >As I mentioned I was gearing up for a hacking run...
> >
> >The vast majority of results I look at are actually obtained via
> >looking at packet captures. I mostly use benchmarks as abstractions
> >to see if they make some sense relative to the captures and tend
> >to compare different benchmarks against each other.
> >
> >I realize others don't go into that level of detail, so you have given
> >fair warning! In our case we used the web page benchmarker as
> >a means to try and rapidly acquire some typical distributions of
> >get and tcp stream requests from things like the alexa top 1000,
> >and as a way to A/B different aqm/packet scheduling setups.
> >
> >... but the only easily publishable results were from the benchmark
> >itself,
> >and we (reluctantly) only published one graph from all the work that
> >went into it 2+ years back and used it as a test driver for the famous
> >ietf video, comparing two identical boxes running it at the same time
> >under different network conditions:
> >
> >https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#IETF-demo-s
> >ide-by-side-of-a-normal-cable-modem-vs-fq_codel
> >
> >from what I fiddled with today, it is at least still useful for that?
> >
> >moving on...
> >
> >The web model in the cablelabs work doesn't look much like my captures,
> >in addition to not modeling dns at all, and using a smaller IW than google
> >it looks like this:
> >
> >>> Model single user web page download as follows:
> >
> >>> - Web page modeled as single HTML page + 100 objects spread evenly
> >>> across 4 servers. Web object sizes are currently fixed at 25 kB
> each,
> >>> whereas the initial HTML page is 100 kB. Appendix A provides an
> >>> alternative page model that may be explored in future work.
> >
> >Where what I see is a huge number of stuff that fits into a single
> >iw10 slow start episode and some level of pipelining on larger stuff, so
> >that a
> >large number of object sizes of less than 7k with a lightly tailed
> >distribution
> >outside of that makes more sense.
> >
> >(I'm not staring at appendix A right now, I'm under the impression
> > it was better)
> >
> >I certainly would like more suggestions for models and types
> >of web traffic, as well as simulation of https + pfs traffic,
> >spdy, quic, etc....
> >
> >>> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).
> >
> >Server RTTs from my own web history tend to be lower than 50ms.
> >
> >>> - Initial HTTP GET to retrieve a moderately sized object (100 kB
> HTML
> >>> page) from server 1.
> >
> >An initial GET to google fits into iw10 - it's about 7k.
> >
> >>> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP
> GETs
> >>> (via separate TCP connections), 6 connections each to 4 different
> >>> server nodes
> >
> >I usually don't see more than 15. and certainly not 25k sized objects.
> >
> > > - Once each individual HTTP GET completes, initiate a subsequent GET
> >> to the same server, until 25 objects have been retrieved from each
> >> server.
> >
> >
> >> * We don't make sure to flush all the network state in between runs, so
> >>if
> >> you're using that option, don't trust it to work.
> >
> >The typical scenario we used was a run against dozens or hundreds of urls,
> >capturing traffic, while varying network conditions.
> >
> >Regarded the first run as the most interesting.
> >
> >Can exit the browser and restart after a run like that.
> >
> >At moment, merely plan to use the tool primarily to survey various
> >web sites and load times while doing packet captures. Hope was
> >to get valid data from the network portion of the load, tho...
> >
> >> * If you have an advanced Chromium setup, this definitely does not
> >>work. I
> >> advise using the benchmark extension only with a separate Chromium
> >>profile
> >> for testing purposes. Our flushing of sockets, caches, etc does not
> >>actually
> >> work correctly when you use the Chromium multiprofile feature and also
> >>fails
> >> to flush lots of our other network caches.
> >
> >noted.
> >
> >
> >> * No one on Chromium really believes the time to paint numbers that we
> >> output :) It's complicated. Our graphics stack is complicated. The time
> >>from
> >
> >I actually care only about time-to-full layout as that's a core network
> >effect...
> >
> >> when Blink thinks it painted to when the GPU actually blits to the
> >>screen
> >> cannot currently be corroborated with any high degree of accuracy from
> >> within our code.
> >
> >> * It has not been maintained since 2010. It is quite likely there are
> >>many
> >> other subtle inaccuracies here.
> >
> >Grok.
> >
> >> In short, while you can expect it to give you a very high level
> >> understanding of performance issues, I advise against placing
> >>non-trivial
> >> confidence in the accuracy of the numbers generated by the benchmark
> >> extension. The fact that numbers are produced by the extension should
> >>not be
> >> treated as evidence that the extension actually functions correctly.
> >
> >OK, noted. Still delighted to be able to have a simple load generator
> >that exercises the browsers and generates some results, however
> >dubious.
> >
> >>
> >> Cheers.
> >>
> >>
> >> On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com>
> wrote:
> >>>
> >>> Getting a grip on real web page load time behavior in an age of
> >>> sharded websites,
> >>> dozens of dns lookups, javascript, and fairly random behavior in ad
> >>> services
> >>> and cdns against how a modern browsers behaves is very, very hard.
> >>>
> >>> it turns out if you run
> >>>
> >>> google-chrome --enable-benchmarking --enable-net-benchmarking
> >>>
> >>> (Mac users have to embed these options in their startup script - see
> >>>  http://www.chromium.org/developers/how-tos/run-chromium-with-flags
> )
> >>>
> >>> enable developer options and install and run the chrome web page
> >>> benchmarker,
> >>> (
> >>>
> >>>https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomah
> >>>ekjcahlbpccbgaopjll?hl=en
> >>> )
> >>>
> >>> that it works (at least for me, on a brief test of the latest
> chrome,
> >>>on
> >>> linux.
> >>> Can someone try windows and mac?)
> >>>
> >>> You can then feed in a list of urls to test against, and post
> process
> >>> the resulting .csv file to your hearts content. We used to use this
> >>> benchmark a lot while trying to characterise typical web behaviors
> >>> under aqm and packet scheduling systems under load. Running
> >>> it simultaneously with a rrul test or one of the simpler tcp upload
> or
> >>> download
> >>> tests in the rrul suite was often quite interesting.
> >>>
> >>> It turned out the doc has been wrong a while as to the name of the
> >>>second
> >>> command lnie option. I was gearing up mentally for having to look at
> >>> the source....
> >>>
> >>> http://code.google.com/p/chromium/issues/detail?id=338705
> >>>
> >>> /me happy
> >>>
> >>> --
> >>> Dave Täht
> >>>
> >>> Heartbleed POC on wifi campus networks with EAP auth:
> >>> http://www.eduroam.edu.au/advisory.html
> >>>
> >>> _______________________________________________
> >>> aqm mailing list
> >>> aqm@ietf.org
> >>> https://www.ietf.org/mailman/listinfo/aqm
> >>
> >>
> >
> >
> >
> >--
> >Dave Täht
> >
> >NSFW:
> >https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indec
> >ent.article
> >
> >_______________________________________________
> >aqm mailing list
> >aqm@ietf.org
> >https://www.ietf.org/mailman/listinfo/aqm
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 16836 bytes --]

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Greg White]

[-- Attachment #1: Type: text/plain, Size: 14707 bytes --]

In most cases it was *far* less than 1%.  In our DOCSIS 3.1 scenarios it is more like 1e-5 to 1e-4 (1e-3 worst case for the bufferbloated drop tail baseline) which means that implementing dns in the model would have affected on the order of 1 out of every 2500 page loads.

As with any model of reality, there is a tradeoff between complexity and fidelity, right? We chose 4 servers as being a reasonable and practical number.  Many of the most popular sites actually don't distribute content that widely (e.g. Google only used 3 servers in our measurements), and the worst case was 32 servers.   In reality every web page is different, and you could come up with an arbitrarily complex model and it still wouldn't match some version of reality.  In my view, we want to have something that is practical to implement and close enough to reality to be useful.  In the AQM-evaluation draft currently, the model is a single 700kB file served from a single server.

Greg

From: "dpreed@reed.com<mailto:dpreed@reed.com>" <dpreed@reed.com<mailto:dpreed@reed.com>>
Date: Friday, April 18, 2014 at 12:48 PM
To: Greg White <g.white@cablelabs.com<mailto:g.white@cablelabs.com>>
Cc: Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>>, "William Chan (陈智昌)" <willchan@chromium.org<mailto:willchan@chromium.org>>, "aqm@ietf.org<mailto:aqm@ietf.org>" <aqm@ietf.org<mailto:aqm@ietf.org>>, "cerowrt-devel@lists.bufferbloat.net<mailto:cerowrt-devel@lists.bufferbloat.net>" <cerowrt-devel@lists.bufferbloat.net<mailto:cerowrt-devel@lists.bufferbloat.net>>, bloat <bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>>
Subject: Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed


Why is the DNS PLR so high?  1% is pretty depressing.



Also, it seems odd to eliminate 19% of the content retrieval because the tail is fat and long rather than short.  Wouldn't it be better to have 1000 servers?






On Friday, April 18, 2014 2:15pm, "Greg White" <g.white@CableLabs.com<mailto:g.white@CableLabs.com>> said:


> Dave,
>
> We used the 25k object size for a short time back in 2012 until we had
> resources to build a more advanced model (appendix A). I did a bunch of
> captures of real web pages back in 2011 and compared the object size
> statistics to models that I'd seen published. Lognormal didn't seem to be
> *exactly* right, but it wasn't a bad fit to what I saw. I've attached a
> CDF.
>
> The choice of 4 servers was based somewhat on logistics, and also on a
> finding that across our data set, the average web page retrieved 81% of
> its resources from the top 4 servers. Increasing to 5 servers only
> increased that percentage to 84%.
>
> The choice of RTTs also came from the web traffic captures. I saw
> RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.
>
> Much of this can be found in
> https://tools.ietf.org/html/draft-white-httpbis-spdy-analysis-00
>
> In many of the cases that we've simulated, the packet drop probability is
> less than 1% for DNS packets. In our web model, there are a total of 4
> servers, so 4 DNS lookups assuming none of the addresses are cached. If
> PLR = 1%, there would be a 3.9% chance of losing one or more DNS packets
> (with a resulting ~5 second additional delay on load time). I've probably
> oversimplified this, but Kathie N. and I made the call that it would be
> significantly easier to just do this math than to build a dns
> implementation in ns2. We've open sourced the web model (it's on Kathie's
> web page and will be part of ns2.36) with an encouragement to the
> community to improve on it. If you'd like to port it to ns3 and add a dns
> model, that would be fantastic.
>
> -Greg
>
>
> On 4/17/14, 3:07 PM, "Dave Taht" <dave.taht@gmail.com<mailto:dave.taht@gmail.com>> wrote:
>
> >On Thu, Apr 17, 2014 at 12:01 PM, William Chan (陈智昌)
> ><willchan@chromium.org<mailto:willchan@chromium.org>> wrote:
> >> Speaking as the primary Chromium developer in charge of this relevant
> >>code,
> >> I would like to caution putting too much trust in the numbers
> >>generated. Any
> >> statistical claims about the numbers are probably unreasonable to make.
> >
> >Sigh. Other benchmarks such as the apache ("ab") benchmark
> >are primarily designed as stress testers for web servers, not as realistic
> >traffic. Modern web traffic has such a high level of dynamicism in it,
> >that static web page loads along any distribution, seem insufficient,
> >passive analysis of aggregated traffic "feels" incorrect relative to the
> >sorts of home and small business traffic I've seen, and so on.
> >
> >Famous papers, such as this one:
> >
> >http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-leland.pdf
> >
> >Seem possibly irrelevant to draw conclusions from given the kind
> >of data they analysed and proceeding from an incorrect model or
> >gut feel for how the network behaves today seems to be foolish.
> >
> >Even the most basic of tools, such as httping, had three basic bugs
> >that I found in a few minutes of trying to come up with some basic
> >behaviors yesterday:
> >
> >https://lists.bufferbloat.net/pipermail/bloat/2014-April/001890.html
> >
> >Those are going to be a lot easier to fix than diving into the chromium
> >codebase!
> >
> >There are very few tools worth trusting, and I am always dubious
> >of papers that publish results with unavailable tools and data. The only
> >tools I have any faith in for network analysis are netperf,
> >netperf-wrapper,
> >tcpdump and xplot.org, and to a large extent wireshark. Toke and I have
> >been tearing apart d-itg and I hope to one day be able to add that to
> >my trustable list... but better tools are needed!
> >
> >Tools that I don't have a lot of faith in include that, iperf, anything
> >written
> >in java or other high level languages, speedtest.net, and things like
> >shaperprobe.
> >
> >Have very little faith in ns2, slightly more in ns3, and I've been meaning
> >to look over the mininet and other simulators whenever I got some spare
> >time; the mininet results stanford gets seem pretty reasonable and I
> >adore their reproducing results effort. Haven't explored ndt, keep meaning
> >to...
> >
> >> Reasons:
> >> * We don't actively maintain this code. It's behind the command line
> >>flags.
> >> They are broken. The fact that it still results in numbers on the
> >>benchmark
> >> extension is an example of where unmaintained code doesn't have the UI
> >> disabled, even though the internal workings of the code fail to
> >>guarantee
> >> correct operation. We haven't disabled it because, well, it's
> >>unmaintained.
> >
> >As I mentioned I was gearing up for a hacking run...
> >
> >The vast majority of results I look at are actually obtained via
> >looking at packet captures. I mostly use benchmarks as abstractions
> >to see if they make some sense relative to the captures and tend
> >to compare different benchmarks against each other.
> >
> >I realize others don't go into that level of detail, so you have given
> >fair warning! In our case we used the web page benchmarker as
> >a means to try and rapidly acquire some typical distributions of
> >get and tcp stream requests from things like the alexa top 1000,
> >and as a way to A/B different aqm/packet scheduling setups.
> >
> >... but the only easily publishable results were from the benchmark
> >itself,
> >and we (reluctantly) only published one graph from all the work that
> >went into it 2+ years back and used it as a test driver for the famous
> >ietf video, comparing two identical boxes running it at the same time
> >under different network conditions:
> >
> >https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#IETF-demo-s
> >ide-by-side-of-a-normal-cable-modem-vs-fq_codel
> >
> >from what I fiddled with today, it is at least still useful for that?
> >
> >moving on...
> >
> >The web model in the cablelabs work doesn't look much like my captures,
> >in addition to not modeling dns at all, and using a smaller IW than google
> >it looks like this:
> >
> >>> Model single user web page download as follows:
> >
> >>> - Web page modeled as single HTML page + 100 objects spread evenly
> >>> across 4 servers. Web object sizes are currently fixed at 25 kB
> each,
> >>> whereas the initial HTML page is 100 kB. Appendix A provides an
> >>> alternative page model that may be explored in future work.
> >
> >Where what I see is a huge number of stuff that fits into a single
> >iw10 slow start episode and some level of pipelining on larger stuff, so
> >that a
> >large number of object sizes of less than 7k with a lightly tailed
> >distribution
> >outside of that makes more sense.
> >
> >(I'm not staring at appendix A right now, I'm under the impression
> > it was better)
> >
> >I certainly would like more suggestions for models and types
> >of web traffic, as well as simulation of https + pfs traffic,
> >spdy, quic, etc....
> >
> >>> - Server RTTs set as follows (20 ms, 30 ms, 50 ms, 100 ms).
> >
> >Server RTTs from my own web history tend to be lower than 50ms.
> >
> >>> - Initial HTTP GET to retrieve a moderately sized object (100 kB
> HTML
> >>> page) from server 1.
> >
> >An initial GET to google fits into iw10 - it's about 7k.
> >
> >>> - Once initial HTTP GET completes, initiate 24 simultaneous HTTP
> GETs
> >>> (via separate TCP connections), 6 connections each to 4 different
> >>> server nodes
> >
> >I usually don't see more than 15. and certainly not 25k sized objects.
> >
> > > - Once each individual HTTP GET completes, initiate a subsequent GET
> >> to the same server, until 25 objects have been retrieved from each
> >> server.
> >
> >
> >> * We don't make sure to flush all the network state in between runs, so
> >>if
> >> you're using that option, don't trust it to work.
> >
> >The typical scenario we used was a run against dozens or hundreds of urls,
> >capturing traffic, while varying network conditions.
> >
> >Regarded the first run as the most interesting.
> >
> >Can exit the browser and restart after a run like that.
> >
> >At moment, merely plan to use the tool primarily to survey various
> >web sites and load times while doing packet captures. Hope was
> >to get valid data from the network portion of the load, tho...
> >
> >> * If you have an advanced Chromium setup, this definitely does not
> >>work. I
> >> advise using the benchmark extension only with a separate Chromium
> >>profile
> >> for testing purposes. Our flushing of sockets, caches, etc does not
> >>actually
> >> work correctly when you use the Chromium multiprofile feature and also
> >>fails
> >> to flush lots of our other network caches.
> >
> >noted.
> >
> >
> >> * No one on Chromium really believes the time to paint numbers that we
> >> output :) It's complicated. Our graphics stack is complicated. The time
> >>from
> >
> >I actually care only about time-to-full layout as that's a core network
> >effect...
> >
> >> when Blink thinks it painted to when the GPU actually blits to the
> >>screen
> >> cannot currently be corroborated with any high degree of accuracy from
> >> within our code.
> >
> >> * It has not been maintained since 2010. It is quite likely there are
> >>many
> >> other subtle inaccuracies here.
> >
> >Grok.
> >
> >> In short, while you can expect it to give you a very high level
> >> understanding of performance issues, I advise against placing
> >>non-trivial
> >> confidence in the accuracy of the numbers generated by the benchmark
> >> extension. The fact that numbers are produced by the extension should
> >>not be
> >> treated as evidence that the extension actually functions correctly.
> >
> >OK, noted. Still delighted to be able to have a simple load generator
> >that exercises the browsers and generates some results, however
> >dubious.
> >
> >>
> >> Cheers.
> >>
> >>
> >> On Thu, Apr 17, 2014 at 10:49 AM, Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>>
> wrote:
> >>>
> >>> Getting a grip on real web page load time behavior in an age of
> >>> sharded websites,
> >>> dozens of dns lookups, javascript, and fairly random behavior in ad
> >>> services
> >>> and cdns against how a modern browsers behaves is very, very hard.
> >>>
> >>> it turns out if you run
> >>>
> >>> google-chrome --enable-benchmarking --enable-net-benchmarking
> >>>
> >>> (Mac users have to embed these options in their startup script - see
> >>> http://www.chromium.org/developers/how-tos/run-chromium-with-flags
> )
> >>>
> >>> enable developer options and install and run the chrome web page
> >>> benchmarker,
> >>> (
> >>>
> >>>https://chrome.google.com/webstore/detail/page-benchmarker/channimfdomah
> >>>ekjcahlbpccbgaopjll?hl=en
> >>> )
> >>>
> >>> that it works (at least for me, on a brief test of the latest
> chrome,
> >>>on
> >>> linux.
> >>> Can someone try windows and mac?)
> >>>
> >>> You can then feed in a list of urls to test against, and post
> process
> >>> the resulting .csv file to your hearts content. We used to use this
> >>> benchmark a lot while trying to characterise typical web behaviors
> >>> under aqm and packet scheduling systems under load. Running
> >>> it simultaneously with a rrul test or one of the simpler tcp upload
> or
> >>> download
> >>> tests in the rrul suite was often quite interesting.
> >>>
> >>> It turned out the doc has been wrong a while as to the name of the
> >>>second
> >>> command lnie option. I was gearing up mentally for having to look at
> >>> the source....
> >>>
> >>> http://code.google.com/p/chromium/issues/detail?id=338705
> >>>
> >>> /me happy
> >>>
> >>> --
> >>> Dave Täht
> >>>
> >>> Heartbleed POC on wifi campus networks with EAP auth:
> >>> http://www.eduroam.edu.au/advisory.html
> >>>
> >>> _______________________________________________
> >>> aqm mailing list
> >>> aqm@ietf.org<mailto:aqm@ietf.org>
> >>> https://www.ietf.org/mailman/listinfo/aqm
> >>
> >>
> >
> >
> >
> >--
> >Dave Täht
> >
> >NSFW:
> >https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indec
> >ent.article
> >
> >_______________________________________________
> >aqm mailing list
> >aqm@ietf.org<mailto:aqm@ietf.org>
> >https://www.ietf.org/mailman/listinfo/aqm
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net<mailto:Cerowrt-devel@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 20967 bytes --]

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Dave Taht]

On Fri, Apr 18, 2014 at 11:15 AM, Greg White <g.white@cablelabs.com> wrote:
> Dave,
>
> We used the 25k object size for a short time back in 2012 until we had
> resources to build a more advanced model (appendix A).  I did a bunch of
> captures of real web pages back in 2011 and compared the object size
> statistics to models that I'd seen published.  Lognormal didn't seem to be
> *exactly* right, but it wasn't a bad fit to what I saw.  I've attached a
> CDF.

That does seem a bit large on the initial 20%.

Hmm.

There is a second kind of major case, where you are
moving around on the same web property, and hopefully many
core portions of the web page(s) such as the css and javascript,
basic logos and other images, are cached. Caching is handled
two ways, one is to explicitly mark the data as cacheable for
a certain period, the other is an if-modified-since request,
which costs RTTs for setup and the query. I am
under the impression that we generally see a lot more of the
latter than the former these days.

> The choice of 4 servers was based somewhat on logistics, and also on a
> finding that across our data set, the average web page retrieved 81% of
> its resources from the top 4 servers.  Increasing to 5 servers only
> increased that percentage to 84%.
>
> The choice of RTTs also came from the web traffic captures. I saw
> RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.

Get a median?

My own stats are probably quite skewed lower from being in california,
and doing some tests from places like isc.org in redwood city, which
is insanely well
co-located.

> Much of this can be found in
> https://tools.ietf.org/html/draft-white-httpbis-spdy-analysis-00

Thx!

> In many of the cases that we've simulated, the packet drop probability is
> less than 1% for DNS packets.  In our web model, there are a total of 4

I think we have the ability to get a better number for dns loss now.

> servers, so 4 DNS lookups assuming none of the addresses are cached. If
> PLR = 1%, there would be a 3.9% chance of losing one or more DNS packets
> (with a resulting ~5 second additional delay on load time).  I've probably
> oversimplified this, but Kathie N. and I made the call that it would be
> significantly easier to just do this math than to build a dns
> implementation in ns2.


The specific thing I've been concerned about was not the probability of
a dns loss, although as you note the consequences are huge -
but the frequency and cost of a cache miss and the resulting fill.

This is a very simple namebench test against the alexa top 1000:

http://snapon.lab.bufferbloat.net/~d/namebench/namebench_2014-03-20_1255.html

This is a more comprehensive one taken against my own recent web history file.

http://snapon.lab.bufferbloat.net/~d/namebench/namebench_2014-03-24_1541.html

Both of these were taken against the default SQM system in cerowrt
against a cable modem, so you can
pretty safely assume the ~20ms (middle) knee in the curve is basically
based on physical
RTT to the nearest upstream DNS server.

And it's a benchmark so I don't generally believe in the relative hit
ratios vs a vs "normal traffic", but do think the baseline RTT, and
the knees in the curves in the cost of a miss and file are relevant.

(it's also not clear to me if all cable modems run a local dns server)

Recently simon kelly added support for gathering hit and miss
statistics to dnsmasq 2.69.

They can be obtained via  a simple dns lookup as answers to queries of
class CHAOS and type TXT in domain bind. The domain names are
cachesize.bind, insertions.bind, evictions.bind, misses.bind,
hits.bind, auth.bind and servers.bind. An example command to query
this, using the dig utility would be

dig +short chaos txt cachesize.bind

It would be very interesting to see the differences between dnsmasq
without DNSSEC, with DNSSEC and with DNSSEC and
--dnssec-check-unsigned (checking for proof of non-existence) - we've
been a bit concerned about the overheads of the last in particular.

Getting more elaborate stats (hit, miss, and fill costs) is under discussion.

> We've open sourced the web model (it's on Kathie's
> web page and will be part of ns2.36) with an encouragement to the
> community to improve on it.  If you'd like to port it to ns3 and add a dns
> model, that would be fantastic.

As part of the google summer of code I am signed up to mentor a
student with tom for the *codel related bits
in ns3, and certainly plan to get fingers dirty in the cablelabs drop,
and there was a very encouraging patch set distributed around for
tcp-cubic with hystart support recently as well as a halfway decent
802.11 mac emulation.

As usual, I have no funding, personally, to tackle the job, but I'll
do what I can anyway. It would be wonderful to finally have all the
ns2 and ns3 code mainlined for more people to use it.



-- 
Dave Täht

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Dave Taht]

> The specific thing I've been concerned about was not the probability of
> a dns loss, although as you note the consequences are huge -
> but the frequency and cost of a cache miss and the resulting fill.
>
> This is a very simple namebench test against the alexa top 1000:
>
> http://snapon.lab.bufferbloat.net/~d/namebench/namebench_2014-03-20_1255.html
>
> This is a more comprehensive one taken against my own recent web history file.
>
> http://snapon.lab.bufferbloat.net/~d/namebench/namebench_2014-03-24_1541.html
>
> Both of these were taken against the default SQM system in cerowrt
> against a cable modem, so you can
> pretty safely assume the ~20ms (middle) knee in the curve is basically
> based on physical
> RTT to the nearest upstream DNS server.
>
> And it's a benchmark so I don't generally believe in the relative hit
> ratios vs a vs "normal traffic", but do think the baseline RTT, and
> the knees in the curves in the cost of a miss and file are relevant.

To be utterly clear here, the ~20ms knee in this data is a variable dependent
on the RTT to the nearest upstream DNS server. Most - nearly all -
ISPs - have a hefty dns server in their co-location facility, but the
baseline physical RTT is dependent on the actual technology in use.

The actual RTT of an query is dependent on the outstanding queue
length in a single queue AQM, unless otherwise prioritized. If prioritized
(cerowrt's 3 band system does this for queries coming from the router)
I imagine the packet loss rate drops hugely, also.

To give an extreme example of the DNS rtt problem, dns lookups over
satellite links take
800+ms, and this is one reason why web proxy servers are so common
in such environments as the whole query is shipped to a local-to-the-internet
proxy server so as to avoid this rtt cost. This technique is of increasingly
limited value in an age of e2e encryption.

Also: recently we've seen increasing use of non-local or otherwise redirected
dns servers such as here

http://www.cnet.com/news/google-confirms-turkey-is-blocking-its-dns-service/

It would be a good research project for someone to catagorize typical
nearest-upstream DNS RTTs, the availability of local-to-site dns
servers, hit/miss ratios
in homes and small business, the cost of dnssec, etc.


-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Greg White]

On 4/18/14, 1:05 PM, "Dave Taht" <dave.taht@gmail.com> wrote:

>On Fri, Apr 18, 2014 at 11:15 AM, Greg White <g.white@cablelabs.com>
>wrote:
>>
>> The choice of RTTs also came from the web traffic captures. I saw
>> RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.
>
>Get a median?

Median value was 62ms.

>
>My own stats are probably quite skewed lower from being in california,
>and doing some tests from places like isc.org in redwood city, which
>is insanely well
>co-located.

Mine are probably skewed too. I was told that global median (at the time I
collected this data) was around 100ms.

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Dave Taht]

On Fri, Apr 18, 2014 at 1:41 PM, Greg White <g.white@cablelabs.com> wrote:
> On 4/18/14, 1:05 PM, "Dave Taht" <dave.taht@gmail.com> wrote:
>
>>On Fri, Apr 18, 2014 at 11:15 AM, Greg White <g.white@cablelabs.com>
>>wrote:
>>>
>>> The choice of RTTs also came from the web traffic captures. I saw
>>> RTTmin=16ms, RTTmean=53.8ms, RTTmax=134ms.
>>
>>Get a median?
>
> Median value was 62ms.
>
>>
>>My own stats are probably quite skewed lower from being in california,
>>and doing some tests from places like isc.org in redwood city, which
>>is insanely well
>>co-located.
>
> Mine are probably skewed too. I was told that global median (at the time I
> collected this data) was around 100ms.

Well, the future is already here, just not evenly distributed. Nearly every
sample I'd taken at the same time as from, almost entirely from major
cities, came in at under 70ms median.

It strikes me that a possibly useful metric would be object size vs RTT,
over time.

>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

Re: [Cerowrt-devel] [aqm] chrome web page benchmarker fixed

[Dave Taht]

Doug Orr recommended to us that we give

http://www.chromium.org/developers/telemetry

a shot in generating reproducible web traffic models.