[Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2.1: Type: text/html, Size: 546 bytes --]

[-- Attachment #2.2: fq_simplest_tcstab_8000_700a.png --]
[-- Type: image/png, Size: 200986 bytes --]

[-- Attachment #2.3: fq_simplest_htb_8000_700a.png --]
[-- Type: image/png, Size: 241826 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

Thank you. 

It would be useful know what the different traffic classes used by the tool are. BE == Best Effort.

Beyond that I suspect the only person who knows is its Danish author.



On 15 Aug 2013, at 02:53, Dave Taht <dave.taht@gmail.com> wrote:

> The server in germany is demo.tohojo.dk - it's tokes, don't abuse it too much
> 
> The server in england isn't even in dns: 176.58.107.8 2a01:7e00::f03c:91ff:feae:7028
> 
> The latter can't do more than 8Mbit up.
> 
> we had most of a deal with google mlabs to dramatically expand the coverage (100+ servers) but it got hung up on details.
> 
> 
> 
> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
> <fq_simplest_tcstab_8000_700a.png>
> <fq_simplest_htb_8000_700a.png>
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


[-- Attachment #2: Type: text/html, Size: 2221 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 3809 bytes --]

0) What cero version is this? I have a slightle optimization for codel in
3.10.6 that I'd hoped would improve < 4mbit behavior... basically it turns
off maxpacket (and was discussed earlier this month on the codel list) as
not being useful outside of the ns2 environment.

1) I kind of prefer graphs get stuck on a website somewhere, rather than
email. Having to approve big postings manually adds to the 10 spams I have
to deal with per day, per list.

We would certainly like to add an "upload this test" feature to rrul one
day, that captures more data about the user's configuration and the test
data (from both sides!), but that project and servers remain unfunded, and
toke's really busy with his masters thesis...

2) Test #1 at T+48 or so appears to have a glitch - either caused by local
traffic on the link or something else. The long diagonal lines that you see
are bugs in the python-matplotlib library, they are fixed in ubuntu 13.4
and latest versions of arch. The choppy resolution of the second graph in
each chart is due to the sample interval being kind of small relative to
the bandwidth and the RTT. That's sort of fixable, but it's readable
without it....

Moving to what I see here, you are approximately 50ms (?) or so from the
icei.org server which is located on the east coast of the US (NJ, in
linode's co-location facility) (services on this box are graciously donated
by the icei organization that has been working in the background to help
out in many ways)

The black line is an average of 4 streams in the first and second graphs in
each chart. So you can multiply by 4 to get a rough estimate of actual
bandwidth on this link, but you do have to factor in the measurement
streams (graph 3), and the overhead of acks in each direction (which is
usually 66 bytes every other packet for ipv4), which are hard to measure.
So you are showing 6Mbit of raw bandwidth down, and about 480 up. Factoring
in the ack overhead of the the down, into the up, gets pretty close to your
set limit of 700. You experienced packet loss at time T+6 (not unusual)
that killed the non-ping measurement flows. (some day in the future we will
add one way ping measurements and NOT stop measuring after the first loss)

(There are multiple other plot types. do a --list-plots rrul. You can
generate a new plot on the same data by taking the *json.gz file and
supplying a different output filename (-o filename.svg) and plot type (-p
totals)

I regard the cdf plots as the most useful, but ALWAYS check this main graph
type to see glitches. Otherwise a cdf can be very misleading.)

So in this test latency spikes by about 100ms. Why does it do that? Well,
you have to fit 6k bytes (4 1500 byte flows), + 122 bytes (2 acks), + 65
bytes (ping) into the queues, and at 700kb/second that queue is far less
than the default 5ms target we start with. a 1500 byte packet takes 13ms to
transmit at 1Mbit, so we are ending up here with sufficient "standing
queue" to

Frankly, it should, eventually, achieve a tcp window size that will reduce
the latency to something lower than 100ms, but it obviously isn't.
nfq_codel is "tighter", but who knows, we're still in very early days of
trying to optimize for these bandwidths, and, like I said, I just killed
the maxpacket thing which might help some. A longer test (-l 300) at this
rtt) might be more revealing.

Clear as mud?



On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:

>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #1.2: Type: text/html, Size: 4796 bytes --]

[-- Attachment #2: fq_simplest_htb_8000_700a.png --]
[-- Type: image/png, Size: 241826 bytes --]

[-- Attachment #3: fq_simplest_tcstab_8000_700a.png --]
[-- Type: image/png, Size: 200986 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2434 bytes --]

They are 802.11e markings. http://en.wikipedia.org/wiki/IEEE_802.11e-2005

BE = best effort
BK = background
VO = Voice
CS5 or VI = video (diffserv marking)

This test originated with the wifi work and then ended up being used for
tons more stuff.

And the specification is mine. I was delighted when toke ended up
implementing the about 1/3 the spec in 2 months flat. Unfortunately work
stalled out since then. We've added a few more interesting tests, but the
really hard ones (web and one way delay measurements) have been, well,
hard. I've been evaluating owamp's methods for one way delays. There is
also a prototype web interface lying around.

The last version of the specification can be seen here, and I do hope one
day we get to implementing all of it, and/or, me, revising it. I had hopes
to submit it to the ippm ietf group and/or the ITU.

https://github.com/dtaht/deBloat/blob/master/spec/rrule.doc?raw=true





On Wed, Aug 14, 2013 at 7:14 PM, Fred Stratton <fredstratton@imap.cc> wrote:

> Thank you.
>
> It would be useful know what the different traffic classes used by the
> tool are. BE == Best Effort.
>
> Beyond that I suspect the only person who knows is its Danish author.
>
>
>
> On 15 Aug 2013, at 02:53, Dave Taht <dave.taht@gmail.com> wrote:
>
> The server in germany is demo.tohojo.dk - it's tokes, don't abuse it too
> much
>
> The server in england isn't even in dns: 176.58.107.8
> 2a01:7e00::f03c:91ff:feae:7028
>
> The latter can't do more than 8Mbit up.
>
> we had most of a deal with google mlabs to dramatically expand the
> coverage (100+ servers) but it got hung up on details.
>
>
>
> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc>wrote:
>
>> <fq_simplest_tcstab_8000_700a.png>
>> <fq_simplest_htb_8000_700a.png>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: Type: text/html, Size: 4341 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 2594 bytes --]

Excellent. Thank you.


On 15 Aug 2013, at 03:21, Dave Taht <dave.taht@gmail.com> wrote:

> They are 802.11e markings. http://en.wikipedia.org/wiki/IEEE_802.11e-2005
> 
> BE = best effort
> BK = background
> VO = Voice
> CS5 or VI = video (diffserv marking)
> 
> This test originated with the wifi work and then ended up being used for tons more stuff. 
> 
> And the specification is mine. I was delighted when toke ended up implementing the about 1/3 the spec in 2 months flat. Unfortunately work stalled out since then. We've added a few more interesting tests, but the really hard ones (web and one way delay measurements) have been, well, hard. I've been evaluating owamp's methods for one way delays. There is also a prototype web interface lying around.
> 
> The last version of the specification can be seen here, and I do hope one day we get to implementing all of it, and/or, me, revising it. I had hopes to submit it to the ippm ietf group and/or the ITU.
> 
> https://github.com/dtaht/deBloat/blob/master/spec/rrule.doc?raw=true
> 
> 
> 
> 
> 
> On Wed, Aug 14, 2013 at 7:14 PM, Fred Stratton <fredstratton@imap.cc> wrote:
> Thank you. 
> 
> It would be useful know what the different traffic classes used by the tool are. BE == Best Effort.
> 
> Beyond that I suspect the only person who knows is its Danish author.
> 
> 
> 
> On 15 Aug 2013, at 02:53, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> The server in germany is demo.tohojo.dk - it's tokes, don't abuse it too much
>> 
>> The server in england isn't even in dns: 176.58.107.8 2a01:7e00::f03c:91ff:feae:7028
>> 
>> The latter can't do more than 8Mbit up.
>> 
>> we had most of a deal with google mlabs to dramatically expand the coverage (100+ servers) but it got hung up on details.
>> 
>> 
>> 
>> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>> <fq_simplest_tcstab_8000_700a.png>
>> <fq_simplest_htb_8000_700a.png>
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> 
>> 
>> 
>> -- 
>> Dave Täht
>> 
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


[-- Attachment #2: Type: text/html, Size: 4788 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

I do not intend to promulgate any data.


On 15 Aug 2013, at 03:25, Dave Taht <dave.taht@gmail.com> wrote:

> and the mlabs thing was off the record. damn it. 
> 
> 
> On Wed, Aug 14, 2013 at 7:24 PM, Dave Taht <dave.taht@gmail.com> wrote:
> I really didn't want those ips published. thanks a lot. Among other things, they are not well secured.
> 
> 
> On Wed, Aug 14, 2013 at 7:14 PM, Fred Stratton <fredstratton@imap.cc> wrote:
> Thank you. 
> 
> It would be useful know what the different traffic classes used by the tool are. BE == Best Effort.
> 
> Beyond that I suspect the only person who knows is its Danish author.
> 
> 
> 
> On 15 Aug 2013, at 02:53, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> The server in germany is demo.tohojo.dk - it's tokes, don't abuse it too much
>> 
>> The server in england isn't even in dns: 176.58.107.8 2a01:7e00::f03c:91ff:feae:7028
>> 
>> The latter can't do more than 8Mbit up.
>> 
>> we had most of a deal with google mlabs to dramatically expand the coverage (100+ servers) but it got hung up on details.
>> 
>> 
>> 
>> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>> <fq_simplest_tcstab_8000_700a.png>
>> <fq_simplest_htb_8000_700a.png>
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> 
>> 
>> 
>> -- 
>> Dave Täht
>> 
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


[-- Attachment #2: Type: text/html, Size: 4348 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]

I do not yet have sufficient paranoia about buffering.

I have 2 D-Link power line adaptors between the gateway device and the wndr 3800.

http://caia.swin.edu.au/reports/130417A/CAIA-TR-130417A.pdf

Perhaps I should attempt to position both before the power line link.

On 15 Aug 2013, at 03:25, Fred Stratton <fredstratton@imap.cc> wrote:

> Excellent. Thank you.
> 
> 
> On 15 Aug 2013, at 03:21, Dave Taht <dave.taht@gmail.com> wrote:
> 
>> They are 802.11e markings. http://en.wikipedia.org/wiki/IEEE_802.11e-2005
>> 
>> BE = best effort
>> BK = background
>> VO = Voice
>> CS5 or VI = video (diffserv marking)
>> 
>> This test originated with the wifi work and then ended up being used for tons more stuff. 
>> 
>> And the specification is mine. I was delighted when toke ended up implementing the about 1/3 the spec in 2 months flat. Unfortunately work stalled out since then. We've added a few more interesting tests, but the really hard ones (web and one way delay measurements) have been, well, hard. I've been evaluating owamp's methods for one way delays. There is also a prototype web interface lying around.
>> 
>> The last version of the specification can be seen here, and I do hope one day we get to implementing all of it, and/or, me, revising it. I had hopes to submit it to the ippm ietf group and/or the ITU.
>> 
>> https://github.com/dtaht/deBloat/blob/master/spec/rrule.doc?raw=true
>> 
>> 
>> 
>> 
>> 
>> On Wed, Aug 14, 2013 at 7:14 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>> Thank you. 
>> 
>> It would be useful know what the different traffic classes used by the tool are. BE == Best Effort.
>> 
>> Beyond that I suspect the only person who knows is its Danish author.
>> 
>> 
>> 
>> On 15 Aug 2013, at 02:53, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>>> The server in germany is demo.tohojo.dk - it's tokes, don't abuse it too much
>>> 
>>> The server in england isn't even in dns: 176.58.107.8 2a01:7e00::f03c:91ff:feae:7028
>>> 
>>> The latter can't do more than 8Mbit up.
>>> 
>>> we had most of a deal with google mlabs to dramatically expand the coverage (100+ servers) but it got hung up on details.
>>> 
>>> 
>>> 
>>> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>>> <fq_simplest_tcstab_8000_700a.png>
>>> <fq_simplest_htb_8000_700a.png>
>>> 
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Dave Täht
>>> 
>>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>> 
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> 
>> 
>> 
>> -- 
>> Dave Täht
>> 
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel


[-- Attachment #2: Type: text/html, Size: 5859 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

I see the error and apologise.

I shall stop posting.


On 15 Aug 2013, at 03:33, Dave Taht <dave.taht@gmail.com> wrote:

> you just did. twice. now it's on the record in the public archive.  now I gotta go do an security audit and lock down some stuff in chrooted jails and make sure the monitoring tools don't allow multiple test attempts. I DON'T have time to maintain these servers. that was the whole point of not talking about it. please don't reply to this mail. I have no idea why your last mails ended up going to cerowrt-devel as well!?
> 
> I'm sorry I told you where they were, and please feel free to setup your own netserver instances somewhere, and using these, but, jeeze, I got enough problems today...
> 
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


[-- Attachment #2: Type: text/html, Size: 1434 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

I have taken down the server in england until I have time to get it
configured properly. Sorry for the inconvienence.


On Wed, Aug 14, 2013 at 7:36 PM, Fred Stratton <fredstratton@imap.cc> wrote:

> I see the error and apologise.
>
> I shall stop posting.
>
>
> On 15 Aug 2013, at 03:33, Dave Taht <dave.taht@gmail.com> wrote:
>
> you just did. twice. now it's on the record in the public archive.  now I
> gotta go do an security audit and lock down some stuff in chrooted jails
> and make sure the monitoring tools don't allow multiple test attempts. I
> DON'T have time to maintain these servers. that was the whole point of not
> talking about it. please don't reply to this mail. I have no idea why your
> last mails ended up going to cerowrt-devel as well!?
>
> I'm sorry I told you where they were, and please feel free to setup your
> own netserver instances somewhere, and using these, but, jeeze, I got
> enough problems today...
>
>
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: Type: text/html, Size: 2375 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 5950 bytes --]

Hi Dave, hi Fred,


On Aug 15, 2013, at 04:15 , Dave Taht <dave.taht@gmail.com> wrote:

> 0) What cero version is this? I have a slightle optimization for codel in 3.10.6 that I'd hoped would improve < 4mbit behavior... basically it turns off maxpacket (and was discussed earlier this month on the codel list) as not being useful outside of the ns2 environment. 

	Interesting, I did my latest tests wit 3.10.6-1 and saw decent behavior for 14.7Mbit/s down, 2.4Mbit/s up, 40ms ping RTTs from 25-30ms unloaded, alas I have no comparison data for 3.10.1-1, as my router wiped out while I wanted to run those tests and I had to ref lash it via TFTP (and I decided to go to 3.10.6-1 directly, not knowing about the differences in fq_codel, even though I think you announced them somewhere)

> 
> 1) I kind of prefer graphs get stuck on a website somewhere, rather than email. Having to approve big postings manually adds to the 10 spams I have to deal with per day, per list. 

	I will look around to find a way to post things, would google+ work?

> 
> We would certainly like to add an "upload this test" feature to rrul one day, that captures more data about the user's configuration and the test data (from both sides!), but that project and servers remain unfunded, and toke's really busy with his masters thesis... 
> 
> 2) Test #1 at T+48 or so appears to have a glitch - either caused by local traffic on the link or something else. The long diagonal lines that you see are bugs in the python-matplotlib library, they are fixed in ubuntu 13.4 and latest versions of arch.

	Ah, but you can install matplotlib version 1.3 under ubuntu 12.04 in the terminal:
1) sudo apt-get build-dep python-matplotlib

2) potentially required:
	sudo pip install --upgrade freetype-py

3)  sudo pip install --upgrade matpltolib

(I might have forgotten a required step, so should anyone get stuck, just contact me)


> The choppy resolution of the second graph in each chart is due to the sample interval being kind of small relative to the bandwidth and the RTT. That's sort of fixable, but it's readable without it….

	Is there a simple way to fix this in netperf-wrapper?

> 
> Moving to what I see here, you are approximately 50ms (?) or so from the icei.org server which is located on the east coast of the US (NJ, in linode's co-location facility) (services on this box are graciously donated by the icei organization that has been working in the background to help out in many ways) 
> 
> The black line is an average of 4 streams in the first and second graphs in each chart. So you can multiply by 4 to get a rough estimate of actual bandwidth on this link, but you do have to factor in the measurement streams (graph 3), and the overhead of acks in each direction (which is usually 66 bytes every other packet for ipv4), which are hard to measure.

	Ah, so these ACKs will fill cause around 38 byte of padding in a packet of 3 ATM cells, leading to 26.4% increase in effective bandwidth used by the ATM stream versus what is send out over ge00 (ethernet). Together with the small ping and UDP RTT probes this explains nicely why proper link layer adjustments decrease ping time as well as decrease of the TCP rates. 

> So you are showing 6Mbit of raw bandwidth down, and about 480 up. Factoring in the ack overhead of the the down, into the up, gets pretty close to your set limit of 700. You experienced packet loss at time T+6 (not unusual) that killed the non-ping measurement flows. (some day in the future we will add one way ping measurements and NOT stop measuring after the first loss) 
> 
> (There are multiple other plot types. do a --list-plots rrul. You can generate a new plot on the same data by taking the *json.gz file and supplying a different output filename (-o filename.svg) and plot type (-p totals
> 
> I regard the cdf plots as the most useful, but ALWAYS check this main graph type to see glitches. Otherwise a cdf can be very misleading.)

	Ah, for one I can really relate, in MRI analysis the mantra to teach newcomers is always "look at your raw data".


> 
> So in this test latency spikes by about 100ms. Why does it do that? Well, you have to fit 6k bytes (4 1500 byte flows), + 122 bytes (2 acks), + 65 bytes (ping) into the queues, and at 700kb/second that queue is far less than the default 5ms target we start with. a 1500 byte packet takes 13ms to transmit at 1Mbit, so we are ending up here with sufficient "standing queue" to 
> 
> Frankly, it should, eventually, achieve a tcp window size that will reduce the latency to something lower than 100ms, but it obviously isn't. nfq_codel is "tighter", but who knows, we're still in very early days of trying to optimize for these bandwidths, and, like I said, I just killed the maxpacket thing which might help some. A longer test (-l 300) at this rtt) might be more revealing.


	So here is my result against "an unnamed netperf server in Germany" for 300seconds using 3.10.6-1 with simple.qos and fq_codel. Ping times only increase from ~30ms to 40ms (the same link gets ~300ms ping RTT without AQM, and ~80ms without proper linklayer adaptation). (Plus you can see my Macbook obviously is doing some periodically things (roughly every 15 seconds) that eats bandwidth and causes RTTs to increase a lot). 


> 
> Clear as mud?
> 
> 
> 
> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
> 
> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel


[-- Attachment #2.1: Type: text/html, Size: 7678 bytes --]

[-- Attachment #2.2: PastedGraphic-1.tiff --]
[-- Type: image/tiff, Size: 946958 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 9947 bytes --]

On Thu, Aug 15, 2013 at 2:32 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Dave, hi Fred,
>
>
> On Aug 15, 2013, at 04:15 , Dave Taht <dave.taht@gmail.com> wrote:
>
> 0) What cero version is this? I have a slightle optimization for codel in
> 3.10.6 that I'd hoped would improve < 4mbit behavior... basically it turns
> off maxpacket (and was discussed earlier this month on the codel list) as
> not being useful outside of the ns2 environment.
>
>
> Interesting, I did my latest tests wit 3.10.6-1 and saw decent behavior
> for 14.7Mbit/s down, 2.4Mbit/s up, 40ms ping RTTs from 25-30ms unloaded,
> alas I have no comparison data for 3.10.1-1, as my router wiped out while I
> wanted to run those tests and I had to ref lash it via TFTP (and I decided
> to go to 3.10.6-1 directly, not knowing about the differences in fq_codel,
> even though I think you announced them somewhere)
>
>
That's pretty good. The best you can do would be +10 ms (as you are adding
5ms in each direction).

I note that the nstat utility (under linux) will give a total number of
packets on the link and a variety of other statistics (that I mostly don't
understand), which will make it more possible to understand what else is
going on on that host, tcp's underlying behavior (packet loss, fast
recovery, ecn usage, etc (ecn tracking was just added to it, btw))

So future versions of rrul will probably run something like:

nstat > /dev/null # wipe out the statistics
some_rrul_related_test
nstat > saveitsomewhere

How to actually present the data? Damned if I know. Also have no idea if a
similar tool exists for other oses. It's using some semi-standard snmp
counters... might be possible to pull it off the router... dunno



>
> 1) I kind of prefer graphs get stuck on a website somewhere, rather than
> email. Having to approve big postings manually adds to the 10 spams I have
> to deal with per day, per list.
>
>
> I will look around to find a way to post things, would google+ work?
>
>
No tiffs, please? :) pngs are MUCH smaller, svg's show more detail....

>
> We would certainly like to add an "upload this test" feature to rrul one
> day, that captures more data about the user's configuration and the test
> data (from both sides!), but that project and servers remain unfunded, and
> toke's really busy with his masters thesis...
>
> 2) Test #1 at T+48 or so appears to have a glitch - either caused by local
> traffic on the link or something else. The long diagonal lines that you see
> are bugs in the python-matplotlib library, they are fixed in ubuntu 13.4
> and latest versions of arch.
>
>
> Ah, but you can install matplotlib version 1.3 under ubuntu 12.04 in the
> terminal:
> 1) sudo apt-get build-dep python-matplotlib
>
> 2) potentially required:
> sudo pip install --upgrade freetype-py
>
> 3)  sudo pip install --upgrade matpltolib
>
> (I might have forgotten a required step, so should anyone get stuck, just
> contact me)
>
>
good to know, thx.


>
> The choppy resolution of the second graph in each chart is due to the
> sample interval being kind of small relative to the bandwidth and the RTT.
> That's sort of fixable, but it's readable without it….
>
>
> Is there a simple way to fix this in netperf-wrapper?
>
>
Well, it kind of comes down to presenting raw data as raw data. The
academic and sysadm universe is in the habit of presenting highly processed
data, showing averages, eliminating the 95 percentile, etc, and by
constantly promoting looking hard at the raw data rather than the processed
stuff, I have hoped that at least I'd be able to consistently show people
that latency and bandwidth utilization are tightly interrelated, and that
raw data is very important when outliers are present - which they always
are, in networking.

As you noticed, you had a pattern that formed on an interval. I've seen
patterns all the time - one caused by cron every 60 seconds, another caused
by instruction traps on ipv6 - these turned out to be very important! -
I've seen other odd patterns that have formed on various intervals as well,
that were useful to understand.

And you see interesting patterns if you do things like run other traffic at
the same time as rrul. I'd be very interested if you ran the chrome web
page benchmarker against the alexa top 10 during a rrul or the simpler
tcp_upload or bidirectional tests on your setup (hint, hint)

An example of "smoothing" things that make me crazy are the five minute
averages that mrtg reports, when networks run at microsecond scales. At
least things like smokeping are a lot closer to being able to pickup on
outliers.

So, no, I'd rather not smooth that plot or change the sample interval (yes
you can change the sample interval)

I would like to have box plots one day.  Those can be genuinely useful but
also require a trained eye to read....

http://en.wikipedia.org/wiki/Box_plot


> Moving to what I see here, you are approximately 50ms (?) or so from the
> icei.org server which is located on the east coast of the US (NJ, in
> linode's co-location facility) (services on this box are graciously donated
> by the icei organization that has been working in the background to help
> out in many ways)
>
>
> The black line is an average of 4 streams in the first and second graphs
> in each chart. So you can multiply by 4 to get a rough estimate of actual
> bandwidth on this link, but you do have to factor in the measurement
> streams (graph 3), and the overhead of acks in each direction (which is
> usually 66 bytes every other packet for ipv4), which are hard to measure.
>
>
> Ah, so these ACKs will fill cause around 38 byte of padding in a packet of
> 3 ATM cells, leading to 26.4% increase in effective bandwidth used by the
> ATM stream versus what is send out over ge00 (ethernet). Together with the
> small ping and UDP RTT probes this explains nicely why proper link layer
> adjustments decrease ping time as well as decrease of the TCP rates.
>
>
yea, you are getting close to ideal. Could you re-run your setup set to
fred's settings? The atm etc mods won't kick in, but it would be
interesting to see if you have problems converging below 100ms...


> So you are showing 6Mbit of raw bandwidth down, and about 480 up.
> Factoring in the ack overhead of the the down, into the up, gets pretty
> close to your set limit of 700. You experienced packet loss at time T+6
> (not unusual) that killed the non-ping measurement flows. (some day in the
> future we will add one way ping measurements and NOT stop measuring after
> the first loss)
>
> (There are multiple other plot types. do a --list-plots rrul. You can
> generate a new plot on the same data by taking the *json.gz file and
> supplying a different output filename (-o filename.svg) and plot type (-p
> totals
>
>
> I regard the cdf plots as the most useful, but ALWAYS check this main
> graph type to see glitches. Otherwise a cdf can be very misleading.)
>
>
> Ah, for one I can really relate, in MRI analysis the mantra to teach
> newcomers is always "look at your raw data".
>
>
Outliers can kill you. Fact. I've pointed to frank rowands talks on this
subject a couple times.

The list of space launch failures due to stupid stuff like off by one bugs
and misplaced decimal points is rather high. And measurement error - or
worse, measuring the wrong thing - can mess up your whole day.

http://www.youtube.com/watch?v=2eGiqqoYP5E



>
>
> So in this test latency spikes by about 100ms. Why does it do that? Well,
> you have to fit 6k bytes (4 1500 byte flows), + 122 bytes (2 acks), + 65
> bytes (ping) into the queues, and at 700kb/second that queue is far less
> than the default 5ms target we start with. a 1500 byte packet takes 13ms to
> transmit at 1Mbit, so we are ending up here with sufficient "standing
> queue" to
>
> Frankly, it should, eventually, achieve a tcp window size that will reduce
> the latency to something lower than 100ms, but it obviously isn't.
> nfq_codel is "tighter", but who knows, we're still in very early days of
> trying to optimize for these bandwidths, and, like I said, I just killed
> the maxpacket thing which might help some. A longer test (-l 300) at this
> rtt) might be more revealing.
>
>
>
> So here is my result against "an unnamed netperf server in Germany" for
> 300seconds using 3.10.6-1 with simple.qos and fq_codel. Ping times only
> increase from ~30ms to 40ms (the same link gets ~300ms ping RTT without
> AQM, and ~80ms without proper linklayer adaptation). (Plus you can see my
> Macbook obviously is doing some periodically things (roughly every 15
> seconds) that eats bandwidth and causes RT
>

Excellent. If you re-run that test with "simple.qos" instead you can see
the classification classes, "doing something", at least on upload. On
download, if you don't see it "doing anything", it generally means that
your ToS values were stomped on in transit.



> Ts to increase a lot).
>

to get closer to an accurate value for traffic on the link, do the nstat
trick I mentioned above.


>
>
>
> Clear as mud?
>
>
>
> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc
> > wrote:
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>


-- 
Dave Täht

Fixing bufferbloat with cerowrt:
http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #1.2: Type: text/html, Size: 14435 bytes --]

[-- Attachment #2: PastedGraphic-1.tiff --]
[-- Type: image/tiff, Size: 946958 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 12863 bytes --]

Hi Dave, hi List,

On Aug 15, 2013, at 17:17 , Dave Taht <dave.taht@gmail.com> wrote:

> 
> 
> 
> On Thu, Aug 15, 2013 at 2:32 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dave, hi Fred,
> 
> 
> On Aug 15, 2013, at 04:15 , Dave Taht <dave.taht@gmail.com> wrote:
> 
>> 0) What cero version is this? I have a slightle optimization for codel in 3.10.6 that I'd hoped would improve < 4mbit behavior... basically it turns off maxpacket (and was discussed earlier this month on the codel list) as not being useful outside of the ns2 environment. 
> 
> 	Interesting, I did my latest tests wit 3.10.6-1 and saw decent behavior for 14.7Mbit/s down, 2.4Mbit/s up, 40ms ping RTTs from 25-30ms unloaded, alas I have no comparison data for 3.10.1-1, as my router wiped out while I wanted to run those tests and I had to ref lash it via TFTP (and I decided to go to 3.10.6-1 directly, not knowing about the differences in fq_codel, even though I think you announced them somewhere)
> 
> 
> That's pretty good. The best you can do would be +10 ms (as you are adding 5ms in each direction).

	I agree, it took a while to get things working right, but the results are, but with proper link layer adjustments DSL works nicely. There is one test left to do, shaping to <50% of link rate or below should eradicate the effect of proper link layer adjustments (worst case is 49 bytes of date using two full ATM cells worth 96 bytes of data, plus 10 bytes ATM cell overhead). The hypothesis being that with that shaping link layer adjustments will not change the ping RTT anymore, but I have not yet tested that...


> 
> I note that the nstat utility (under linux) will give a total number of packets on the link and a variety of other statistics (that I mostly don't understand), which will make it more possible to understand what else is going on on that host, tcp's underlying behavior (packet loss, fast recovery, ecn usage, etc (ecn tracking was just added to it, btw))

	Does not seem to exist under macosx, so until I get my linux MRI analysis machine installed and up and running I will not be able to peruse stat...

> 
> So future versions of rrul will probably run something like:
> 
> nstat > /dev/null # wipe out the statistics
> some_rrul_related_test
> nstat > saveitsomewhere
> 
> How to actually present the data? Damned if I know. Also have no idea if a similar tool exists for other oses. It's using some semi-standard snmp counters... might be possible to pull it off the router… dunno

	Sounds interesting and challenging and out of my league :)

> 
>  
>> 
>> 1) I kind of prefer graphs get stuck on a website somewhere, rather than email. Having to approve big postings manually adds to the 10 spams I have to deal with per day, per list. 
> 
> 	I will look around to find a way to post things, would google+ work?
> 
> 
> No tiffs, please? :) pngs are MUCH smaller, svg's show more detail…. 

	Argh, it seems that in deleting the hostnames I also changed the file format. I guess select copy and paste through the clipboard is a bit haphazard, will try to attach smaller plots in the future. (Just a thought the plots like they would be small and ceps in postscript...)


>> 
>> We would certainly like to add an "upload this test" feature to rrul one day, that captures more data about the user's configuration and the test data (from both sides!), but that project and servers remain unfunded, and toke's really busy with his masters thesis... 
>> 
>> 2) Test #1 at T+48 or so appears to have a glitch - either caused by local traffic on the link or something else. The long diagonal lines that you see are bugs in the python-matplotlib library, they are fixed in ubuntu 13.4 and latest versions of arch.
> 
> 	Ah, but you can install matplotlib version 1.3 under ubuntu 12.04 in the terminal:
> 1) sudo apt-get build-dep python-matplotlib
> 
> 2) potentially required:
> 	sudo pip install --upgrade freetype-py
> 
> 3)  sudo pip install --upgrade matpltolib
> 
> (I might have forgotten a required step, so should anyone get stuck, just contact me)
> 
> 
> good to know, thx.
>  
> 
>> The choppy resolution of the second graph in each chart is due to the sample interval being kind of small relative to the bandwidth and the RTT. That's sort of fixable, but it's readable without it….
> 
> 	Is there a simple way to fix this in netperf-wrapper?
> 
> 
> Well, it kind of comes down to presenting raw data as raw data. The academic and sysadm universe is in the habit of presenting highly processed data, showing averages, eliminating the 95 percentile, etc, and by constantly promoting looking hard at the raw data rather than the processed stuff, I have hoped that at least I'd be able to consistently show people that latency and bandwidth utilization are tightly interrelated, and that raw data is very important when outliers are present - which they always are, in networking.

	Ah, I realize that reading your sentence again would have helped, I was under the impression the uplink graph was under sampled-explaining the "missing bits", so went away with the impression all that would be needed would be a higher sampling rate… But you talked about the sampling interval,,, (in my defense I would always call this the sampling period, but that does not excuse not reading what you wrote...)

> 
> As you noticed, you had a pattern that formed on an interval. I've seen patterns all the time - one caused by cron every 60 seconds, another caused by instruction traps on ipv6 - these turned out to be very important! - I've seen other odd patterns that have formed on various intervals as well, that were useful to understand.
> 
> And you see interesting patterns if you do things like run other traffic at the same time as rrul. I'd be very interested if you ran the chrome web page benchmarker against the alexa top 10 during a rrul or the simpler tcp_upload or bidirectional tests on your setup (hint, hint)

	Ah, so far I refused to install chrome (mostly due to googles insistence to upgrade it whenever they see fit without asking for my permission), but I guess you ask, so I will try to do this (since the family is on visit for the weekend expect nothing before next week though...)

> 
> An example of "smoothing" things that make me crazy are the five minute averages that mrtg reports, when networks run at microsecond scales. At least things like smokeping are a lot closer to being able to pickup on outliers. 
> 
> So, no, I'd rather not smooth that plot or change the sample interval (yes you can change the sample interval)

	Yeah, once I actuelly understand you I am fully with you, decreasing sampling rate is not going to help (unless we initially sampled way hove 2*nyquist frequency :) )

> 
> I would like to have box plots one day.  Those can be genuinely useful but also require a trained eye to read….

	I agree Box wisker plots are quite challenging to full grasp, also they typically imply non-parametric tests (as otherwise mean plus confidence interval would be  easier to parse)

> 
> http://en.wikipedia.org/wiki/Box_plot 
> 
>> 
>> Moving to what I see here, you are approximately 50ms (?) or so from the icei.org server which is located on the east coast of the US (NJ, in linode's co-location facility) (services on this box are graciously donated by the icei organization that has been working in the background to help out in many ways) 
>> 
>> The black line is an average of 4 streams in the first and second graphs in each chart. So you can multiply by 4 to get a rough estimate of actual bandwidth on this link, but you do have to factor in the measurement streams (graph 3), and the overhead of acks in each direction (which is usually 66 bytes every other packet for ipv4), which are hard to measure.
> 
> 	Ah, so these ACKs will fill cause around 38 byte of padding in a packet of 3 ATM cells, leading to 26.4% increase in effective bandwidth used by the ATM stream versus what is send out over ge00 (ethernet). Together with the small ping and UDP RTT probes this explains nicely why proper link layer adjustments decrease ping time as well as decrease of the TCP rates. 
> 
> 
> yea, you are getting close to ideal. Could you re-run your setup set to fred's settings? The atm etc mods won't kick in, but it would be interesting to see if you have problems converging below 100ms…

	Oh, I did something close (using htb_private with AQM active at my setting, but a shorter run):

This basically hovers around 80ms in ping RTTs (but I get the same if I just use AQM to shape the up and downlink without taking overhead or link layer into account…) But that might not be the test you deem interesting, just let me know what exactlyy to run.


>  
>> So you are showing 6Mbit of raw bandwidth down, and about 480 up. Factoring in the ack overhead of the the down, into the up, gets pretty close to your set limit of 700. You experienced packet loss at time T+6 (not unusual) that killed the non-ping measurement flows. (some day in the future we will add one way ping measurements and NOT stop measuring after the first loss) 
>> 
>> (There are multiple other plot types. do a --list-plots rrul. You can generate a new plot on the same data by taking the *json.gz file and supplying a different output filename (-o filename.svg) and plot type (-p totals
>> 
>> I regard the cdf plots as the most useful, but ALWAYS check this main graph type to see glitches. Otherwise a cdf can be very misleading.)
> 
> 	Ah, for one I can really relate, in MRI analysis the mantra to teach newcomers is always "look at your raw data".
> 
> 
> Outliers can kill you. Fact. I've pointed to frank rowands talks on this subject a couple times.
> 
> The list of space launch failures due to stupid stuff like off by one bugs and misplaced decimal points is rather high. And measurement error - or worse, measuring the wrong thing - can mess up your whole day. 
> 
> http://www.youtube.com/watch?v=2eGiqqoYP5E
> 
>  
> 
>> 
>> So in this test latency spikes by about 100ms. Why does it do that? Well, you have to fit 6k bytes (4 1500 byte flows), + 122 bytes (2 acks), + 65 bytes (ping) into the queues, and at 700kb/second that queue is far less than the default 5ms target we start with. a 1500 byte packet takes 13ms to transmit at 1Mbit, so we are ending up here with sufficient "standing queue" to 
>> 
>> Frankly, it should, eventually, achieve a tcp window size that will reduce the latency to something lower than 100ms, but it obviously isn't. nfq_codel is "tighter", but who knows, we're still in very early days of trying to optimize for these bandwidths, and, like I said, I just killed the maxpacket thing which might help some. A longer test (-l 300) at this rtt) might be more revealing.
> 
> 
> 	So here is my result against "an unnamed netperf server in Germany" for 300seconds using 3.10.6-1 with simple.qos and fq_codel. Ping times only increase from ~30ms to 40ms (the same link gets ~300ms ping RTT without AQM, and ~80ms without proper linklayer adaptation). (Plus you can see my Macbook obviously is doing some periodically things (roughly every 15 seconds) that eats bandwidth and causes RT
> 
> Excellent. If you re-run that test with "simple.qos"

	But all I ever tested was simple.qos, I guess I should have mentioned that explicitly… Or do you mean simplest.qos?

> instead you can see the classification classes, "doing something", at least on upload. On download, if you don't see it "doing anything", it generally means that your ToS values were stomped on in transit.

	I would not be amazed though if the whole classification would not work well under macosx, my current only availablt OS...

> 
>  
> Ts to increase a lot). 
> 
> to get closer to an accurate value for traffic on the link, do the nstat trick I mentioned above.

	Good idea, I guess I should bring up my linux analysis machine soon, currently no stat under macosx.


Best Regards
	Sebastian


>  
> 
> 
> 
>> 
>> Clear as mud?
>> 
>> 
>> 
>> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>> 
>> 
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> 
>> 
>> 
>> -- 
>> Dave Täht
>> 
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html


[-- Attachment #2.1: Type: text/html, Size: 16189 bytes --]

[-- Attachment #2.2: htb_for_dave copy.jpeg --]
[-- Type: image/jpg, Size: 113765 bytes --]

Re: [Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 15223 bytes --]

Hi Dave hi List,

just a bit more information about the ATM encapsulation issue I wanted to share.

On Aug 16, 2013, at 21:46 , Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Dave, hi List,
> 
> On Aug 15, 2013, at 17:17 , Dave Taht <dave.taht@gmail.com> wrote:
> 
>> 
>> 
>> 
>> On Thu, Aug 15, 2013 at 2:32 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Dave, hi Fred,
>> 
>> 
>> On Aug 15, 2013, at 04:15 , Dave Taht <dave.taht@gmail.com> wrote:
>> 
>>> 0) What cero version is this? I have a slightle optimization for codel in 3.10.6 that I'd hoped would improve < 4mbit behavior... basically it turns off maxpacket (and was discussed earlier this month on the codel list) as not being useful outside of the ns2 environment. 
>> 
>> 	Interesting, I did my latest tests wit 3.10.6-1 and saw decent behavior for 14.7Mbit/s down, 2.4Mbit/s up, 40ms ping RTTs from 25-30ms unloaded, alas I have no comparison data for 3.10.1-1, as my router wiped out while I wanted to run those tests and I had to ref lash it via TFTP (and I decided to go to 3.10.6-1 directly, not knowing about the differences in fq_codel, even though I think you announced them somewhere)
>> 
>> 
>> That's pretty good. The best you can do would be +10 ms (as you are adding 5ms in each direction).
> 
> 	I agree, it took a while to get things working right, but the results are, but with proper link layer adjustments DSL works nicely. There is one test left to do, shaping to <50% of link rate or below should eradicate the effect of proper link layer adjustments (worst case is 49 bytes of date using two full ATM cells worth 96 bytes of data, plus 10 bytes ATM cell overhead). The hypothesis being that with that shaping link layer adjustments will not change the ping RTT anymore, but I have not yet tested that…

	So I finally got to run this test on cerowrt.3.10.6-1 with simple.qos; and as I assumed, at 50% shaping both the working tc_stab and the busted htb_private methods and even no link layer adjustments give a very similar ping RTT of 40 to 45 ms to Toke's German netsurf server. (In this artificial test both no link layer adjustment and the broken htb_prtivate one also giver higher average transfer bandwidth, just as to be expected from how the link layer adjustments should work).
	Since the tc stab mechanism, unlike the htb_private link layer adjustment mechanism does not fiddle with HTB rate tables directly, but with the apparent size of the packets it probably should be the option to switch to for link layer adjustments for cerowrt (and openwrt). The kernel already manipulates the apparent packet size to account for the ethernet headers (and more), so a method that simply fudges the kernels idea of the packet size a bit more seems more robust to change, as regressions in this functionality will affect many more users than the tiny fraction of DSL users that actually have sufficient control over their routers.
	So keeping the hopefully fixed htb_private method as a crosscheck for tc_stab might be reasonable (since at least in theory the kernel will carry this functionality for ever), but I digress.

	Also I think it is save to say that RRUL really is great tool for testing latency under load, just as you intended it to be :) And a working link layer adjustment has two hallmarks, reduced ping RTT under load over a real ATM link and reduced apparent  upload and download bandwidth (reduced by all the over head). So it should be possible to test linllayer adjustments easily and quickly without having to have an ATM link at hand, by just putting the net server on the wan side of the cerowrt box…

	So the nest thing to look at is the chrome tests that Dave wanted...

Best
	Sebastian




> 
> 
>> 
>> I note that the nstat utility (under linux) will give a total number of packets on the link and a variety of other statistics (that I mostly don't understand), which will make it more possible to understand what else is going on on that host, tcp's underlying behavior (packet loss, fast recovery, ecn usage, etc (ecn tracking was just added to it, btw))
> 
> 	Does not seem to exist under macosx, so until I get my linux MRI analysis machine installed and up and running I will not be able to peruse stat...
> 
>> 
>> So future versions of rrul will probably run something like:
>> 
>> nstat > /dev/null # wipe out the statistics
>> some_rrul_related_test
>> nstat > saveitsomewhere
>> 
>> How to actually present the data? Damned if I know. Also have no idea if a similar tool exists for other oses. It's using some semi-standard snmp counters... might be possible to pull it off the router… dunno
> 
> 	Sounds interesting and challenging and out of my league :)
> 
>> 
>>  
>>> 
>>> 1) I kind of prefer graphs get stuck on a website somewhere, rather than email. Having to approve big postings manually adds to the 10 spams I have to deal with per day, per list. 
>> 
>> 	I will look around to find a way to post things, would google+ work?
>> 
>> 
>> No tiffs, please? :) pngs are MUCH smaller, svg's show more detail…. 
> 
> 	Argh, it seems that in deleting the hostnames I also changed the file format. I guess select copy and paste through the clipboard is a bit haphazard, will try to attach smaller plots in the future. (Just a thought the plots like they would be small and ceps in postscript...)
> 
> 
>>> 
>>> We would certainly like to add an "upload this test" feature to rrul one day, that captures more data about the user's configuration and the test data (from both sides!), but that project and servers remain unfunded, and toke's really busy with his masters thesis... 
>>> 
>>> 2) Test #1 at T+48 or so appears to have a glitch - either caused by local traffic on the link or something else. The long diagonal lines that you see are bugs in the python-matplotlib library, they are fixed in ubuntu 13.4 and latest versions of arch.
>> 
>> 	Ah, but you can install matplotlib version 1.3 under ubuntu 12.04 in the terminal:
>> 1) sudo apt-get build-dep python-matplotlib
>> 
>> 2) potentially required:
>> 	sudo pip install --upgrade freetype-py
>> 
>> 3)  sudo pip install --upgrade matpltolib
>> 
>> (I might have forgotten a required step, so should anyone get stuck, just contact me)
>> 
>> 
>> good to know, thx.
>>  
>> 
>>> The choppy resolution of the second graph in each chart is due to the sample interval being kind of small relative to the bandwidth and the RTT. That's sort of fixable, but it's readable without it….
>> 
>> 	Is there a simple way to fix this in netperf-wrapper?
>> 
>> 
>> Well, it kind of comes down to presenting raw data as raw data. The academic and sysadm universe is in the habit of presenting highly processed data, showing averages, eliminating the 95 percentile, etc, and by constantly promoting looking hard at the raw data rather than the processed stuff, I have hoped that at least I'd be able to consistently show people that latency and bandwidth utilization are tightly interrelated, and that raw data is very important when outliers are present - which they always are, in networking.
> 
> 	Ah, I realize that reading your sentence again would have helped, I was under the impression the uplink graph was under sampled-explaining the "missing bits", so went away with the impression all that would be needed would be a higher sampling rate… But you talked about the sampling interval,,, (in my defense I would always call this the sampling period, but that does not excuse not reading what you wrote...)
> 
>> 
>> As you noticed, you had a pattern that formed on an interval. I've seen patterns all the time - one caused by cron every 60 seconds, another caused by instruction traps on ipv6 - these turned out to be very important! - I've seen other odd patterns that have formed on various intervals as well, that were useful to understand.
>> 
>> And you see interesting patterns if you do things like run other traffic at the same time as rrul. I'd be very interested if you ran the chrome web page benchmarker against the alexa top 10 during a rrul or the simpler tcp_upload or bidirectional tests on your setup (hint, hint)
> 
> 	Ah, so far I refused to install chrome (mostly due to googles insistence to upgrade it whenever they see fit without asking for my permission), but I guess you ask, so I will try to do this (since the family is on visit for the weekend expect nothing before next week though...)
> 
>> 
>> An example of "smoothing" things that make me crazy are the five minute averages that mrtg reports, when networks run at microsecond scales. At least things like smokeping are a lot closer to being able to pickup on outliers. 
>> 
>> So, no, I'd rather not smooth that plot or change the sample interval (yes you can change the sample interval)
> 
> 	Yeah, once I actuelly understand you I am fully with you, decreasing sampling rate is not going to help (unless we initially sampled way hove 2*nyquist frequency :) )
> 
>> 
>> I would like to have box plots one day.  Those can be genuinely useful but also require a trained eye to read….
> 
> 	I agree Box wisker plots are quite challenging to full grasp, also they typically imply non-parametric tests (as otherwise mean plus confidence interval would be  easier to parse)
> 
>> 
>> http://en.wikipedia.org/wiki/Box_plot 
>> 
>>> 
>>> Moving to what I see here, you are approximately 50ms (?) or so from the icei.org server which is located on the east coast of the US (NJ, in linode's co-location facility) (services on this box are graciously donated by the icei organization that has been working in the background to help out in many ways) 
>>> 
>>> The black line is an average of 4 streams in the first and second graphs in each chart. So you can multiply by 4 to get a rough estimate of actual bandwidth on this link, but you do have to factor in the measurement streams (graph 3), and the overhead of acks in each direction (which is usually 66 bytes every other packet for ipv4), which are hard to measure.
>> 
>> 	Ah, so these ACKs will fill cause around 38 byte of padding in a packet of 3 ATM cells, leading to 26.4% increase in effective bandwidth used by the ATM stream versus what is send out over ge00 (ethernet). Together with the small ping and UDP RTT probes this explains nicely why proper link layer adjustments decrease ping time as well as decrease of the TCP rates. 
>> 
>> 
>> yea, you are getting close to ideal. Could you re-run your setup set to fred's settings? The atm etc mods won't kick in, but it would be interesting to see if you have problems converging below 100ms…
> 
> 	Oh, I did something close (using htb_private with AQM active at my setting, but a shorter run):
> 
> This basically hovers around 80ms in ping RTTs (but I get the same if I just use AQM to shape the up and downlink without taking overhead or link layer into account…) But that might not be the test you deem interesting, just let me know what exactlyy to run.
> 
> 
>>  
>>> So you are showing 6Mbit of raw bandwidth down, and about 480 up. Factoring in the ack overhead of the the down, into the up, gets pretty close to your set limit of 700. You experienced packet loss at time T+6 (not unusual) that killed the non-ping measurement flows. (some day in the future we will add one way ping measurements and NOT stop measuring after the first loss) 
>>> 
>>> (There are multiple other plot types. do a --list-plots rrul. You can generate a new plot on the same data by taking the *json.gz file and supplying a different output filename (-o filename.svg) and plot type (-p totals
>>> 
>>> I regard the cdf plots as the most useful, but ALWAYS check this main graph type to see glitches. Otherwise a cdf can be very misleading.)
>> 
>> 	Ah, for one I can really relate, in MRI analysis the mantra to teach newcomers is always "look at your raw data".
>> 
>> 
>> Outliers can kill you. Fact. I've pointed to frank rowands talks on this subject a couple times.
>> 
>> The list of space launch failures due to stupid stuff like off by one bugs and misplaced decimal points is rather high. And measurement error - or worse, measuring the wrong thing - can mess up your whole day. 
>> 
>> http://www.youtube.com/watch?v=2eGiqqoYP5E
>> 
>>  
>> 
>>> 
>>> So in this test latency spikes by about 100ms. Why does it do that? Well, you have to fit 6k bytes (4 1500 byte flows), + 122 bytes (2 acks), + 65 bytes (ping) into the queues, and at 700kb/second that queue is far less than the default 5ms target we start with. a 1500 byte packet takes 13ms to transmit at 1Mbit, so we are ending up here with sufficient "standing queue" to 
>>> 
>>> Frankly, it should, eventually, achieve a tcp window size that will reduce the latency to something lower than 100ms, but it obviously isn't. nfq_codel is "tighter", but who knows, we're still in very early days of trying to optimize for these bandwidths, and, like I said, I just killed the maxpacket thing which might help some. A longer test (-l 300) at this rtt) might be more revealing.
>> 
>> 
>> 	So here is my result against "an unnamed netperf server in Germany" for 300seconds using 3.10.6-1 with simple.qos and fq_codel. Ping times only increase from ~30ms to 40ms (the same link gets ~300ms ping RTT without AQM, and ~80ms without proper linklayer adaptation). (Plus you can see my Macbook obviously is doing some periodically things (roughly every 15 seconds) that eats bandwidth and causes RT
>> 
>> Excellent. If you re-run that test with "simple.qos"
> 
> 	But all I ever tested was simple.qos, I guess I should have mentioned that explicitly… Or do you mean simplest.qos?
> 
>> instead you can see the classification classes, "doing something", at least on upload. On download, if you don't see it "doing anything", it generally means that your ToS values were stomped on in transit.
> 
> 	I would not be amazed though if the whole classification would not work well under macosx, my current only availablt OS...
> 
>> 
>>  
>> Ts to increase a lot). 
>> 
>> to get closer to an accurate value for traffic on the link, do the nstat trick I mentioned above.
> 
> 	Good idea, I guess I should bring up my linux analysis machine soon, currently no stat under macosx.
> 
> 
> Best Regards
> 	Sebastian
> 
> 
>>  
>> 
>> 
>> 
>>> 
>>> Clear as mud?
>>> 
>>> 
>>> 
>>> On Wed, Aug 14, 2013 at 2:28 PM, Fred Stratton <fredstratton@imap.cc> wrote:
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Dave Täht
>>> 
>>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> 
>> 
>> 
>> -- 
>> Dave Täht
>> 
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> 


[-- Attachment #2.1: Type: text/html, Size: 18617 bytes --]

[-- Attachment #2.2: htb_for_dave copy.jpeg --]
[-- Type: image/jpg, Size: 113765 bytes --]

[Cerowrt-devel] tc-stab versus htb on ADSL 8000/700

[Fred Stratton]

[-- Attachment #1: Type: text/plain, Size: 54 bytes --]



It would be good to be able to interpret these data.

[-- Attachment #2.1: Type: text/html, Size: 448 bytes --]

[-- Attachment #2.2: fq_simple_htb_8000_700a.png --]
[-- Type: image/png, Size: 220324 bytes --]