[Bloat] Progress with latency-under-load tool

[Bloat] Progress with latency-under-load tool

[Jonathan Morton]

I'm finally getting a handle on sockets programming again - the API has actually *changed* since I last used it - so the testing tool I was talking about is starting to actually do semi-useful things.

For example, just over the localhost "connection", maxRTT is already almost 2ms with no load.  That's CPU scheduling latency.  The tool prints this out as "Link Responsiveness: 556", since it displays Hz in order to give marketing-friendly "bigger is better" numbers.

Now to write a couple of lovely functions called "spew" and "chug".  I'll let you guess what they do...

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Rick Jones]

On Wed, 2011-03-16 at 04:31 +0200, Jonathan Morton wrote:
> I'm finally getting a handle on sockets programming again - the API has
>  actually *changed* since I last used it - so the testing tool I was
>  talking about is starting to actually do semi-useful things.
> 
> For example, just over the localhost "connection", maxRTT is already
>  almost 2ms with no load.  That's CPU scheduling latency.  The tool
>  prints this out as "Link Responsiveness: 556", since it displays Hz in
>  order to give marketing-friendly "bigger is better" numbers.
> 
> Now to write a couple of lovely functions called "spew" and "chug". 
>  I'll let you guess what they do...

FWIW, the output of two netperfs - one running RR the other
"spewing" (running STREAM).  Folks can probably guess when the STREAM
test started and ended:

raj@tardy:~/netperf2_trunk$ src/netperf -t omni -D 1 -H s7 -l 30 -- -d
rr
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to s7.cup.hp.com
(16.89.132.27) port 0 AF_INET : demo
Interim result: 3757.61 Trans/s over 1.00 seconds
Interim result: 3769.75 Trans/s over 1.00 seconds
Interim result: 3724.28 Trans/s over 1.01 seconds
Interim result: 3770.74 Trans/s over 1.00 seconds
Interim result: 3761.83 Trans/s over 1.00 seconds
Interim result: 3757.62 Trans/s over 1.00 seconds
Interim result: 3729.01 Trans/s over 1.01 seconds
Interim result: 3752.77 Trans/s over 1.00 seconds
Interim result: 3751.26 Trans/s over 1.00 seconds
Interim result: 2806.23 Trans/s over 1.34 seconds
Interim result:  903.67 Trans/s over 3.11 seconds
Interim result:  965.98 Trans/s over 1.03 seconds
Interim result:  729.16 Trans/s over 1.32 seconds
Interim result:  948.56 Trans/s over 1.00 seconds
Interim result:  678.77 Trans/s over 1.40 seconds
Interim result:  913.86 Trans/s over 1.00 seconds
Interim result: 1622.61 Trans/s over 1.00 seconds
Interim result: 3820.19 Trans/s over 1.00 seconds
Interim result: 3794.34 Trans/s over 1.01 seconds
Interim result: 3803.88 Trans/s over 1.00 seconds
Interim result: 3807.65 Trans/s over 1.00 seconds
Interim result: 3780.38 Trans/s over 1.01 seconds
Interim result: 3785.74 Trans/s over 1.00 seconds
Interim result: 3773.94 Trans/s over 1.00 seconds
Interim result: 3727.44 Trans/s over 1.01 seconds
Interim result: 3743.35 Trans/s over 1.00 seconds
Local       Local       Remote      Remote      Request Response Elapsed
Throughput Throughput 
Send Socket Recv Socket Recv Socket Send Socket Size    Size     Time
Units      
Size        Size        Size        Size        Bytes   Bytes
(sec)                         
Final       Final       Final
Final                                                      
16384       87380       87380       16384       1       1        30.00
2792.25    Trans/s    


raj@tardy:~/netperf2_trunk$ src/netperf -t omni -H s7 -D 1
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to s7.cup.hp.com
(16.89.132.27) port 0 AF_INET : demo
Interim result:  779.29 10^6bits/s over 1.01 seconds
Interim result:  739.22 10^6bits/s over 1.05 seconds
Interim result:  773.96 10^6bits/s over 1.00 seconds
Interim result:  771.44 10^6bits/s over 1.00 seconds
Interim result:  803.50 10^6bits/s over 1.00 seconds
Interim result:  807.18 10^6bits/s over 1.00 seconds
Interim result:  820.04 10^6bits/s over 1.00 seconds
Interim result:  831.19 10^6bits/s over 1.00 seconds
Interim result:  800.17 10^6bits/s over 1.04 seconds
Local       Local       Local  Elapsed Throughput Throughput  
Send Socket Send Socket Send   Time               Units       
Size        Size        Size   (sec)                          
Final       Final                                             
646400      646400      16384  10.00   793.36     10^6bits/s  

happy benchmarking,

rick jones
I'd never flog a dead horse :)

As for why it wasn't always interim results every second, that stems
from the algorithm being used to decide when to emit interim results
when ./configured with --enable-demo and -D is used - it tries to avoid
making a gettimeofday() call in front of every socket call, and so
guesses how many "operations" will complete in the demo interval.  Big
drops in the number of operations per interval will have those spikes of
longer intervals.

>  - Jonathan
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Progress with latency-under-load tool

[grenville armitage]

So, this is probably tangential to what the latency-under-load tool is
aiming to achieve, but we have a tool that can be used to measure
RTT between two points (e.g. either side of an 802.11 link) using
tcpdump on consumer-grade PCs and no active probing (ala ping).

http://caia.swin.edu.au/tools/spp/

Our SPP tool takes two tcpdump files (captured at two points in
the network) and seeks out 'packet pairs' observed passing both points.
 From these, it can calculate the RTT between the two measurement points.
So far, pretty boring and standard.  But it can be useful for people who:

  - only have consumer-grade PCs with only modestly synchronised clocks
	at the measurement points (i.e. that don't drift much in an RTT,
	and are within a few tens of seconds of each other in absolute time)

  - don't want to add active probe traffic to the network path being
	tested (e.g. measuring the RTT as experienced by actual application
	flows over 802.11 wireless links without additional ICMP probing)

  - Have asymmetric traffic flow between the two points (e.g. an online
	FPS game sending 50ms updates in one direction and 10-40ms updates
	in the other -- SPP uses a subset of the two-way traffic to calculate
	RTT between the measurement points)

Anyway, thought this might be of interested for anyone looking to measure the
latency across their home networking gear under load (assuming you can
tcpdump capture packets at two points either side of the box whose buffers
you're intrigued by).

cheers,
gja

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 18 Mar, 2011, at 1:38 am, grenville armitage wrote:

> So, this is probably tangential to what the latency-under-load tool is
> aiming to achieve, but we have a tool that can be used to measure
> RTT between two points (e.g. either side of an 802.11 link) using
> tcpdump on consumer-grade PCs and no active probing (ala ping).

The average actual consumer isn't going to be able to set that up.  Sounds great for research though.

After much headbutting of walls, my tool finally seems to work.  The biggest challenge turned out to be getting both ends to shut down gracefully (and more-or-less simultaneously) after each phase of the test.

With that solved, I'm now running a full test run between two of my less powerful machines on a wired LAN.  I can immediately see that even a 1996-vintage PC can saturate a 100Mbps link hard enough to exhibit bloat-related problems.  Once again, anyone claiming this to be a "new problem" is grossly misinformed.

More details when I have them.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.

It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.

It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.

There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.

The Capacity calculations are a bit complex to understand.  For each flow (in either direction), the average goodput is measured over the entire lifetime of the flow.  All of the flows in each direction for that scenario are the aggregated by taking the harmonic mean and multiplying by the number of flows; this biases the total downwards if the flows were mutually unfair.  Finally, the relevant measures across all scenarios are aggregated using the harmonic mean, thus biasing the overall measure towards cases of under-utilisation.  The totals are then reported in binary kilobytes per second.

The Link Responsiveness measure is much easier.  I simply take the maximum latency of the "pinger" channel (which is also used for commanding flows to stop) and invert it to produce a measure in Hertz.  This is rounded down to the next integer for display; if you see a zero here (which is surprisingly likely), it means that a latency of more than one second was encountered at some point during the entire test.

The Flow Smoothness measure refers to the application-level data inter-arrival timings.  The maximum delay between data chunks arriving is measured across all flows in all scenarios, but excludes the first megabyte of each flow so as to avoid picking up the RTT during TCP startup.

Here are some numbers I got from a test over a switched 100base-TX LAN:

    Upload Capacity: 1018 KiB/s
  Download Capacity: 2181 KiB/s
Link Responsiveness: 2 Hz
    Flow Smoothness: 1 Hz

Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.

 - Jonathan


[-- Attachment #2: loadlatency.tar.gz --]
[-- Type: application/x-gzip, Size: 6651 bytes --]

Re: [Bloat] Progress with latency-under-load tool

[grenville armitage]

On 03/20/2011 21:45, Jonathan Morton wrote:
	[..]
> Here are some numbers I got from a test over a switched 100base-TX LAN:
>
>      Upload Capacity: 1018 KiB/s
>    Download Capacity: 2181 KiB/s
> Link Responsiveness: 2 Hz
>      Flow Smoothness: 1 Hz
>
> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.

Perhaps a tangential 2 cents from me, but I'm unclear how helpful Hertz is as
a unit of measurement for the challenge of raising awareness of bufferbloat.
I suspect data networking people (whether network designers, admins, product
managers, etc) don't think of IP networks as having characteristics measured in
Hertz. And I'd imagine most ISP helpdesks wont know how to parse customer complaints
along the lines of "your service doesn't give me enough Hertz".  Perhaps your tool
could also add a parenthetical 'translation' of the responsiveness and smoothness
values into something expressed in milliseconds?

cheers,
gja

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

grenville armitage <garmitage@swin.edu.au> writes:

> On 03/20/2011 21:45, Jonathan Morton wrote:
> 	[..]
>> Here are some numbers I got from a test over a switched 100base-TX LAN:
>>
>>      Upload Capacity: 1018 KiB/s
>>    Download Capacity: 2181 KiB/s
>> Link Responsiveness: 2 Hz
>>      Flow Smoothness: 1 Hz
>>
>> Horrible, isn't it?  I deliberately left these machines with standard
>> configurations in order to show that.
>
> Perhaps a tangential 2 cents from me, but I'm unclear how helpful
> Hertz is as a unit of measurement for the challenge of raising
> awareness of bufferbloat.  I suspect data networking people (whether
> network designers, admins, product managers, etc) don't think of IP
> networks as having characteristics measured in Hertz. And I'd imagine
> most ISP helpdesks wont know how to parse customer complaints along
> the lines of "your service doesn't give me enough Hertz".  Perhaps
> your tool could also add a parenthetical 'translation' of the
> responsiveness and smoothness values into something expressed in
> milliseconds?

Concur. And resolution past 3 significant digits from there seems
possible with posix timers. I regret that I have not had time to look
over the code yet this weekend.

How'd they do debloated?

>
> cheers,
> gja
> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 20 Mar, 2011, at 10:33 pm, grenville armitage wrote:

>> Here are some numbers I got from a test over a switched 100base-TX LAN:
>> 
>>    Upload Capacity: 1018 KiB/s
>>  Download Capacity: 2181 KiB/s
>> Link Responsiveness: 2 Hz
>>    Flow Smoothness: 1 Hz
>> 
>> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.
> 
> Perhaps a tangential 2 cents from me, but I'm unclear how helpful Hertz is as
> a unit of measurement for the challenge of raising awareness of bufferbloat.

Customers won't be asking for "more Hertz" - or at least, none that have any sense.  They'll be asking for "more smoothness" for their video streams, or "more responsiveness" for their online games.  They already ask for "more bandwidth", not "more megabits per second".

Hertz makes sense as a unit because, when talking about latency or transmission delays, shorter times are better, but people understand "bigger is better" much more easily.  Hard drives used to measure their seek times in milliseconds too, but now they are measured in IOPS instead (a trend mostly associated with SSDs, which have IOPS numbers several orders of magnitude better than mechanical drives).

Let me manually translate that for you, though.  That Responsiveness rating of 2Hz means that the practical RTT went above 334ms - and this on a switched Fast Ethernet being driven by good-quality 1996-vintage hardware on one side and a cheap nettop on the other.  It actually reflects not pure latency as 'ping' would have measured it, but a packet loss and the time required to recover from it.

And the "smoothness" rating actually contrived to be *worse*, at somewhere north of 500ms.  At Fast Ethernet speeds, that implies megabytes of buffering, on what really is a very old computer.

It's a clear sign of something very broken in the network stack, especially as I get broadly similar results (with much higher throughput numbers) when I run the same test on GigE hardware with much more powerful computers (which can actually saturate GigE).

You really don't want to see what I got on my 3G test runs.  I got 0.09 Hz from a single flow, and these tests run all the way up to 32 flows.  I think the modem switched down into GPRS mode for a few minutes as well, even though there was no obvious change in propagation conditions.

> And resolution past 3 significant digits from there seems
> possible with posix timers.

If the latency was staying in the single-digit milliseconds as it should be on a LAN, you'd have three s.f. with just integers.  I do print the smoothness numbers out to 2 decimal places for the individual scenarios, though - these are the numbers meant for investigating problems:

Scenario 1: 0 uploads, 1 downloads... 1343 KiB/s down, 31.52 Hz smoothness
Scenario 2: 1 uploads, 0 downloads... 3670 KiB/s up, 22.24 Hz smoothness
Scenario 3: 0 uploads, 2 downloads... 2077 KiB/s down, 19.70 Hz smoothness
Scenario 4: 1 uploads, 1 downloads... 2855 KiB/s up, 322 KiB/s down, 6.44 Hz smoothness

That's from a different test, where you can see the effect of a WLAN and having Vegas on one side of the connection.

With that said, since the single-digit results are so ubiquitous, perhaps some extra precision is warranted after all.  Perhaps I can take the opportunity to squash some more minor bugs, and add an interpretation of the goodput in terms of gigabytes per month.

> How'd they do debloated?

I'm still investigating that, partly as the older hardware wasn't yet set up with kernels capable of running advanced qdiscs.  It takes a while to compile a kernel on a Pentium-MMX.  I'm also really not sure where to get debloated drivers for a VIA Rhine or a Sun GEM.  ;-)  Mind you, such a thing may not be needed, if the device drivers are already slim so that cutting txqueuelen is sufficient.

I can't run debloated 3G tests - I don't have access to the buffer controls on the base tower.  :-(

- Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 20 Mar, 2011, at 10:33 pm, grenville armitage wrote:
>
> Customers won't be asking for "more Hertz" - or at least, none that
> have any sense.  They'll be asking for "more smoothness" for their
> video streams, or "more responsiveness" for their online games.  They
> already ask for "more bandwidth", not "more megabits per second".

I think despite your quest for a marketing number, that also reporting
actual RTT would be helpful (and horrifying, as I just got a .02HZ
rating on the still ongoing test...). 

I still can't claim to fully understand what this test is measuring.

Scenario 8: 2 uploads, 1 downloads... 4743 KiB/s up, 3422 KiB/s down, 0.02 Hz smoothness
Scenario 9: 3 uploads, 0 downloads... 7558 KiB/s up, 1.39 Hz smoothness
Scenario 10: 0 uploads, 4 downloads... 6719 KiB/s down, 2.08 Hz smoothness
Scenario 11: 1 uploads, 3 downloads... 4372 KiB/s up, 4067 KiB/s down, 1.30 Hz smoothness

The interesting outlier thus far is 8... I'm tempted to stop the test
now and recompile for testing 3 u/l 3 d/l first....

Even better, we could use a different name for your usage of hz or
smoothness here. Mortons?

(There, I named it for you. You cannot be accused of ego for using this
 new unit now. Enjoy. )

> Hertz makes sense as a unit because, when talking about latency or
> transmission delays, shorter times are better, but people understand
> "bigger is better" much more easily.  Hard drives used to measure
> their seek times in milliseconds too, but now they are measured in
> IOPS instead (a trend mostly associated with SSDs, which have IOPS
> numbers several orders of magnitude better than mechanical drives).
>
> Let me manually translate that for you, though.  That Responsiveness
> rating of 2Hz means that the practical RTT went above 334ms - and this
> on a switched Fast Ethernet being driven by good-quality 1996-vintage
> hardware on one side and a cheap nettop on the other.  It actually
> reflects not pure latency as 'ping' would have measured it, but a
> packet loss and the time required to recover from it.

Your explanation here is good. Perhaps it could be an automated textual
description. 

> And the "smoothness" rating actually contrived to be *worse*, at
> somewhere north of 500ms.  At Fast Ethernet speeds, that implies
> megabytes of buffering, on what really is a very old computer.

Txqueuelen = 1000 = ~1.5Mbyte (and I'm struggling with using consistent
decimal or base 2 units this month) Mibibyte?

Normal dma tx ring size ranges from 64 to 512, can be seen sometimes
with ethtool -g device

> It's a clear sign of something very broken in the network stack,
> especially as I get broadly similar results (with much higher
> throughput numbers) when I run the same test on GigE hardware with
> much more powerful computers (which can actually saturate GigE).
>
> You really don't want to see what I got on my 3G test runs.  I got
> 0.09 Hz from a single flow, and these tests run all the way up to 32
> flows.  I think the modem switched down into GPRS mode for a few
> minutes as well, even though there was no obvious change in
> propagation conditions.
>

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

d@taht.net (Dave Täht) writes:

> The interesting outlier thus far is 8... I'm tempted to stop the test
> now and recompile for testing 3 u/l 3 d/l first....

Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness

... Still running tests ...

> Txqueuelen = 1000 = ~1.5Mbyte (and I'm struggling with using consistent
> decimal or base 2 units this month) Mibibyte?
>
> Normal dma tx ring size ranges from 64 to 512, can be seen sometimes
> with ethtool -g device

With wireless you also have TX_RETRIES = 13 for these (ath9) devices
drivers under test. (Reduced to 4 in the less bloated version I can't
test right now)

You also have contention from other devices, and the occasional complete
retrain from a base figure of 1Mbit/sec up, while devices struggle to
regain sync.

There are (presumably helpful) effects of packet aggregation.

You also have modulo issues against the size of the tx queues, an
interaction with the network flow scheduler (what triggers filling the
tx queue?), SACK/DSACK and lost ACK retransmits, and probably more
issues than I've been able to come up with offhand.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:

>> The interesting outlier thus far is 8... I'm tempted to stop the test
>> now and recompile for testing 3 u/l 3 d/l first....
> 
> Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness
> 
> ... Still running tests ...

I should probably explain what the scenarios are.

Every combination of up/down flows is tried for each of 1, 2, 3, 4 total flows.  That's 14 scenarios covering basic downloading and uploading through to heavy whole-family use.

Then I jump ahead to 32 flows, with 32/0, 31/1, 16/16, 1/31, 0/32 splits, to simulate BitTorrent style traffic.

Total 19 scenarios.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:
>
>>> The interesting outlier thus far is 8... I'm tempted to stop the test
>>> now and recompile for testing 3 u/l 3 d/l first....
>> 
>> Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s
>> down, 0.81 Hz smoothness
>> 
>> ... Still running tests ...
>
> I should probably explain what the scenarios are.
>
> Every combination of up/down flows is tried for each of 1, 2, 3, 4
> total flows.  That's 14 scenarios covering basic downloading and
> uploading through to heavy whole-family use.
>
> Then I jump ahead to 32 flows, with 32/0, 31/1, 16/16, 1/31, 0/32
> splits, to simulate BitTorrent style traffic.
>
> Total 19 scenarios.

I'll still argue that starting with the worst case scenarios and working
down would be best... 3/3 perhaps

Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness
Scenario 13: 3 uploads, 1 downloads... 4917 KiB/s up, 3175 KiB/s down, 0.59 Hz smoothness


>
>  - Jonathan
>

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:
>
>>> The interesting outlier thus far is 8... I'm tempted to stop the test
>>> now and recompile for testing 3 u/l 3 d/l first....


I found the cause of the outlier, the arm box had a hiccough on its
hard disk controller for some reason around then. Or at least that
appears to be the case, but judging from this:

Scenario 16: 1 uploads, 31 downloads... 4401 KiB/s up, 4111 KiB/s down, 0.59 Hz smoothness
Scenario 17: 16 uploads, 16 downloads... 6108 KiB/s up, 2103 KiB/s down,
0.00 Hz smoothness

Maybe not.

Me thinks we need 3 digits of precision here.



-- 
Dave Taht
http://nex-6.taht.net

[Bloat] Some results of the latency under load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> Here are some numbers I got from a test over a switched 100base-TX LAN:
>
>     Upload Capacity: 1018 KiB/s
>   Download Capacity: 2181 KiB/s
> Link Responsiveness: 2 Hz
>     Flow Smoothness: 1 Hz
>
> Horrible, isn't it?  I deliberately left these machines with standard
> configurations in order to show that.

I successfully compiled and ran this on both an debian squeeze/arm based
platform (an openrd), and a 64 bit ubuntu 10.10. It generated some
warnings on 64 bit.

Run on my partially debloated wireless/wired network[1][2] via the the
openrd->nano-m->nano-m->davepc path (180Mbit wireless) over IPv6 (wow,
transparent IPv6! Thanks!), over IPv6, server=davepc

Scenario 1: 0 uploads, 1 downloads... 8024 KiB/s down, 12.71 Hz smoothness
Scenario 2: 1 uploads, 0 downloads... 6526 KiB/s up, 5.52 Hz smoothness
Scenario 3: 0 uploads, 2 downloads... 6703 KiB/s down, 19.07 Hz smoothness
Scenario 4: 1 uploads, 1 downloads... 3794 KiB/s up, 4084 KiB/s down, 1.82 Hz smoothness
Scenario 5: 2 uploads, 0 downloads... 7503 KiB/s up, 4.11 Hz smoothness
Scenario 6: 0 uploads, 3 downloads... 7298 KiB/s down, 2.08 Hz smoothness
Scenario 7: 1 uploads, 2 downloads... 4323 KiB/s up, 4690 KiB/s down, 1.38 Hz smoothness

My pure wireless and more debloated network is down right now[2]

I'm still watching it run other scenarios.

Comments/Suggestions:

A) From a "quicker results" perspective, do the heaviest network tests first,
then the lightest tests, then the rest? :)

B) Due to the dependence on the gnu scientific library it seems unlikely
this will build on openwrt. Also, if there is heavy math anywhere in
this app, it needs to be done outside the testing loop on a non FPU
enabled processor such as most arms to not skew the test.

It ate 17% of cpu on the arm, and ~8% of cpu on davepc.

Similarly there are rng's in the Linux kernel and hardware sources, I'm
not sure to what extent you need randomness (I looked at the code  only briefly)

C) Do you have this in a repository somewhere? (I use github). I would
certainly love to have more tools like this living together in
harmony. Needn't be github.

D) It's never been clear to me what resolution gettimeofday has on most
platforms, and the posix clock_gettime and the corresponding

clock_getres(clockid_t clk_id, struct timespec *res)

Can make that clear. CLOCK_REALTIME also discards any clock adjustments
in the interval for even higher potential accuracy. (On the interval
anyway, not perhaps against wall-clock time) Given the period of this
test, do you think that the precision can only be 1ms?

It's certainly my hope that most systems today implement posix timers.

I can produce a patch if you want.

Thanks for such an interesting tool!

-- 
Dave Taht
http://nex-6.taht.net

[1] The openrd has a very small txqueuelen (4) and dma-tx-queue of 20 on
the GigE card, which is connected at 100Mbit to the nano-m5s, which DOES
NOT have my ath9k patch or eBDP enabled but does have a small
txqueuelen, the laptop is running debloat-testing (txqueulen 4, dma tx
queue 64), connected via 100Mbit to the other nano-m5 radio.

tcp cubic throughout. ECN and SACK/DSACK are enabled but there is no
shaping in place.

The wireless radios inbetween DO tend to show bloat under worse
conditions than this test.

(I note that debloat-testings patches to the e1000 series ethernet
driver do not allow dma tx queues of less than 64 as yet)

[2] for my network diagram see:
 http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700

This is an old test I need to repeat it with debloat testing.

Re: [Bloat] Some results of the latency under load tool

[Jonathan Morton]

On 20 Mar, 2011, at 11:50 pm, Dave Täht wrote:

> B) Due to the dependence on the gnu scientific library it seems unlikely
> this will build on openwrt. Also, if there is heavy math anywhere in
> this app, it needs to be done outside the testing loop on a non FPU
> enabled processor such as most arms to not skew the test.

I'm aware of the limitations of older ARM CPUs, as I have had to work with them on occasion.  I don't have any with Ethernet at home though.  The maths inside the main loops is limited to calculating time differences, comparing them and determining when to stop - there may be some optimisation possible there, but it's not ridiculous.

The main loops are actually slightly elastic, because recv() will give whatever data is available at the time it is called, and the fattest calculations are only done once per loop.  Under normal circumstances, that should be every 1.2 KB, but it scales up to 64KB if the CPU temporarily can't keep up (or if a lot of data becomes available in one go).

> It ate 17% of cpu on the arm, and ~8% of cpu on davepc.

Looks tolerable to me.  Even my ancient CISC PC can trigger the symptoms well enough.

> Similarly there are rng's in the Linux kernel and hardware sources, I'm
> not sure to what extent you need randomness (I looked at the code  only briefly)

The spew() function uses the PRNG to generate incompressible traffic, since compressibility does matter to some network technologies.  I used GSL purely to get access to a very fast PRNG that still produces good numbers - it's just a shame that this is a relatively large, general-purpose library.  I originally intended to checksum the data and thus ensure that it arrived intact, but I had to delete that feature to get around a race condition.

> So you are saying lower values of hz is horrible?

Yes.  0.02 Hz means the application was waiting 50 seconds for data somehow.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 21 Mar, 2011, at 6:09 am, Daniel Brooks wrote:

> Oof. I guess the loopback driver could use some work as well :)

Either that or there is some strange timing effect going on that I should be aware of.  I'm going to try to run some more tests to see if I can get the smoothness and responsiveness values up, but if any of you notice something bizarre...

Be aware however that both smoothness and responsiveness are taken as the absolute worst cases observed.  If there is a bell curve, you're going to be looking at outliers.  Perhaps I should calculate something like a 7-number summary to sanity-check it.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Kim Hawtin]

Hi Jonathan,

On 21/03/11 17:13, Jonathan Morton wrote:
 > On 21 Mar, 2011, at 6:09 am, Daniel Brooks wrote:
 >> Oof. I guess the loopback driver could use some work as well :)
 >
 > Either that or there is some strange timing effect going on that I 
should be aware of.  I'm going to try to run some more tests to see if I 
can get the smoothness and responsiveness values up, but if any of you 
notice something bizarre...
 >
 > Be aware however that both smoothness and responsiveness are taken as 
the absolute worst cases observed.  If there is a bell curve, you're 
going to be looking at outliers.  Perhaps I should calculate something 
like a 7-number summary to sanity-check it.
 >

dtaht pointed me in the direction of loadlatency.c to get my head around 
the issues. its been a while since i've done a lot of C, but had a poke 
around ...

It builds fine on Debian;
  $ uname -a
  Linux greyarea 2.6.32-5-amd64 #1 SMP Wed Jan 12 05:14:59 UTC 2011 
x86_64 GNU/Linux

but fails to link on RHEL5;
  $ uname -a
  Linux zima.services.adelaide.edu.au 2.6.18-194.11.3.el5 #1 SMP Mon Aug 
23 15:51:38 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

I have included the build and the YUM package info below.

dtaht suggested that the packages libblas and libgsp are used only for 
RNG? Is there another/better RNG that could be used that less system 
dependant?

Perhaps this one as it can be distributed directyly;
   http://ccan.ozlabs.org/info/isaac.html

regards,

Kim





$ gcc -O3 loadlatency.c -o loadlatency -lgsl -lblas -lpthread
loadlatency.c: In function ‘pingpong’:
loadlatency.c:140: warning: passing argument 2 of ‘send’ discards 
qualifiers from pointer target type
loadlatency.c:145: warning: passing argument 2 of ‘recv’ discards 
qualifiers from pointer target type
loadlatency.c: In function ‘spew_conn’:
loadlatency.c:346: warning: cast from pointer to integer of different size
loadlatency.c: In function ‘chug_conn’:
loadlatency.c:357: warning: cast from pointer to integer of different size
loadlatency.c: In function ‘pingpong_conn’:
loadlatency.c:368: warning: cast from pointer to integer of different size
loadlatency.c: In function ‘client’:
loadlatency.c:424: warning: cast to pointer from integer of different size
loadlatency.c:457: warning: cast to pointer from integer of different size
loadlatency.c:471: warning: passing argument 2 of ‘pthread_join’ from 
incompatible pointer type
loadlatency.c: In function ‘server_conn’:
loadlatency.c:514: warning: cast from pointer to integer of different size
loadlatency.c: In function ‘server’:
loadlatency.c:614: warning: cast to pointer from integer of different size
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ctrmv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zswap'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zsymm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cgeru'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sgemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ctrsv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sgemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_srotg'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zgemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cdotu_sub'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dznrm2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ddot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_csymm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cher2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zdotu_sub'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sdot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sdsdot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cgerc'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_scnrm2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ctrmm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_strmm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zcopy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsyrk'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cherk'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zherk'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zher'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ztrmv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_drotmg'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dswap'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_scopy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_scasum'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_drotg'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zdotc_sub'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_caxpy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zsyr2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssyr2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_csyr2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ztrsm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ctrsm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssymm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cdotc_sub'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zher2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_saxpy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_idamax'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_snrm2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsdot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cher'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dger'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cher2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dzasum'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsyr2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssyrk'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_strmv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dgemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cgemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dasum'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsymv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_srotm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sswap'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dtrmv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_isamax'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zhemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zaxpy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dcopy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sasum'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsyr'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_strsm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_drot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ztrsv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_izamax'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssymv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_chemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zgeru'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dtrsv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_sger'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_srot'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsymm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ztrmm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zdscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zher2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssyr'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dtrmm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zgerc'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_csscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dsyr2k'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_chemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dnrm2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_drotm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_icamax'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zgemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zhemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cgemv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ssyr2'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_strsv'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dscal'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dgemm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_srotmg'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_dtrsm'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_ccopy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_zsyrk'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_cswap'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_daxpy'
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib64/libgsl.so: 
undefined reference to `cblas_csyrk'
collect2: ld returned 1 exit status

this is the YUM package info;

# yum info gsl-devel.i386 gsl-devel.x86_64 gsl-static.x86_64 
blas-devel.i386 blas-devel.x86_64

Loaded plugins: rhnplugin, security
Installed Packages
Name       : blas-devel
Arch       : i386
Version    : 3.0
Release    : 37.el5
Size       : 456 k
Repo       : installed
Summary    : LAPACK development libraries
URL        : http://www.netlib.org/lapack/
License    : Freely distributable
Description: BLAS development libraries for applications that link 
statically.

Name       : blas-devel
Arch       : x86_64
Version    : 3.0
Release    : 37.el5
Size       : 580 k
Repo       : installed
Summary    : LAPACK development libraries
URL        : http://www.netlib.org/lapack/
License    : Freely distributable
Description: BLAS development libraries for applications that link 
statically.

Name       : gsl-devel
Arch       : i386
Version    : 1.13
Release    : 3.el5
Size       : 1.2 M
Repo       : installed
Summary    : Static libraries and header files for GSL development.
URL        : http://www.gnu.org/software/gsl/
License    : GPLv3 and GFDL and BSD
Description: The gsl-devel package contains the header files necessary for
            : developing programs using the GSL (GNU Scientific Library).

Name       : gsl-devel
Arch       : x86_64
Version    : 1.13
Release    : 3.el5
Size       : 1.2 M
Repo       : installed
Summary    : Static libraries and header files for GSL development.
URL        : http://www.gnu.org/software/gsl/
License    : GPLv3 and GFDL and BSD
Description: The gsl-devel package contains the header files necessary for
            : developing programs using the GSL (GNU Scientific Library).

Name       : gsl-static
Arch       : x86_64
Version    : 1.13
Release    : 3.el5
Size       : 4.3 M
Repo       : installed
Summary    : Static libraries of the GSL package
URL        : http://www.gnu.org/software/gsl/
License    : GPLv3 and GFDL and BSD
Description: The gsl-static package includes static libraries of GSL.

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 22 Mar, 2011, at 3:13 am, Kim Hawtin wrote:

> Summary    : LAPACK development libraries

This suggests that you should replace -lblas with -llapack.  I also have to do this for Cygwin.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Otto Solares Cabrera]

[-- Attachment #1: Type: text/plain, Size: 2072 bytes --]

On Sun, Mar 20, 2011 at 12:45:10PM +0200, Jonathan Morton wrote:
> Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.
> 
> It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.
> 
> It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.
> 
> There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.

Hello Jonathan,

Excellent tool! hopefully with more testing we can validate it's
results and improve it as testing all those scenarios seems the
correct path to understand bufferbloat and latency in our networks.

Sadly it's very difficult for a netadmin like me to follow your "units"
as the world have been standarized on bits per second in base10 or
decimal as opposed to the bytes per second in base2 or binary as
reported by this tool, "smoothness" too is a very radical unit to
measure latency which could be in milliseconds as the tool name implies.

Please don't take it as an offense but I cooked this small patch to
address this problems in the case you want to change it or for others
if they feel the need for more "normal" measuring units which in turn
can lead to more people testing their networks with this tool.

It's a diff against nbd's git repository, it seems to work for me but
honestly I have yet to figure out all the data it reports, hopefully
I didn't b0rk it too much :)
-
 Otto

[-- Attachment #2: loadlatency_units.patch --]
[-- Type: text/x-diff, Size: 6398 bytes --]

--- loadlatency.orig/loadlatency.c	2011-03-23 03:27:16.171982917 -0600
+++ loadlatency/loadlatency.c	2011-03-23 03:43:15.981962124 -0600
@@ -144,7 +144,7 @@
 			}
 
 			if(!settled && numPings > 16 && now-firstPing > 5.0) {
-				printf("MinRTT: %.1fms\n", minRTT * 1000);
+				printf("MinRTT: %.1f ms\n", minRTT * 1000);
 				settled = 1;
 			}
 		} else {
@@ -185,8 +185,8 @@
 typedef struct
 {
 //	uint32_t crc;
-	uint32_t Bps;    // bytes per second, will saturate at 4GB/s per flow.
-	uint32_t smooth; // fixed point 24.8, units are Hz
+	uint32_t bps;     // bits per second, will saturate at 32Gb/s per flow.
+	float    latency; // milliseconds
 } stats_packet;
 
 static void* spew(int sockfd, bool server)
@@ -227,12 +227,12 @@
 		goto bail;
 	}
 //	stats->crc = ntohl(stats->crc);
-	stats->Bps = ntohl(stats->Bps);
-	stats->smooth = ntohl(stats->smooth);
+	stats->bps = ntohl(stats->bps);
+	stats->latency = (float) ntohl((uint32_t) stats->latency);
 //	if(stats->crc != ~crc)
 //		memset(stats, 0, sizeof(stats_packet));
 
-//	printf("Raw stats received: Bps=%u smooth=%u\n", stats->Bps, stats->smooth);
+//	printf("Raw stats received: bps=%u latency=%.1f\n", stats->bps, stats->latency);
 
 bail:
 	close(sockfd);
@@ -244,7 +244,7 @@
 	uint8_t fid = get_flow_id();
 	uint8_t buffer[65536];
 //	uint32_t crc = ~0;
-	uint64_t bytes = 0;
+	uint64_t bits = 0;
 	stats_packet *stats = calloc(sizeof(stats), 1);
 	double startTime = gettime(), finishTime = 0;
 	double now = 0, then = 0;
@@ -262,11 +262,11 @@
 			break;
 
 //		crc = crc32(crc, buffer, rv);
-		bytes += rv;
+		bits += rv*8;
 
-		goodput = bytes / (now - startTime);
+		goodput = bits / (now - startTime);
 
-		if(bytes > 1024*1024) {
+		if(bits*8 > 1000*1000) {
 			double wait = now-then;
 			if(wait > maxWait) {
 				maxWait = wait;
@@ -284,10 +284,10 @@
 		if(goodput > maxGoodput) {
 			maxGoodput = goodput;
 			whenEvent = now;
-//			printf("maxGoodput increased to %.0f KiB/s\n", goodput/1024);
+//			printf("maxGoodput increased to %.0f Kb/s\n", goodput/1000);
 		}
 
-		if(!cancelled && (((now - whenEvent) > 60 && bytes > (maxGoodput * maxRTT * 100) && bytes > (maxGoodput * maxWait * 100)) || (now - startTime) > 600)) {
+		if(!cancelled && (((now - whenEvent) > 60 && bits > (maxGoodput * maxRTT * 100) && bits > (maxGoodput * maxWait * 100)) || (now - startTime) > 600)) {
 			// Maybe time to stop, so signal this as appropriate
 			// carry on receiving data until it stops
 			cancel_flow(fid);
@@ -298,21 +298,21 @@
 
 	if(cancelled && !(chug_mask | spew_mask)) {
 		finishTime = now;
-		goodput = bytes / (finishTime - startTime);
+		goodput = bits / (finishTime - startTime);
 
 //		stats->crc = htonl(~crc);
-		stats->Bps = htonl((uint32_t) goodput);
-		stats->smooth = htonl((uint32_t)(256 / maxWait));
+		stats->bps = htonl((uint32_t) goodput);
+		stats->latency = maxWait;
 
 		if(send(sockfd, stats, sizeof(stats_packet), 0) < sizeof(stats_packet))
-			stats->Bps = stats->smooth = 0;
+			stats->bps = stats->latency = 0;
 
-		stats->Bps = ntohl(stats->Bps);
-		stats->smooth = ntohl(stats->smooth);
+		stats->bps = ntohl(stats->bps);
+//		stats->latency = stats->latency;
 
-//		printf("Raw stats sent: Bps=%u smooth=%u\n", stats->Bps, stats->smooth);
+//		printf("Raw stats sent: bps=%u latency=%.1f\n", stats->bps, stats->latency*1000);
 	} else {
-		stats->Bps = stats->smooth = 0;
+		stats->bps = stats->latency = 0;
 	}
 
 	// drain the connection to make the network quiescent and avoid RST packet bursts
@@ -369,7 +369,7 @@
 	const uint8_t scenario_flows[] = { 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4, 32, 32, 32, 32, 32, 0 };
 	const uint8_t scenario_uplds[] = { 0, 1, 0, 1, 2, 0, 1, 2, 3, 0, 1, 2, 3, 4,  0,  1, 16, 31, 32, 0 };
 	int scenario, upScenarios = 0, dnScenarios = 0;
-	double flowSmoothness = 0, upCapacity = 0, dnCapacity = 0;
+	double flowLatency = 0, upCapacity = 0, dnCapacity = 0;
 	randctx ctx;
 
 	memset(&hints, 0, sizeof hints);
@@ -421,8 +421,7 @@
 		pthread_t bulk_thread[32] = {0};
 		stats_packet *stats[32] = {NULL};
 		int flow, flows = scenario_flows[scenario], ups = scenario_uplds[scenario], dns = flows-ups;
-		double upCap = 0, dnCap = 0, smoothness = 0;
-		uint32_t worstSmooth = 0;
+		double upCap = 0, dnCap = 0, latency = 0, worstLatency = 0;
 
 		printf("Scenario %u: %u uploads, %u downloads... ", scenario+1, ups, dns);
 		fflush(stdout);
@@ -458,31 +457,31 @@
 			pthread_join(bulk_thread[flow], &stats[flow]);
 
 			if(flow < ups)
-				upCap += 1.0 / stats[flow]->Bps;
+				upCap += 1.0 / stats[flow]->bps;
 			else
-				dnCap += 1.0 / stats[flow]->Bps;
+				dnCap += 1.0 / stats[flow]->bps;
 
-			if(!flow || worstSmooth > stats[flow]->smooth)
-				worstSmooth = stats[flow]->smooth;
+			if(!flow || worstLatency < stats[flow]->latency)
+				worstLatency = stats[flow]->latency;
 		}
 
 		if(ups) {
 			upCap = ups*ups/upCap;
-			printf("%u KiB/s up, ", (uint32_t)(upCap / 1024));
+			printf("%u Kb/s up, ", (uint32_t)(upCap / 1000));
 			upCapacity += 1.0 / upCap;
 			upScenarios++;
 		}
 		if(dns) {
 			dnCap = dns*dns/dnCap;
-			printf("%u KiB/s down, ", (uint32_t)(dnCap / 1024));
+			printf("%u Kb/s down, ", (uint32_t)(dnCap / 1000));
 			dnCapacity += 1.0 / dnCap;
 			dnScenarios++;
 		}
 
-		smoothness = worstSmooth / 256.0;
-		printf("%.2f Hz smoothness\n", smoothness);
-		if(!scenario || smoothness < flowSmoothness)
-			flowSmoothness = smoothness;
+		latency = worstLatency;
+		printf("%.1f ms latency\n", latency*1000);
+		if(!scenario || latency > flowLatency)
+			flowLatency = latency;
 	}
 
 	printf("\nOVERALL:\n");
@@ -490,10 +489,10 @@
 	finished = 1;
 	pthread_join(pingpong_thread, NULL);
 
-	printf("    Upload Capacity: %u KiB/s\n", (unsigned int) floor((upScenarios / 1024.0) / upCapacity));
-	printf("  Download Capacity: %u KiB/s\n", (unsigned int) floor((dnScenarios / 1024.0) / dnCapacity));
-	printf("Link Responsiveness: %u Hz\n", (unsigned int) floor(1.0/maxRTT));
-	printf("    Flow Smoothness: %u Hz\n", (unsigned int) floor(flowSmoothness));
+	printf("    Upload Capacity: %u Kb/s\n", (unsigned int) floor((upScenarios / 1000.0) / upCapacity));
+	printf("  Download Capacity: %u Kb/s\n", (unsigned int) floor((dnScenarios / 1000.0) / dnCapacity));
+	printf("       Link Latency: %.1f ms\n", maxRTT*1000);
+	printf("       Flow Latency: %.1f ms\n", flowLatency*1000);
 }
 
 static void* server_conn(void *arg)

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 

At the moment I am working to prove that high smoothness and responsiveness can actually be measured, which requires first setting up a network which can genuinely achieve it. I was surprised to find that pure Ethernet was not by default sufficient, though it is easily explainable. 

I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 

The key to knowledge is not to rely on others to teach you it. 

On 23 Mar 2011, at 12:33, Otto Solares Cabrera <solca@guug.org> wrote:

> On Sun, Mar 20, 2011 at 12:45:10PM +0200, Jonathan Morton wrote:
>> Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.
>> 
>> It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.
>> 
>> It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.
>> 
>> There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.
> 
> Hello Jonathan,
> 
> Excellent tool! hopefully with more testing we can validate it's
> results and improve it as testing all those scenarios seems the
> correct path to understand bufferbloat and latency in our networks.
> 
> Sadly it's very difficult for a netadmin like me to follow your "units"
> as the world have been standarized on bits per second in base10 or
> decimal as opposed to the bytes per second in base2 or binary as
> reported by this tool, "smoothness" too is a very radical unit to
> measure latency which could be in milliseconds as the tool name implies.
> 
> Please don't take it as an offense but I cooked this small patch to
> address this problems in the case you want to change it or for others
> if they feel the need for more "normal" measuring units which in turn
> can lead to more people testing their networks with this tool.
> 
> It's a diff against nbd's git repository, it seems to work for me but
> honestly I have yet to figure out all the data it reports, hopefully
> I didn't b0rk it too much :)
> -
> Otto
> <loadlatency_units.patch>

Re: [Bloat] Progress with latency-under-load tool

[Otto Solares]

On Wed, Mar 23, 2011 at 01:26:59PM +0200, Jonathan Morton wrote:
> Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 

Yeah, after hours of running some results seems wrong, hopefully is not
broken on my missing part of converting a float to network byte order.

> I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 

Understood your PoV, sadly even my bosses (who lacks any degree of
technicallity) knows that the Internet is sold to us in Mb/s (decimal)
and delay (as he call it) is measured in ms.

Good luck doing that nontechnical people use a CLI tool! ;)
-
 Otto

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 23 Mar, 2011, at 9:27 pm, Otto Solares wrote:

>> Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 
> 
> Yeah, after hours of running some results seems wrong, hopefully is not
> broken on my missing part of converting a float to network byte order.

Well, it was that incorrect endianness-conversion which finally made me throw my hands up in despair.  I didn't bother reading to the end of the patch.

By contrast, changing the random-number generator has only a minor impact on the program, namely how fast it can generate traffic and whether a network compression engine might find compressible weaknesses in it (the default UNIX rand() function is potentially weak enough for that).  I could quite happily drop in a standalone Mersenne Twister implementation, so long as I could still show it was fast enough on my old Pentium-MMX.

>> I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 
> 
> Understood your PoV, sadly even my bosses (who lacks any degree of
> technicallity) knows that the Internet is sold to us in Mb/s (decimal)
> and delay (as he call it) is measured in ms.

I do plan to add the traditional units as a secondary output, but I want to finish proving that high-frequency networks actually do exist first.  The existing units will remain primary for the reasons outlined below.

> Good luck doing that nontechnical people use a CLI tool! ;)

That is also a valid point, though I expect the tool to be used by at least moderately technical people, and the results to be usable by less technical people.  It's also possible that it might eventually be developed into a GUI tool.  At the moment, the length of time it takes to run a test creates a substantial selection for patience - but this is partially deliberate, because it takes time to be sure of exercising the network's worst-case performance modes.

The real point is that applications don't care one jot about bits-per-second, which is universally contaminated by overheads such as packet headers, error correction and retransmissions.  What they care about is the bytes in the payload, so that's what I'm measuring.  Even if the Internet is *sold* in Mbps, it is *used* in KiB/s, and the usual conversion factors between the two are routinely found to be inaccurate (not least when the weasel-words "up to" are involved).

Similarly, games run in frames-per-second, so comparing the Responsiveness number to the performance of your graphics card let you know how many frames of lag are induced purely by network conditions.  Equivalently, the Smoothness number compared to the framerate of a typical video (30fps = 30Hz) tell you how many frames the video player needs to buffer before it can reliably let you see anything.  There is not such an obvious link between the network and the application if you measure the network in milliseconds, although the smoothness of current networks is so poor that you can often see it in a Web browser's download progress bar.

The smoothness and responsiveness numbers I'm getting over Ethernet are both absurdly low, suggesting that packet loss due to queue overstuffing is endemic, even with packet buffers that are already far too large in attempt to stave it off.  In some cases I am even seeing connections dropping due to complete starvation - multiple consecutive retransmissions are being dropped.  This is not how people think of a modern network, especially not the wired, full-duplex, switched Ethernet that I'm measuring right now.

 - Jonathan