Re: [Bloat] Progress with latency-under-load tool

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.

It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.

It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.

There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.

The Capacity calculations are a bit complex to understand.  For each flow (in either direction), the average goodput is measured over the entire lifetime of the flow.  All of the flows in each direction for that scenario are the aggregated by taking the harmonic mean and multiplying by the number of flows; this biases the total downwards if the flows were mutually unfair.  Finally, the relevant measures across all scenarios are aggregated using the harmonic mean, thus biasing the overall measure towards cases of under-utilisation.  The totals are then reported in binary kilobytes per second.

The Link Responsiveness measure is much easier.  I simply take the maximum latency of the "pinger" channel (which is also used for commanding flows to stop) and invert it to produce a measure in Hertz.  This is rounded down to the next integer for display; if you see a zero here (which is surprisingly likely), it means that a latency of more than one second was encountered at some point during the entire test.

The Flow Smoothness measure refers to the application-level data inter-arrival timings.  The maximum delay between data chunks arriving is measured across all flows in all scenarios, but excludes the first megabyte of each flow so as to avoid picking up the RTT during TCP startup.

Here are some numbers I got from a test over a switched 100base-TX LAN:

    Upload Capacity: 1018 KiB/s
  Download Capacity: 2181 KiB/s
Link Responsiveness: 2 Hz
    Flow Smoothness: 1 Hz

Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.

 - Jonathan


[-- Attachment #2: loadlatency.tar.gz --]
[-- Type: application/x-gzip, Size: 6651 bytes --]

Fwd: [Bloat] Progress with latency-under-load tool

[Pedro Tumusok]

Keep forgetting to reply all, sorry!


---------- Forwarded message ----------
From: Pedro Tumusok <pedro.tumusok@gmail.com>
Date: Sun, Mar 20, 2011 at 3:28 PM
Subject: Re: [Bloat] Progress with latency-under-load tool
To: Jonathan Morton <chromatix99@gmail.com>


>
> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.
>
Is there any point in running this on a CPE? Or would it be better to
have it for host to host connections?


--
Best regards / Mvh
Jan Pedro Tumusok



-- 
Best regards / Mvh
Jan Pedro Tumusok

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 20 Mar, 2011, at 4:33 pm, Pedro Tumusok wrote:

>> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.
>> 
> Is there any point in running this on a CPE? Or would it be better to
> have it for host to host connections?

Some CPE is multifunctional, eg. fileserver, webcache...  for those types it is definitely worthwhile.

Otherwise, the CPE is just another component of the network between two hosts, but an important one because it often controls the bottleneck (in at least the upload direction).  You culd therefore productively modify the CPE's device drivers or network stack, and then retest it using ordinary hosts.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[grenville armitage]

On 03/20/2011 21:45, Jonathan Morton wrote:
	[..]
> Here are some numbers I got from a test over a switched 100base-TX LAN:
>
>      Upload Capacity: 1018 KiB/s
>    Download Capacity: 2181 KiB/s
> Link Responsiveness: 2 Hz
>      Flow Smoothness: 1 Hz
>
> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.

Perhaps a tangential 2 cents from me, but I'm unclear how helpful Hertz is as
a unit of measurement for the challenge of raising awareness of bufferbloat.
I suspect data networking people (whether network designers, admins, product
managers, etc) don't think of IP networks as having characteristics measured in
Hertz. And I'd imagine most ISP helpdesks wont know how to parse customer complaints
along the lines of "your service doesn't give me enough Hertz".  Perhaps your tool
could also add a parenthetical 'translation' of the responsiveness and smoothness
values into something expressed in milliseconds?

cheers,
gja

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

grenville armitage <garmitage@swin.edu.au> writes:

> On 03/20/2011 21:45, Jonathan Morton wrote:
> 	[..]
>> Here are some numbers I got from a test over a switched 100base-TX LAN:
>>
>>      Upload Capacity: 1018 KiB/s
>>    Download Capacity: 2181 KiB/s
>> Link Responsiveness: 2 Hz
>>      Flow Smoothness: 1 Hz
>>
>> Horrible, isn't it?  I deliberately left these machines with standard
>> configurations in order to show that.
>
> Perhaps a tangential 2 cents from me, but I'm unclear how helpful
> Hertz is as a unit of measurement for the challenge of raising
> awareness of bufferbloat.  I suspect data networking people (whether
> network designers, admins, product managers, etc) don't think of IP
> networks as having characteristics measured in Hertz. And I'd imagine
> most ISP helpdesks wont know how to parse customer complaints along
> the lines of "your service doesn't give me enough Hertz".  Perhaps
> your tool could also add a parenthetical 'translation' of the
> responsiveness and smoothness values into something expressed in
> milliseconds?

Concur. And resolution past 3 significant digits from there seems
possible with posix timers. I regret that I have not had time to look
over the code yet this weekend.

How'd they do debloated?

>
> cheers,
> gja
> _______________________________________________
> Bloat-devel mailing list
> Bloat-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat-devel

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 20 Mar, 2011, at 10:33 pm, grenville armitage wrote:

>> Here are some numbers I got from a test over a switched 100base-TX LAN:
>> 
>>    Upload Capacity: 1018 KiB/s
>>  Download Capacity: 2181 KiB/s
>> Link Responsiveness: 2 Hz
>>    Flow Smoothness: 1 Hz
>> 
>> Horrible, isn't it?  I deliberately left these machines with standard configurations in order to show that.
> 
> Perhaps a tangential 2 cents from me, but I'm unclear how helpful Hertz is as
> a unit of measurement for the challenge of raising awareness of bufferbloat.

Customers won't be asking for "more Hertz" - or at least, none that have any sense.  They'll be asking for "more smoothness" for their video streams, or "more responsiveness" for their online games.  They already ask for "more bandwidth", not "more megabits per second".

Hertz makes sense as a unit because, when talking about latency or transmission delays, shorter times are better, but people understand "bigger is better" much more easily.  Hard drives used to measure their seek times in milliseconds too, but now they are measured in IOPS instead (a trend mostly associated with SSDs, which have IOPS numbers several orders of magnitude better than mechanical drives).

Let me manually translate that for you, though.  That Responsiveness rating of 2Hz means that the practical RTT went above 334ms - and this on a switched Fast Ethernet being driven by good-quality 1996-vintage hardware on one side and a cheap nettop on the other.  It actually reflects not pure latency as 'ping' would have measured it, but a packet loss and the time required to recover from it.

And the "smoothness" rating actually contrived to be *worse*, at somewhere north of 500ms.  At Fast Ethernet speeds, that implies megabytes of buffering, on what really is a very old computer.

It's a clear sign of something very broken in the network stack, especially as I get broadly similar results (with much higher throughput numbers) when I run the same test on GigE hardware with much more powerful computers (which can actually saturate GigE).

You really don't want to see what I got on my 3G test runs.  I got 0.09 Hz from a single flow, and these tests run all the way up to 32 flows.  I think the modem switched down into GPRS mode for a few minutes as well, even though there was no obvious change in propagation conditions.

> And resolution past 3 significant digits from there seems
> possible with posix timers.

If the latency was staying in the single-digit milliseconds as it should be on a LAN, you'd have three s.f. with just integers.  I do print the smoothness numbers out to 2 decimal places for the individual scenarios, though - these are the numbers meant for investigating problems:

Scenario 1: 0 uploads, 1 downloads... 1343 KiB/s down, 31.52 Hz smoothness
Scenario 2: 1 uploads, 0 downloads... 3670 KiB/s up, 22.24 Hz smoothness
Scenario 3: 0 uploads, 2 downloads... 2077 KiB/s down, 19.70 Hz smoothness
Scenario 4: 1 uploads, 1 downloads... 2855 KiB/s up, 322 KiB/s down, 6.44 Hz smoothness

That's from a different test, where you can see the effect of a WLAN and having Vegas on one side of the connection.

With that said, since the single-digit results are so ubiquitous, perhaps some extra precision is warranted after all.  Perhaps I can take the opportunity to squash some more minor bugs, and add an interpretation of the goodput in terms of gigabytes per month.

> How'd they do debloated?

I'm still investigating that, partly as the older hardware wasn't yet set up with kernels capable of running advanced qdiscs.  It takes a while to compile a kernel on a Pentium-MMX.  I'm also really not sure where to get debloated drivers for a VIA Rhine or a Sun GEM.  ;-)  Mind you, such a thing may not be needed, if the device drivers are already slim so that cutting txqueuelen is sufficient.

I can't run debloated 3G tests - I don't have access to the buffer controls on the base tower.  :-(

- Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 20 Mar, 2011, at 10:33 pm, grenville armitage wrote:
>
> Customers won't be asking for "more Hertz" - or at least, none that
> have any sense.  They'll be asking for "more smoothness" for their
> video streams, or "more responsiveness" for their online games.  They
> already ask for "more bandwidth", not "more megabits per second".

I think despite your quest for a marketing number, that also reporting
actual RTT would be helpful (and horrifying, as I just got a .02HZ
rating on the still ongoing test...). 

I still can't claim to fully understand what this test is measuring.

Scenario 8: 2 uploads, 1 downloads... 4743 KiB/s up, 3422 KiB/s down, 0.02 Hz smoothness
Scenario 9: 3 uploads, 0 downloads... 7558 KiB/s up, 1.39 Hz smoothness
Scenario 10: 0 uploads, 4 downloads... 6719 KiB/s down, 2.08 Hz smoothness
Scenario 11: 1 uploads, 3 downloads... 4372 KiB/s up, 4067 KiB/s down, 1.30 Hz smoothness

The interesting outlier thus far is 8... I'm tempted to stop the test
now and recompile for testing 3 u/l 3 d/l first....

Even better, we could use a different name for your usage of hz or
smoothness here. Mortons?

(There, I named it for you. You cannot be accused of ego for using this
 new unit now. Enjoy. )

> Hertz makes sense as a unit because, when talking about latency or
> transmission delays, shorter times are better, but people understand
> "bigger is better" much more easily.  Hard drives used to measure
> their seek times in milliseconds too, but now they are measured in
> IOPS instead (a trend mostly associated with SSDs, which have IOPS
> numbers several orders of magnitude better than mechanical drives).
>
> Let me manually translate that for you, though.  That Responsiveness
> rating of 2Hz means that the practical RTT went above 334ms - and this
> on a switched Fast Ethernet being driven by good-quality 1996-vintage
> hardware on one side and a cheap nettop on the other.  It actually
> reflects not pure latency as 'ping' would have measured it, but a
> packet loss and the time required to recover from it.

Your explanation here is good. Perhaps it could be an automated textual
description. 

> And the "smoothness" rating actually contrived to be *worse*, at
> somewhere north of 500ms.  At Fast Ethernet speeds, that implies
> megabytes of buffering, on what really is a very old computer.

Txqueuelen = 1000 = ~1.5Mbyte (and I'm struggling with using consistent
decimal or base 2 units this month) Mibibyte?

Normal dma tx ring size ranges from 64 to 512, can be seen sometimes
with ethtool -g device

> It's a clear sign of something very broken in the network stack,
> especially as I get broadly similar results (with much higher
> throughput numbers) when I run the same test on GigE hardware with
> much more powerful computers (which can actually saturate GigE).
>
> You really don't want to see what I got on my 3G test runs.  I got
> 0.09 Hz from a single flow, and these tests run all the way up to 32
> flows.  I think the modem switched down into GPRS mode for a few
> minutes as well, even though there was no obvious change in
> propagation conditions.
>

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

d@taht.net (Dave Täht) writes:

> The interesting outlier thus far is 8... I'm tempted to stop the test
> now and recompile for testing 3 u/l 3 d/l first....

Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness

... Still running tests ...

> Txqueuelen = 1000 = ~1.5Mbyte (and I'm struggling with using consistent
> decimal or base 2 units this month) Mibibyte?
>
> Normal dma tx ring size ranges from 64 to 512, can be seen sometimes
> with ethtool -g device

With wireless you also have TX_RETRIES = 13 for these (ath9) devices
drivers under test. (Reduced to 4 in the less bloated version I can't
test right now)

You also have contention from other devices, and the occasional complete
retrain from a base figure of 1Mbit/sec up, while devices struggle to
regain sync.

There are (presumably helpful) effects of packet aggregation.

You also have modulo issues against the size of the tx queues, an
interaction with the network flow scheduler (what triggers filling the
tx queue?), SACK/DSACK and lost ACK retransmits, and probably more
issues than I've been able to come up with offhand.

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:

>> The interesting outlier thus far is 8... I'm tempted to stop the test
>> now and recompile for testing 3 u/l 3 d/l first....
> 
> Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness
> 
> ... Still running tests ...

I should probably explain what the scenarios are.

Every combination of up/down flows is tried for each of 1, 2, 3, 4 total flows.  That's 14 scenarios covering basic downloading and uploading through to heavy whole-family use.

Then I jump ahead to 32 flows, with 32/0, 31/1, 16/16, 1/31, 0/32 splits, to simulate BitTorrent style traffic.

Total 19 scenarios.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:
>
>>> The interesting outlier thus far is 8... I'm tempted to stop the test
>>> now and recompile for testing 3 u/l 3 d/l first....
>> 
>> Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s
>> down, 0.81 Hz smoothness
>> 
>> ... Still running tests ...
>
> I should probably explain what the scenarios are.
>
> Every combination of up/down flows is tried for each of 1, 2, 3, 4
> total flows.  That's 14 scenarios covering basic downloading and
> uploading through to heavy whole-family use.
>
> Then I jump ahead to 32 flows, with 32/0, 31/1, 16/16, 1/31, 0/32
> splits, to simulate BitTorrent style traffic.
>
> Total 19 scenarios.

I'll still argue that starting with the worst case scenarios and working
down would be best... 3/3 perhaps

Scenario 12: 2 uploads, 2 downloads... 4725 KiB/s up, 3645 KiB/s down, 0.81 Hz smoothness
Scenario 13: 3 uploads, 1 downloads... 4917 KiB/s up, 3175 KiB/s down, 0.59 Hz smoothness


>
>  - Jonathan
>

-- 
Dave Taht
http://nex-6.taht.net

Re: [Bloat] Progress with latency-under-load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> On 21 Mar, 2011, at 12:47 am, Dave Täht wrote:
>
>>> The interesting outlier thus far is 8... I'm tempted to stop the test
>>> now and recompile for testing 3 u/l 3 d/l first....


I found the cause of the outlier, the arm box had a hiccough on its
hard disk controller for some reason around then. Or at least that
appears to be the case, but judging from this:

Scenario 16: 1 uploads, 31 downloads... 4401 KiB/s up, 4111 KiB/s down, 0.59 Hz smoothness
Scenario 17: 16 uploads, 16 downloads... 6108 KiB/s up, 2103 KiB/s down,
0.00 Hz smoothness

Maybe not.

Me thinks we need 3 digits of precision here.



-- 
Dave Taht
http://nex-6.taht.net

Some results of the latency under load tool

[Dave Täht]

Jonathan Morton <chromatix99@gmail.com> writes:

> Here are some numbers I got from a test over a switched 100base-TX LAN:
>
>     Upload Capacity: 1018 KiB/s
>   Download Capacity: 2181 KiB/s
> Link Responsiveness: 2 Hz
>     Flow Smoothness: 1 Hz
>
> Horrible, isn't it?  I deliberately left these machines with standard
> configurations in order to show that.

I successfully compiled and ran this on both an debian squeeze/arm based
platform (an openrd), and a 64 bit ubuntu 10.10. It generated some
warnings on 64 bit.

Run on my partially debloated wireless/wired network[1][2] via the the
openrd->nano-m->nano-m->davepc path (180Mbit wireless) over IPv6 (wow,
transparent IPv6! Thanks!), over IPv6, server=davepc

Scenario 1: 0 uploads, 1 downloads... 8024 KiB/s down, 12.71 Hz smoothness
Scenario 2: 1 uploads, 0 downloads... 6526 KiB/s up, 5.52 Hz smoothness
Scenario 3: 0 uploads, 2 downloads... 6703 KiB/s down, 19.07 Hz smoothness
Scenario 4: 1 uploads, 1 downloads... 3794 KiB/s up, 4084 KiB/s down, 1.82 Hz smoothness
Scenario 5: 2 uploads, 0 downloads... 7503 KiB/s up, 4.11 Hz smoothness
Scenario 6: 0 uploads, 3 downloads... 7298 KiB/s down, 2.08 Hz smoothness
Scenario 7: 1 uploads, 2 downloads... 4323 KiB/s up, 4690 KiB/s down, 1.38 Hz smoothness

My pure wireless and more debloated network is down right now[2]

I'm still watching it run other scenarios.

Comments/Suggestions:

A) From a "quicker results" perspective, do the heaviest network tests first,
then the lightest tests, then the rest? :)

B) Due to the dependence on the gnu scientific library it seems unlikely
this will build on openwrt. Also, if there is heavy math anywhere in
this app, it needs to be done outside the testing loop on a non FPU
enabled processor such as most arms to not skew the test.

It ate 17% of cpu on the arm, and ~8% of cpu on davepc.

Similarly there are rng's in the Linux kernel and hardware sources, I'm
not sure to what extent you need randomness (I looked at the code  only briefly)

C) Do you have this in a repository somewhere? (I use github). I would
certainly love to have more tools like this living together in
harmony. Needn't be github.

D) It's never been clear to me what resolution gettimeofday has on most
platforms, and the posix clock_gettime and the corresponding

clock_getres(clockid_t clk_id, struct timespec *res)

Can make that clear. CLOCK_REALTIME also discards any clock adjustments
in the interval for even higher potential accuracy. (On the interval
anyway, not perhaps against wall-clock time) Given the period of this
test, do you think that the precision can only be 1ms?

It's certainly my hope that most systems today implement posix timers.

I can produce a patch if you want.

Thanks for such an interesting tool!

-- 
Dave Taht
http://nex-6.taht.net

[1] The openrd has a very small txqueuelen (4) and dma-tx-queue of 20 on
the GigE card, which is connected at 100Mbit to the nano-m5s, which DOES
NOT have my ath9k patch or eBDP enabled but does have a small
txqueuelen, the laptop is running debloat-testing (txqueulen 4, dma tx
queue 64), connected via 100Mbit to the other nano-m5 radio.

tcp cubic throughout. ECN and SACK/DSACK are enabled but there is no
shaping in place.

The wireless radios inbetween DO tend to show bloat under worse
conditions than this test.

(I note that debloat-testings patches to the e1000 series ethernet
driver do not allow dma tx queues of less than 64 as yet)

[2] for my network diagram see:
 http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700

This is an old test I need to repeat it with debloat testing.

Re: Some results of the latency under load tool

[Jonathan Morton]

On 20 Mar, 2011, at 11:50 pm, Dave Täht wrote:

> B) Due to the dependence on the gnu scientific library it seems unlikely
> this will build on openwrt. Also, if there is heavy math anywhere in
> this app, it needs to be done outside the testing loop on a non FPU
> enabled processor such as most arms to not skew the test.

I'm aware of the limitations of older ARM CPUs, as I have had to work with them on occasion.  I don't have any with Ethernet at home though.  The maths inside the main loops is limited to calculating time differences, comparing them and determining when to stop - there may be some optimisation possible there, but it's not ridiculous.

The main loops are actually slightly elastic, because recv() will give whatever data is available at the time it is called, and the fattest calculations are only done once per loop.  Under normal circumstances, that should be every 1.2 KB, but it scales up to 64KB if the CPU temporarily can't keep up (or if a lot of data becomes available in one go).

> It ate 17% of cpu on the arm, and ~8% of cpu on davepc.

Looks tolerable to me.  Even my ancient CISC PC can trigger the symptoms well enough.

> Similarly there are rng's in the Linux kernel and hardware sources, I'm
> not sure to what extent you need randomness (I looked at the code  only briefly)

The spew() function uses the PRNG to generate incompressible traffic, since compressibility does matter to some network technologies.  I used GSL purely to get access to a very fast PRNG that still produces good numbers - it's just a shame that this is a relatively large, general-purpose library.  I originally intended to checksum the data and thus ensure that it arrived intact, but I had to delete that feature to get around a race condition.

> So you are saying lower values of hz is horrible?

Yes.  0.02 Hz means the application was waiting 50 seconds for data somehow.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 21 Mar, 2011, at 6:09 am, Daniel Brooks wrote:

> Oof. I guess the loopback driver could use some work as well :)

Either that or there is some strange timing effect going on that I should be aware of.  I'm going to try to run some more tests to see if I can get the smoothness and responsiveness values up, but if any of you notice something bizarre...

Be aware however that both smoothness and responsiveness are taken as the absolute worst cases observed.  If there is a bell curve, you're going to be looking at outliers.  Perhaps I should calculate something like a 7-number summary to sanity-check it.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Otto Solares Cabrera]

[-- Attachment #1: Type: text/plain, Size: 2072 bytes --]

On Sun, Mar 20, 2011 at 12:45:10PM +0200, Jonathan Morton wrote:
> Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.
> 
> It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.
> 
> It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.
> 
> There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.

Hello Jonathan,

Excellent tool! hopefully with more testing we can validate it's
results and improve it as testing all those scenarios seems the
correct path to understand bufferbloat and latency in our networks.

Sadly it's very difficult for a netadmin like me to follow your "units"
as the world have been standarized on bits per second in base10 or
decimal as opposed to the bytes per second in base2 or binary as
reported by this tool, "smoothness" too is a very radical unit to
measure latency which could be in milliseconds as the tool name implies.

Please don't take it as an offense but I cooked this small patch to
address this problems in the case you want to change it or for others
if they feel the need for more "normal" measuring units which in turn
can lead to more people testing their networks with this tool.

It's a diff against nbd's git repository, it seems to work for me but
honestly I have yet to figure out all the data it reports, hopefully
I didn't b0rk it too much :)
-
 Otto

[-- Attachment #2: loadlatency_units.patch --]
[-- Type: text/x-diff, Size: 6398 bytes --]

--- loadlatency.orig/loadlatency.c	2011-03-23 03:27:16.171982917 -0600
+++ loadlatency/loadlatency.c	2011-03-23 03:43:15.981962124 -0600
@@ -144,7 +144,7 @@
 			}
 
 			if(!settled && numPings > 16 && now-firstPing > 5.0) {
-				printf("MinRTT: %.1fms\n", minRTT * 1000);
+				printf("MinRTT: %.1f ms\n", minRTT * 1000);
 				settled = 1;
 			}
 		} else {
@@ -185,8 +185,8 @@
 typedef struct
 {
 //	uint32_t crc;
-	uint32_t Bps;    // bytes per second, will saturate at 4GB/s per flow.
-	uint32_t smooth; // fixed point 24.8, units are Hz
+	uint32_t bps;     // bits per second, will saturate at 32Gb/s per flow.
+	float    latency; // milliseconds
 } stats_packet;
 
 static void* spew(int sockfd, bool server)
@@ -227,12 +227,12 @@
 		goto bail;
 	}
 //	stats->crc = ntohl(stats->crc);
-	stats->Bps = ntohl(stats->Bps);
-	stats->smooth = ntohl(stats->smooth);
+	stats->bps = ntohl(stats->bps);
+	stats->latency = (float) ntohl((uint32_t) stats->latency);
 //	if(stats->crc != ~crc)
 //		memset(stats, 0, sizeof(stats_packet));
 
-//	printf("Raw stats received: Bps=%u smooth=%u\n", stats->Bps, stats->smooth);
+//	printf("Raw stats received: bps=%u latency=%.1f\n", stats->bps, stats->latency);
 
 bail:
 	close(sockfd);
@@ -244,7 +244,7 @@
 	uint8_t fid = get_flow_id();
 	uint8_t buffer[65536];
 //	uint32_t crc = ~0;
-	uint64_t bytes = 0;
+	uint64_t bits = 0;
 	stats_packet *stats = calloc(sizeof(stats), 1);
 	double startTime = gettime(), finishTime = 0;
 	double now = 0, then = 0;
@@ -262,11 +262,11 @@
 			break;
 
 //		crc = crc32(crc, buffer, rv);
-		bytes += rv;
+		bits += rv*8;
 
-		goodput = bytes / (now - startTime);
+		goodput = bits / (now - startTime);
 
-		if(bytes > 1024*1024) {
+		if(bits*8 > 1000*1000) {
 			double wait = now-then;
 			if(wait > maxWait) {
 				maxWait = wait;
@@ -284,10 +284,10 @@
 		if(goodput > maxGoodput) {
 			maxGoodput = goodput;
 			whenEvent = now;
-//			printf("maxGoodput increased to %.0f KiB/s\n", goodput/1024);
+//			printf("maxGoodput increased to %.0f Kb/s\n", goodput/1000);
 		}
 
-		if(!cancelled && (((now - whenEvent) > 60 && bytes > (maxGoodput * maxRTT * 100) && bytes > (maxGoodput * maxWait * 100)) || (now - startTime) > 600)) {
+		if(!cancelled && (((now - whenEvent) > 60 && bits > (maxGoodput * maxRTT * 100) && bits > (maxGoodput * maxWait * 100)) || (now - startTime) > 600)) {
 			// Maybe time to stop, so signal this as appropriate
 			// carry on receiving data until it stops
 			cancel_flow(fid);
@@ -298,21 +298,21 @@
 
 	if(cancelled && !(chug_mask | spew_mask)) {
 		finishTime = now;
-		goodput = bytes / (finishTime - startTime);
+		goodput = bits / (finishTime - startTime);
 
 //		stats->crc = htonl(~crc);
-		stats->Bps = htonl((uint32_t) goodput);
-		stats->smooth = htonl((uint32_t)(256 / maxWait));
+		stats->bps = htonl((uint32_t) goodput);
+		stats->latency = maxWait;
 
 		if(send(sockfd, stats, sizeof(stats_packet), 0) < sizeof(stats_packet))
-			stats->Bps = stats->smooth = 0;
+			stats->bps = stats->latency = 0;
 
-		stats->Bps = ntohl(stats->Bps);
-		stats->smooth = ntohl(stats->smooth);
+		stats->bps = ntohl(stats->bps);
+//		stats->latency = stats->latency;
 
-//		printf("Raw stats sent: Bps=%u smooth=%u\n", stats->Bps, stats->smooth);
+//		printf("Raw stats sent: bps=%u latency=%.1f\n", stats->bps, stats->latency*1000);
 	} else {
-		stats->Bps = stats->smooth = 0;
+		stats->bps = stats->latency = 0;
 	}
 
 	// drain the connection to make the network quiescent and avoid RST packet bursts
@@ -369,7 +369,7 @@
 	const uint8_t scenario_flows[] = { 1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4, 32, 32, 32, 32, 32, 0 };
 	const uint8_t scenario_uplds[] = { 0, 1, 0, 1, 2, 0, 1, 2, 3, 0, 1, 2, 3, 4,  0,  1, 16, 31, 32, 0 };
 	int scenario, upScenarios = 0, dnScenarios = 0;
-	double flowSmoothness = 0, upCapacity = 0, dnCapacity = 0;
+	double flowLatency = 0, upCapacity = 0, dnCapacity = 0;
 	randctx ctx;
 
 	memset(&hints, 0, sizeof hints);
@@ -421,8 +421,7 @@
 		pthread_t bulk_thread[32] = {0};
 		stats_packet *stats[32] = {NULL};
 		int flow, flows = scenario_flows[scenario], ups = scenario_uplds[scenario], dns = flows-ups;
-		double upCap = 0, dnCap = 0, smoothness = 0;
-		uint32_t worstSmooth = 0;
+		double upCap = 0, dnCap = 0, latency = 0, worstLatency = 0;
 
 		printf("Scenario %u: %u uploads, %u downloads... ", scenario+1, ups, dns);
 		fflush(stdout);
@@ -458,31 +457,31 @@
 			pthread_join(bulk_thread[flow], &stats[flow]);
 
 			if(flow < ups)
-				upCap += 1.0 / stats[flow]->Bps;
+				upCap += 1.0 / stats[flow]->bps;
 			else
-				dnCap += 1.0 / stats[flow]->Bps;
+				dnCap += 1.0 / stats[flow]->bps;
 
-			if(!flow || worstSmooth > stats[flow]->smooth)
-				worstSmooth = stats[flow]->smooth;
+			if(!flow || worstLatency < stats[flow]->latency)
+				worstLatency = stats[flow]->latency;
 		}
 
 		if(ups) {
 			upCap = ups*ups/upCap;
-			printf("%u KiB/s up, ", (uint32_t)(upCap / 1024));
+			printf("%u Kb/s up, ", (uint32_t)(upCap / 1000));
 			upCapacity += 1.0 / upCap;
 			upScenarios++;
 		}
 		if(dns) {
 			dnCap = dns*dns/dnCap;
-			printf("%u KiB/s down, ", (uint32_t)(dnCap / 1024));
+			printf("%u Kb/s down, ", (uint32_t)(dnCap / 1000));
 			dnCapacity += 1.0 / dnCap;
 			dnScenarios++;
 		}
 
-		smoothness = worstSmooth / 256.0;
-		printf("%.2f Hz smoothness\n", smoothness);
-		if(!scenario || smoothness < flowSmoothness)
-			flowSmoothness = smoothness;
+		latency = worstLatency;
+		printf("%.1f ms latency\n", latency*1000);
+		if(!scenario || latency > flowLatency)
+			flowLatency = latency;
 	}
 
 	printf("\nOVERALL:\n");
@@ -490,10 +489,10 @@
 	finished = 1;
 	pthread_join(pingpong_thread, NULL);
 
-	printf("    Upload Capacity: %u KiB/s\n", (unsigned int) floor((upScenarios / 1024.0) / upCapacity));
-	printf("  Download Capacity: %u KiB/s\n", (unsigned int) floor((dnScenarios / 1024.0) / dnCapacity));
-	printf("Link Responsiveness: %u Hz\n", (unsigned int) floor(1.0/maxRTT));
-	printf("    Flow Smoothness: %u Hz\n", (unsigned int) floor(flowSmoothness));
+	printf("    Upload Capacity: %u Kb/s\n", (unsigned int) floor((upScenarios / 1000.0) / upCapacity));
+	printf("  Download Capacity: %u Kb/s\n", (unsigned int) floor((dnScenarios / 1000.0) / dnCapacity));
+	printf("       Link Latency: %.1f ms\n", maxRTT*1000);
+	printf("       Flow Latency: %.1f ms\n", flowLatency*1000);
 }
 
 static void* server_conn(void *arg)

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 

At the moment I am working to prove that high smoothness and responsiveness can actually be measured, which requires first setting up a network which can genuinely achieve it. I was surprised to find that pure Ethernet was not by default sufficient, though it is easily explainable. 

I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 

The key to knowledge is not to rely on others to teach you it. 

On 23 Mar 2011, at 12:33, Otto Solares Cabrera <solca@guug.org> wrote:

> On Sun, Mar 20, 2011 at 12:45:10PM +0200, Jonathan Morton wrote:
>> Attached is the initial version of the loadlatency tool.  I'm getting some rather interesting results from it already, although it does take a very long time to run.
>> 
>> It works under Linux, Cygwin and MacOS X on both little- and big-endian machines (and between machines of different byte-sexes), and therefore it should also work under FreeBSD and other UNIXes.  I haven't yet tried compiling it for iOS or Android.
>> 
>> It produces useful results even when one of the machines is rather old and slow, despite using a proper PRNG for traffic generation.  My ancient Pentium-MMX proved capable of generating at least 4.4 MB/s of traffic steadily, and probably produces spikes of even greater bandwidth.  Anything of Y2K vintage or newer should be able to saturate it's network with this.
>> 
>> There are four measures produced:  Upload Capacity, Download Capacity, Link Responsiveness and Flow Smoothness.  All of these are reported in "bigger is better" units to help deal with Layers 8 and 9.
> 
> Hello Jonathan,
> 
> Excellent tool! hopefully with more testing we can validate it's
> results and improve it as testing all those scenarios seems the
> correct path to understand bufferbloat and latency in our networks.
> 
> Sadly it's very difficult for a netadmin like me to follow your "units"
> as the world have been standarized on bits per second in base10 or
> decimal as opposed to the bytes per second in base2 or binary as
> reported by this tool, "smoothness" too is a very radical unit to
> measure latency which could be in milliseconds as the tool name implies.
> 
> Please don't take it as an offense but I cooked this small patch to
> address this problems in the case you want to change it or for others
> if they feel the need for more "normal" measuring units which in turn
> can lead to more people testing their networks with this tool.
> 
> It's a diff against nbd's git repository, it seems to work for me but
> honestly I have yet to figure out all the data it reports, hopefully
> I didn't b0rk it too much :)
> -
> Otto
> <loadlatency_units.patch>

Re: [Bloat] Progress with latency-under-load tool

[Otto Solares]

On Wed, Mar 23, 2011 at 01:26:59PM +0200, Jonathan Morton wrote:
> Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 

Yeah, after hours of running some results seems wrong, hopefully is not
broken on my missing part of converting a float to network byte order.

> I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 

Understood your PoV, sadly even my bosses (who lacks any degree of
technicallity) knows that the Internet is sold to us in Mb/s (decimal)
and delay (as he call it) is measured in ms.

Good luck doing that nontechnical people use a CLI tool! ;)
-
 Otto

Re: [Bloat] Progress with latency-under-load tool

[Jonathan Morton]

On 23 Mar, 2011, at 9:27 pm, Otto Solares wrote:

>> Unfortunately that patch will not work - it completely breaks part of the on-wire protocol. It is much better to simply convert the final results for an auxiliary display. 
> 
> Yeah, after hours of running some results seems wrong, hopefully is not
> broken on my missing part of converting a float to network byte order.

Well, it was that incorrect endianness-conversion which finally made me throw my hands up in despair.  I didn't bother reading to the end of the patch.

By contrast, changing the random-number generator has only a minor impact on the program, namely how fast it can generate traffic and whether a network compression engine might find compressible weaknesses in it (the default UNIX rand() function is potentially weak enough for that).  I could quite happily drop in a standalone Mersenne Twister implementation, so long as I could still show it was fast enough on my old Pentium-MMX.

>> I should also point out that I have very strong reasons for providing the measurements in non-traditional units by default. I'm measuring characteristics as they matter to applications and users, who measure things in bytes and frames per second, not bits and milliseconds. It is also much easier to get nontechnical people (who tend to be in charge of budgets) to respond to bigger-is-better numbers. 
> 
> Understood your PoV, sadly even my bosses (who lacks any degree of
> technicallity) knows that the Internet is sold to us in Mb/s (decimal)
> and delay (as he call it) is measured in ms.

I do plan to add the traditional units as a secondary output, but I want to finish proving that high-frequency networks actually do exist first.  The existing units will remain primary for the reasons outlined below.

> Good luck doing that nontechnical people use a CLI tool! ;)

That is also a valid point, though I expect the tool to be used by at least moderately technical people, and the results to be usable by less technical people.  It's also possible that it might eventually be developed into a GUI tool.  At the moment, the length of time it takes to run a test creates a substantial selection for patience - but this is partially deliberate, because it takes time to be sure of exercising the network's worst-case performance modes.

The real point is that applications don't care one jot about bits-per-second, which is universally contaminated by overheads such as packet headers, error correction and retransmissions.  What they care about is the bytes in the payload, so that's what I'm measuring.  Even if the Internet is *sold* in Mbps, it is *used* in KiB/s, and the usual conversion factors between the two are routinely found to be inaccurate (not least when the weasel-words "up to" are involved).

Similarly, games run in frames-per-second, so comparing the Responsiveness number to the performance of your graphics card let you know how many frames of lag are induced purely by network conditions.  Equivalently, the Smoothness number compared to the framerate of a typical video (30fps = 30Hz) tell you how many frames the video player needs to buffer before it can reliably let you see anything.  There is not such an obvious link between the network and the application if you measure the network in milliseconds, although the smoothness of current networks is so poor that you can often see it in a Web browser's download progress bar.

The smoothness and responsiveness numbers I'm getting over Ethernet are both absurdly low, suggesting that packet loss due to queue overstuffing is endemic, even with packet buffers that are already far too large in attempt to stave it off.  In some cases I am even seeing connections dropping due to complete starvation - multiple consecutive retransmissions are being dropped.  This is not how people think of a modern network, especially not the wired, full-duplex, switched Ethernet that I'm measuring right now.

 - Jonathan

Re: [Bloat] Progress with latency-under-load tool

[Dave Hart]

[Removed bloat@]

On Wed, Mar 23, 2011 at 8:40 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> I do plan to add the traditional units as a secondary output, but I want
> to finish proving that high-frequency networks actually do exist first.
> The existing units will remain primary for the reasons outlined below.

- Hide quoted text -
Hi Jonathan.  Thanks for taking the time to develop this new tool in
the bufferbloat.net arsenal.

I have been hoping you would come around under the overwhelming
one-sided responses to your use of inverted statistics and save me
from weighing in, but no such luck, though I'm not sure there's been a
single expression of support for reporting only Hz.

I fail to see how reporting latency and max observed payload progress
stall time in both time and Hz would prevent the proof you seek, which
leaves me wondering what really underlies your rigidity.

The Hz representation is hiding useful information, because despite
our hopes, the latency and peak jitter do in fact often exceed 1s, and
your tool is limiting the display to so few significant digits the end
result is the summary numbers (0 Hz!) are _useless_ and any
information to be gleaned has to come from our own summary of the
individual test statistics.  I also question using only the worst
result as the overall result -- I fear it will lead to the tool being
viewed as producing noisy numbers requiring many runs to validate
before drawing conclusions.  I suggest min/max/avg/stddev for each.

Your apparent belief is that the people who must be convinced are not
clever enough to properly interpret lower-is-better.  That is a
presumptive and condescending attitude.  I suspect that's a big part
of why the response so far has been so one-sided.

Thanks again for producing your tool and enabling others to measure
and share what they see with it.

Cheers,
Dave Hart