[Bloat] Measuring latency-under-load consistently

[Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

I'm currently resurrecting my socket-programming skills (last used almost 10 years ago when IPv6 really *was* experimental) in the hope of making a usable latency-under-load tester.  This could be run in server-mode on one host, and then as a client on another host could be pointed at the server, followed by several minutes of churning and some nice round numbers.

It would need to make multiple TCP connections simultaneously, one of which would be used to measure latency (using NODELAY marked sockets), and one or more others used to load the network and measure goodput.  It would automatically determine how long to run in order to get a reliable result that can't easily be challenged by (eg.) an ISP.

The output metrics would be:

1) Average goodput for uplink and downlink, for single flows and multiple flows, in binary megabytes per second.  Just for laughs, I might also add the equivalent gigabytes-per-month figures.

2) Maximum latency (in the parallel "interactive" flow) under load, expressed in Hz rather than milliseconds.  This gives a number that gets bigger for better performance, which is much easier for laymen to understand.

3) Flow smoothness, measured as the maximum time between sequential received data for any continuous flow, also expressed in Hz.  This is an important metric for video and radio streaming, and one which CUBIC will probably do extremely badly at if there are large buffers in the path (without AQM or Blackpool).

Any thoughts on this idea?

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[Rick Jones]

On Sat, 2011-03-12 at 02:00 +0200, Jonathan Morton wrote:
> I'm currently resurrecting my socket-programming skills (last used
>  almost 10 years ago when IPv6 really *was* experimental) in the hope
>  of making a usable latency-under-load tester.  This could be run in
>  server-mode on one host, and then as a client on another host could be
>  pointed at the server, followed by several minutes of churning and
>  some nice round numbers.
> 
> It would need to make multiple TCP connections simultaneously, one of
>  which would be used to measure latency (using NODELAY marked sockets),
>  and one or more others used to load the network and measure goodput. 
>  It would automatically determine how long to run in order to get a
>  reliable result that can't easily be challenged by (eg.) an ISP.

Why would it require multiple TCP connections?  Only if none of the
connections have data flowing in the other direction, and your latency
measuring one would need that anyway. 

Also, while NODELAY (TCP_NODELAY I presume) might be interesting with
something that tried to have multiple, sub-MSS transactions in flight at
one time, it won't change anything about how the packets flow on the
network - TCP_NODELAY has no effect beyond the TCP code running the
connection associated with the socket for that connection.

If you only ever have one transaction outstanding at a time, regardless
of size, if TCP_NODELAY improves the latency, it means the TCP stack is
broken wrt how to interpret the Nagle algorithm - likely as not it is
trying to apply it segment by segment rather than by user send by user
send.

> The output metrics would be:
> 
> 1) Average goodput for uplink and downlink, for single flows and
>  multiple flows, in binary megabytes per second.  Just for laughs, I
>  might also add the equivalent gigabytes-per-month figures.
> 
> 2) Maximum latency (in the parallel "interactive" flow) under load,
>  expressed in Hz rather than milliseconds.  This gives a number that
>  gets bigger for better performance, which is much easier for laymen to
>  understand.
> 
> 3) Flow smoothness, measured as the maximum time between sequential
>  received data for any continuous flow, also expressed in Hz.  This is
>  an important metric for video and radio streaming, and one which CUBIC
>  will probably do extremely badly at if there are large buffers in the
>  path (without AQM or Blackpool).
> 
> Any thoughts on this idea?

You may be able to get most of what you want with a top-of-trunk netperf
"burst mode" TCP_RR test. It isn't quite an exact match though.

The idea is to ./configure netperf for burst mode and histogram, and
probably the "omni" tests to get the more complete statistics on RTT
latency, then run a burst mode TCP_RR test - if doing "upload" then with
a large request size and a single byte response size.  If doing
"download" then with a single byte request size and a large response
size.

If you want to hear more, contact me off-list and I can wax philosophic
on it.

rick jones

>  - Jonathan
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 12 Mar, 2011, at 2:13 am, Rick Jones wrote:

> On Sat, 2011-03-12 at 02:00 +0200, Jonathan Morton wrote:
>> I'm currently resurrecting my socket-programming skills (last used
>> almost 10 years ago when IPv6 really *was* experimental) in the hope
>> of making a usable latency-under-load tester.  This could be run in
>> server-mode on one host, and then as a client on another host could be
>> pointed at the server, followed by several minutes of churning and
>> some nice round numbers.
>> 
>> It would need to make multiple TCP connections simultaneously, one of
>> which would be used to measure latency (using NODELAY marked sockets),
>> and one or more others used to load the network and measure goodput. 
>> It would automatically determine how long to run in order to get a
>> reliable result that can't easily be challenged by (eg.) an ISP.
> 
> Why would it require multiple TCP connections?  Only if none of the
> connections have data flowing in the other direction, and your latency
> measuring one would need that anyway. 

Because the latency within a bulk flow is not as interesting as the latency experienced by interactive or realtime flows sharing the same link as a bulk flow (or three).  In the presence of a re-ordering AQM scheme (trivially, SFQ) the two are not the same.

Suppose for example that you're downloading the latest Ubuntu DVD and you suddenly think of something to look up on Wikipedia.  With the 30-second latencies I have personally experienced on some non-AQM links under load, that is intolerably slow.  With something as simple as SFQ on that same queue it would be considerably better because the new packets could bypass the queue associated with the old flow, but measuring only the old flow wouldn't show that.

Note that the occasional packet losses on a plain SFQ drop-tail queue would still show extremely long maximum inter-arrival delays on the bulk flow, and this is captured by the third metric (flow smoothness).

> Also, while NODELAY (TCP_NODELAY I presume) might be interesting with
> something that tried to have multiple, sub-MSS transactions in flight at
> one time, it won't change anything about how the packets flow on the
> network - TCP_NODELAY has no effect beyond the TCP code running the
> connection associated with the socket for that connection.

I'm essentially going to be running a back-and-forth ping inside a TCP session.  Nagle's algorithm can, if it glitches, add hundreds of milliseconds to that, which can be very material - eg. when measuring a LAN or Wifi.  I wouldn't set NODELAY on the bulk flows.

Why ping inside a TCP session, rather than ICMP?  Because I want to bypass any specific optimisations for ICMP and measure what applications can actually use.

> You may be able to get most of what you want with a top-of-trunk netperf
> "burst mode" TCP_RR test. It isn't quite an exact match though.

I don't really see how that would get the measurement I want.

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[Rick Jones]

On Sat, 2011-03-12 at 02:45 +0200, Jonathan Morton wrote:
> On 12 Mar, 2011, at 2:13 am, Rick Jones wrote:
> 
> > On Sat, 2011-03-12 at 02:00 +0200, Jonathan Morton wrote:
> >> I'm currently resurrecting my socket-programming skills (last used
> >> almost 10 years ago when IPv6 really *was* experimental) in the hope
> >> of making a usable latency-under-load tester.  This could be run in
> >> server-mode on one host, and then as a client on another host could be
> >> pointed at the server, followed by several minutes of churning and
> >> some nice round numbers.
> >> 
> >> It would need to make multiple TCP connections simultaneously, one of
> >> which would be used to measure latency (using NODELAY marked sockets),
> >> and one or more others used to load the network and measure goodput. 
> >> It would automatically determine how long to run in order to get a
> >> reliable result that can't easily be challenged by (eg.) an ISP.
> > 
> > Why would it require multiple TCP connections?  Only if none of the
> > connections have data flowing in the other direction, and your latency
> > measuring one would need that anyway. 
> 
> Because the latency within a bulk flow is not as interesting as the
>  latency experienced by interactive or realtime flows sharing the same
>  link as a bulk flow (or three).  In the presence of a re-ordering AQM
>  scheme (trivially, SFQ) the two are not the same.

Good point, something I'd not considered.

> Suppose for example that you're downloading the latest Ubuntu DVD and
>  you suddenly think of something to look up on Wikipedia.  With the
>  30-second latencies I have personally experienced on some non-AQM
>  links under load, that is intolerably slow.  With something as simple
>  as SFQ on that same queue it would be considerably better because the
>  new packets could bypass the queue associated with the old flow, but
>  measuring only the old flow wouldn't show that.
> 
> Note that the occasional packet losses on a plain SFQ drop-tail queue
>  would still show extremely long maximum inter-arrival delays on the
>  bulk flow, and this is captured by the third metric (flow smoothness).
> 
> > Also, while NODELAY (TCP_NODELAY I presume) might be interesting with
> > something that tried to have multiple, sub-MSS transactions in flight at
> > one time, it won't change anything about how the packets flow on the
> > network - TCP_NODELAY has no effect beyond the TCP code running the
> > connection associated with the socket for that connection.
> 
> I'm essentially going to be running a back-and-forth ping inside a TCP
>  session.  Nagle's algorithm can, if it glitches, add hundreds of
>  milliseconds to that, which can be very material - eg. when measuring
>  a LAN or Wifi.  I wouldn't set NODELAY on the bulk flows.
> 
> Why ping inside a TCP session, rather than ICMP?  Because I want to
>  bypass any specific optimisations for ICMP and measure what
>  applications can actually use.
> 
> > You may be able to get most of what you want with a top-of-trunk netperf
> > "burst mode" TCP_RR test. It isn't quite an exact match though.
> 
> I don't really see how that would get the measurement I want.

Then one TCP_STREAM (or TCP_MAERTS) and one TCP_RR with all the RTT
stats and histogram enabled :)

rick jones

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 12 Mar, 2011, at 3:09 am, Rick Jones wrote:

>>> You may be able to get most of what you want with a top-of-trunk netperf
>>> "burst mode" TCP_RR test. It isn't quite an exact match though.
>> 
>> I don't really see how that would get the measurement I want.
> 
> Then one TCP_STREAM (or TCP_MAERTS) and one TCP_RR with all the RTT
> stats and histogram enabled :)

Closer, but that doesn't seem to be self-tuning - and often one bulk flow doesn't fill the pipe any more.

I want to scale up to FTTH and GigE, and down to V.34bis and GPRS - all of which are (still) relevant network technologies today - without any error-prone parameters to enter except the identity of the server.

Ideally it should be robust enough for use by random ISP technicians, regulatory officials and end-users, and the output should be accordingly simple to interpret.

So yes, code reuse is great, but only if it does what I need it to.  I need the practice anyway.  :-)

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[richard]

On Sat, 2011-03-12 at 02:00 +0200, Jonathan Morton wrote:
snip
> 3) Flow smoothness, measured as the maximum time between sequential received data for any continuous flow, also expressed in Hz.  This is an important metric for video and radio streaming, and one which CUBIC will probably do extremely badly at if there are large buffers in the path (without AQM or Blackpool).
> 
Am I correct that your "Flow smoothness" is the inverse if jitter? We
should probably keep to a standard nomenclature. What should we call
this and/or should we call it something else or invert the concept and
call it what we already do - jitter?

richard
> Any thoughts on this idea?
> 
>  - Jonathan
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
-- 
Richard C. Pitt                 Pacific Data Capture
rcpitt@pacdat.net               604-644-9265
http://digital-rag.com          www.pacdat.net
PGP Fingerprint: FCEF 167D 151B 64C4 3333  57F0 4F18 AF98 9F59 DD73

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 12 Mar, 2011, at 5:19 am, richard wrote:

>> 3) Flow smoothness, measured as the maximum time between sequential received data for any continuous flow, also expressed in Hz.  This is an important metric for video and radio streaming, and one which CUBIC will probably do extremely badly at if there are large buffers in the path (without AQM or Blackpool).
>> 
> 
> Am I correct that your "Flow smoothness" is the inverse if jitter? We
> should probably keep to a standard nomenclature. What should we call
> this and/or should we call it something else or invert the concept and
> call it what we already do - jitter?

I'm not certain that it's the same as what you call jitter, but it could be.  Because I'm going to be measuring at the application level, I don't necessarily get to see when every single packet arrives, particularly if they arrive out of order.  So what I'm measuring is the "lumpiness" of the application data-flow progress, but inverted to "smoothness" (ie. measured in Hz rather than ms) so that bigger numbers are better.

Using my big-easy-numbers example, suppose you have a 30-second unmanaged drop-tail queue, and nothing to stop it filling up.  For a while, packets will arrive in order, so the inter-arrival delay seen by the application is at most the RTT (as during the very beginning of the slow-start, which I think I will exclude from the measurement) and usually less as a continuous stream builds up.

But then the queue fills up and a packet is dropped.  At this point, progress as seen by the application will stop *dead* as soon as that missing packet's position reaches the head of the queue.

The sending TCP will now retransmit that packet.  But the queue is still approximately full because the congestion feedback didn't happen until now, so it will take another 30 seconds for the data to reach the application.  At this point the progress is instantaneously very large, and hopefully will continue more smoothly.

But the maximum inter-arrival delay after that episode is now 30 seconds (or 0.033 Hz), even though packets were arriving correctly throughout that time.  That's what I'm measuring here.

Most links are much less severe than that, of course, but it's this kind of thing that stops radio and video streaming from working properly.

On the much less severe end of the scale, this will also measure the burstiness of flows in the case when there's more than one at once - usually you will get a bunch of packets from one flow, then a bunch from another, and so on, but SFQ tends to fix that for you if you have it.  It will probably also pick up some similar effects from 802.11n aggregation and other link-level congestion-avoidance techniques.

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[richard]

OK - you make a good case for a new measure as my understanding of
jitter is latency related and typically measured at the link level (udp)
rather than at the application level.

I infer then that this will do things like impact the CPU load and disk
load, and might for example introduce "ringing" or harmonics into such
sub systems if/when applications end up "in sync" due to being "less
smooth" in their data output to the lower level IP levels.

It will be affected by session drops due to timeouts as well as the need
to "fill the pipe" on a reconnect in such applications as streaming
video (my area) so that a key frame can come into the system and restart
the interrupted video play.

In my case I'm particularly allergic to such "restarts" in the midst of
a streaming program that is going out to multiple (thousands) of
receiving systems.

richard


On Sat, 2011-03-12 at 05:52 +0200, Jonathan Morton wrote:
> On 12 Mar, 2011, at 5:19 am, richard wrote:
> 
> >> 3) Flow smoothness, measured as the maximum time between sequential received data for any continuous flow, also expressed in Hz.  This is an important metric for video and radio streaming, and one which CUBIC will probably do extremely badly at if there are large buffers in the path (without AQM or Blackpool).
> >> 
> > 
> > Am I correct that your "Flow smoothness" is the inverse if jitter? We
> > should probably keep to a standard nomenclature. What should we call
> > this and/or should we call it something else or invert the concept and
> > call it what we already do - jitter?
> 
> I'm not certain that it's the same as what you call jitter, but it could be.  Because I'm going to be measuring at the application level, I don't necessarily get to see when every single packet arrives, particularly if they arrive out of order.  So what I'm measuring is the "lumpiness" of the application data-flow progress, but inverted to "smoothness" (ie. measured in Hz rather than ms) so that bigger numbers are better.
> 
> Using my big-easy-numbers example, suppose you have a 30-second unmanaged drop-tail queue, and nothing to stop it filling up.  For a while, packets will arrive in order, so the inter-arrival delay seen by the application is at most the RTT (as during the very beginning of the slow-start, which I think I will exclude from the measurement) and usually less as a continuous stream builds up.
> 
> But then the queue fills up and a packet is dropped.  At this point, progress as seen by the application will stop *dead* as soon as that missing packet's position reaches the head of the queue.
> 
> The sending TCP will now retransmit that packet.  But the queue is still approximately full because the congestion feedback didn't happen until now, so it will take another 30 seconds for the data to reach the application.  At this point the progress is instantaneously very large, and hopefully will continue more smoothly.
> 
> But the maximum inter-arrival delay after that episode is now 30 seconds (or 0.033 Hz), even though packets were arriving correctly throughout that time.  That's what I'm measuring here.
> 
> Most links are much less severe than that, of course, but it's this kind of thing that stops radio and video streaming from working properly.
> 
> On the much less severe end of the scale, this will also measure the burstiness of flows in the case when there's more than one at once - usually you will get a bunch of packets from one flow, then a bunch from another, and so on, but SFQ tends to fix that for you if you have it.  It will probably also pick up some similar effects from 802.11n aggregation and other link-level congestion-avoidance techniques.
> 
>  - Jonathan
> 
-- 
Richard C. Pitt                 Pacific Data Capture
rcpitt@pacdat.net               604-644-9265
http://digital-rag.com          www.pacdat.net
PGP Fingerprint: FCEF 167D 151B 64C4 3333  57F0 4F18 AF98 9F59 DD73

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 12 Mar, 2011, at 6:04 am, richard wrote:

> OK - you make a good case for a new measure as my understanding of
> jitter is latency related and typically measured at the link level (udp)
> rather than at the application level.
> 
> I infer then that this will do things like impact the CPU load and disk
> load, and might for example introduce "ringing" or harmonics into such
> sub systems if/when applications end up "in sync" due to being "less
> smooth" in their data output to the lower level IP levels.

I'm not sure how significant those effects would be, compared to simple data starvation at the client.  Most Web servers operate with all the frequently-accessed data in RAM (via disk cache) and serve many clients at once or in quick succession, whose network paths don't have the same bottlenecks in general.

It was my understanding that UDP-bsed protocols tended to tolerate packet loss through redundancy and graceful degradation rather than retransmission, though there are always exceptions to the rule.  So a video streaming server would be transmitting smoothly, with the client giving feedback on how much data had been received and how much packet loss it was experiencing.  Even if that status information is considerably delayed, I don't see why load spikes at the server should occur.

A fileserver, on the other hand, would not care very much.  Even if the TCP window has grown to a megabyte, it takes longer to seek disk heads than to read that much off the platter, so these lumps would be absorbed by the normal readahead and elevator algorithms anyway.  However, large TCP windows do consume RAM in both server and client, and with a sufficient number of simultaneous clients, that could theoretically cause trouble.  Constraining TCP windows to near the actual BDP is more efficient all around.

> It will be affected by session drops due to timeouts as well as the need
> to "fill the pipe" on a reconnect in such applications as streaming
> video (my area) so that a key frame can come into the system and restart
> the interrupted video play.

In the event of a TCP session drop, I think I will consider it a test failure and give zero scores across the board.  Sufficient delay or packet loss to cause that indicates a pretty badly broken network.

With that said, I can think of a case where it is likely to happen.  Remember that I was seeing 30 seconds of buffering on a 500kbps 3G link...  now what happens if the link drops to GPRS speeds?  There would be over a megabyte of data queued up behind a 50Kbps link (at best).  No wonder stuff basically didn't work when that happened.

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[Fred Baker]

At the risk of sounding like someone mentioning a product, let me mention a product. This assumes, of course, that you're using Cisco equipment. But it allows you to measure delay (how long does it take to get from here to there), jitter (first derivative of delay/dt), and packet loss.

http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a00801b1a1e.shtml#definedelayjitterpacketloss

Re: [Bloat] Measuring latency-under-load consistently

[richard]

On Sat, 2011-03-12 at 23:57 +0200, Jonathan Morton wrote:
> On 12 Mar, 2011, at 6:04 am, richard wrote:
> 
> > OK - you make a good case for a new measure as my understanding of
> > jitter is latency related and typically measured at the link level (udp)
> > rather than at the application level.
> > 
> > I infer then that this will do things like impact the CPU load and disk
> > load, and might for example introduce "ringing" or harmonics into such
> > sub systems if/when applications end up "in sync" due to being "less
> > smooth" in their data output to the lower level IP levels.
> 
> I'm not sure how significant those effects would be, compared to simple
>  data starvation at the client.  Most Web servers operate with all the
>  frequently-accessed data in RAM (via disk cache) and serve many
>  clients at once or in quick succession, whose network paths don't have
>  the same bottlenecks in general.
> 
> It was my understanding that UDP-bsed protocols tended to tolerate

my bad - meant ICMP (as in ping)

>  packet loss through redundancy and graceful degradation rather than
>  retransmission, though there are always exceptions to the rule.  So a
>  video streaming server would be transmitting smoothly, with the client
>  giving feedback on how much data had been received and how much packet
>  loss it was experiencing.  Even if that status information is
>  considerably delayed, I don't see why load spikes at the server should
>  occur.

some of the video servers I deal with are using TCP (Windows Media for
example unless you configure it otherwise) but in general you're right,
the general rule with concurrent (as opposed to multicast) unicast
streams is that the server clocks the outbound and cares little about
whether the packets actually get there - that's the receiver's problem.

> 
> A fileserver, on the other hand, would not care very much.  Even if the
>  TCP window has grown to a megabyte, it takes longer to seek disk heads
>  than to read that much off the platter, so these lumps would be
>  absorbed by the normal readahead and elevator algorithms anyway. 
>  However, large TCP windows do consume RAM in both server and client,
>  and with a sufficient number of simultaneous clients, that could
>  theoretically cause trouble.  Constraining TCP windows to near the
>  actual BDP is more efficient all around.

Yes - have had RAM exhaustion problems on busy servers with large video
files - major headache.

> 
> > It will be affected by session drops due to timeouts as well as the need
> > to "fill the pipe" on a reconnect in such applications as streaming
> > video (my area) so that a key frame can come into the system and restart
> > the interrupted video play.
> 
> In the event of a TCP session drop, I think I will consider it a test
>  failure and give zero scores across the board.  Sufficient delay or
>  packet loss to cause that indicates a pretty badly broken network.

Yup - that's what the problem is all right
> 
> With that said, I can think of a case where it is likely to happen. 
>  Remember that I was seeing 30 seconds of buffering on a 500kbps 3G
>  link...  now what happens if the link drops to GPRS speeds?  There
>  would be over a megabyte of data queued up behind a 50Kbps link (at
>  best).  No wonder stuff basically didn't work when that happened.
> 
>  - Jonathan
> 

Your insights into this are great - thanks :)

richard

-- 
Richard C. Pitt                 Pacific Data Capture
rcpitt@pacdat.net               604-644-9265
http://digital-rag.com          www.pacdat.net
PGP Fingerprint: FCEF 167D 151B 64C4 3333  57F0 4F18 AF98 9F59 DD73

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 13 Mar, 2011, at 12:21 am, Fred Baker wrote:

> At the risk of sounding like someone mentioning a product, let me mention a product. This assumes, of course, that you're using Cisco equipment. But it allows you to measure delay (how long does it take to get from here to there), jitter (first derivative of delay/dt), and packet loss.

Ping does most of this, and is available on your actual computer.  A little post-processing of the output gives you jitter, if it doesn't supply that natively.

The point is, the existing tools don't typically measure latency *under load*.

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 13 Mar, 2011, at 12:23 am, richard wrote:

>> However, large TCP windows do consume RAM in both server and client,
>> and with a sufficient number of simultaneous clients, that could
>> theoretically cause trouble.  Constraining TCP windows to near the
>> actual BDP is more efficient all around.
> 
> Yes - have had RAM exhaustion problems on busy servers with large video
> files - major headache.

Sounds like a good reason to switch to Vegas or at least Illinois on those servers.  Those are much less aggressive at growing the congestion window than the default CUBIC.

 - Jonathan

Re: [Bloat] Measuring latency-under-load consistently

[Fred Baker]

On Mar 12, 2011, at 3:03 PM, Jonathan Morton wrote:

> 
> On 13 Mar, 2011, at 12:21 am, Fred Baker wrote:
> 
>> At the risk of sounding like someone mentioning a product, let me mention a product. This assumes, of course, that you're using Cisco equipment. But it allows you to measure delay (how long does it take to get from here to there), jitter (first derivative of delay/dt), and packet loss.
> 
> Ping does most of this, and is available on your actual computer.  A little post-processing of the output gives you jitter, if it doesn't supply that natively.
> 
> The point is, the existing tools don't typically measure latency *under load*.

Actually, SAA uses ping, and is intended precisely to do it under load. Ping is part of the existing traffic, and measures the RTT as experienced by traffic following the same path that the ping does.

Not sure exactly where you're going with that...

Re: [Bloat] Measuring latency-under-load consistently

[Jonathan Morton]

On 13 Mar, 2011, at 8:54 am, Fred Baker wrote:

>>> At the risk of sounding like someone mentioning a product, let me mention a product. This assumes, of course, that you're using Cisco equipment. But it allows you to measure delay (how long does it take to get from here to there), jitter (first derivative of delay/dt), and packet loss.
>> 
>> Ping does most of this, and is available on your actual computer.  A little post-processing of the output gives you jitter, if it doesn't supply that natively.
>> 
>> The point is, the existing tools don't typically measure latency *under load*.
> 
> Actually, SAA uses ping, and is intended precisely to do it under load. Ping is part of the existing traffic, and measures the RTT as experienced by traffic following the same path that the ping does.
> 
> Not sure exactly where you're going with that...

While you, and I, and other professional researchers and administrators are quite capable of generating a consistent, relevant load and then measuring around it...  we're the exceptional people.  Most people don't know jitter from a hole in the ground, they just know that Skype doesn't work.

The tool I'm working on does it in an integrated fashion, so the potential for user errors (or plain old marketing BS) creeping into the measurement is reduced.  That is, it generates the load and measures the effect on latency in one step.  The ability to measure some interesting things about the bulk flows at the same time is icing on the cake.

I do have consumer-grade connections in mind for this, though no doubt it will also be useful on professional-grade networks - especially wireless ones.  The consumer space is where the worst problems are currently visible.

 - Jonathan