With some encouragement from Dave, I'm posting a link to a slide deck with a relevant set of metrics that many of you may find of interest. "Will my Netflix/Lync/whatever work?" is a question that many would like to know in advance of committing to a multi-year contract for some piece of network access. We perform that kind of analysis for large fixed and mobile operators as our day job.
The relevant slides are 8 and 9 in
this presentation. It's an annotated version of a technical sales deck that was designed to be talked through, but should make reasonable sense just being read.
The QoE breach metric (red crosses) is a synthetic one that:
- takes the probability distribution of loss and delay
- compares this measurement against a (stochastic) requirement "contract" (quality transport agreement) to quantify the performance hazard
- and as result, the chart indicates the likelihood of an application falling below (or above) its required QoE.
This process generalises the predictive PESQ type of work done for voice to any application.
Red crosses mean the QoE of the application is at risk. (In this case it happens to be voice over a 100Mbps fibre link in a collector arc carrying multiple types of traffic as a single class of service). The green line is the daily usage cycle measured using 5 minute (IIRC) average link utilisation data. There is a correlation, but not a causative relationship.
What makes this data interesting is the robustness of the measurement and mathematics behind this QoE measure. The metric used (ΔQ) is the only (possible) network measure that is also a strong QoE proxy (as relates to application performance).
In this case, we were measuring for a client whether over-provisioning was an efficient and effective approach to dealing with QoE issues. The result reflects similar measures from other fixed and mobile operators: over-provisioning is not an effective means of resolving scheduling issues. This should not be any news to anyone involved in the AQM/bufferbloat space; scheduling choices really do matter, and getting them wrong does affect the customer experience. Hence the client is engaging in improving their scheduling choices, rather than spending capex on capacity that doesn't fix the problem.
The tools used to generate this data are described in
How to X-ray a telecoms network. Critically, you need to measure the
same test stream packet passing
multiple points, and produce the
distribution, and not use averages (e.g. "jitter" or "bandwidth"). How we temporally decompose the data is described in
this presentation.
I have a very long deck I'm still working on that introduces the ΔQ resource model, predictive calculus, and network performance science. The terms used are unfamiliar because producing this kind of data has required the development of a fundamental new branch of mathematics to describe the improper random variables and their composite behaviour.
Please don't hesitate to ask questions. I hope the above is of interest and value, since it represents (to the best of our knowledge) the state of the art in measurement and performance analysis techniques. For more information on them, see
www.pnsol.com.
Martin Geddes