[Rpm] [M-Lab-Discuss] misery metrics & consequences
rjmcmahon
rjmcmahon at rjmcmahon.com
Sat Oct 22 16:18:20 EDT 2022
We've added full histograms into many of iperf 2 metrics. The python
flows codes has an example of how to measure distances between
non-parametric histograms/distributions per the Kolmogorv-Smirnov test.
Statistical process controls (SPC) per techniques like Hotelling T2 can
be useful to identify regressions under controlled tests. In general,
trying to use single scalars or even parametric distribution values,
e.g. mean & variance, need to be thought out as normalizing
distributions loses information. I think others have said network metric
distributions exhibit chaotic behaviors so normalization may not work
nor will predictions too far out. User experience is a multi-label
classification problem and not easily defined - probably need a
supervised system with user feedback/inputs into the model to come up
with that.
Bottom line to me is that network analysis techniques need to us machine
learning or multivariate stats, linear algebra, etc to be useful and
even with all this advanced math it still can be incorrect.
https://en.wikipedia.org/wiki/Central_limit_theorem
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution
https://en.wikipedia.org/wiki/Multi-label_classification
https://en.wikipedia.org/wiki/Chaos_theory
https://sourceforge.net/projects/iperf2/
Bob
> One of the best talks I've ever seen on how to measure customer
> satisfaction properly just went up after the P99 Conference.
>
> It's called Misery Metrics.
>
> After going through a deep dive as to why and how we think and act on
> percentiles, bins, and other statistical methods as to how we use the
> web and internet are *so wrong* (well worth watching and thinking
> about if you are relying on or creating network metrics today), it
> then points to the real metrics that matter to users and the ultimate
> success of an internet business: Timeouts, retries, misses, failed
> queries, angry phone calls, abandoned shopping carts and loss of
> engagement.
>
> https://www.p99conf.io/session/misery-metrics-consequences/
>
> The ending advice was - don't aim to make a specific percentile
> acceptable, aim for an acceptable % of misery.
>
> I enjoyed the p99 conference more than any conference I've attended in
> years.
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
More information about the Rpm
mailing list