[Starlink] [M-Lab-Discuss] misery metrics & consequences

[rjmcmahon]

We've added full histograms into many of iperf 2 metrics. The python 
flows codes has an example of how to measure distances between 
non-parametric histograms/distributions per the Kolmogorv-Smirnov test. 
Statistical process controls (SPC) per techniques like Hotelling T2 can 
be useful to identify regressions under controlled tests. In general, 
trying to use single scalars or even parametric distribution values, 
e.g. mean & variance, need to be thought out as normalizing 
distributions loses information. I think others have said network metric 
distributions exhibit chaotic behaviors so normalization may not work 
nor will predictions too far out. User experience is a multi-label 
classification problem and not easily defined - probably need a 
supervised system with user feedback/inputs into the model to come up 
with that.

Bottom line to me is that network analysis techniques need to us machine 
learning or multivariate stats, linear algebra, etc to be useful and 
even with all this advanced math it still can be incorrect.

https://en.wikipedia.org/wiki/Central_limit_theorem
https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution
https://en.wikipedia.org/wiki/Multi-label_classification
https://en.wikipedia.org/wiki/Chaos_theory
https://sourceforge.net/projects/iperf2/

Bob


> One of the best talks I've ever seen on how to measure customer
> satisfaction properly just went up after the P99 Conference.
> 
> It's called Misery Metrics.
> 
> After going through a deep dive as to why and how we think and act on
> percentiles, bins, and other statistical methods as to how we use the
> web and internet are *so wrong* (well worth watching and thinking
> about if you are relying on or creating network metrics today), it
> then points to the real metrics that matter to users and the ultimate
> success of an internet business: Timeouts, retries, misses, failed
> queries, angry phone calls, abandoned shopping carts and loss of
> engagement.
> 
> https://www.p99conf.io/session/misery-metrics-consequences/
> 
> The ending advice was - don't aim to make a specific percentile
> acceptable, aim for an acceptable % of misery.
> 
> I enjoyed the p99 conference more than any conference I've attended in 
> years.
> 
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC