From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 23B393B29D; Sat, 22 Oct 2022 16:18:21 -0400 (EDT) Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id 44A561B25F; Sat, 22 Oct 2022 13:18:20 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com 44A561B25F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1666469900; bh=2RTV9Sfd8pp9oHlRGqtuAMoC7dvTf3LWk+PnEODuqMc=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=YuinUSGzXJY1HvUMwvjsNEB4I6fwN0Jt38+dITERVu/rRAqhzO5Ji4Y91y6J1wdb6 IdamtcB1TXPKIO8gZ0PFR/dV48CytRlPvkVtSj8gH3teSi2BWA7lEbvvkcCcDP03Mj H2X7uedh0M4NRuKyePwi2OcLLRf4D6wehPBDfumA= MIME-Version: 1.0 Date: Sat, 22 Oct 2022 13:18:20 -0700 From: rjmcmahon To: Dave Taht Cc: Rpm , Measurement Analysis and Tools Working Group , IETF IPPM WG , tsvwg IETF list , discuss , Dave Taht via Starlink In-Reply-To: References: Message-ID: X-Sender: rjmcmahon@rjmcmahon.com Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sun, 23 Oct 2022 20:42:28 -0400 Subject: Re: [Starlink] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Oct 2022 20:18:21 -0000 We've added full histograms into many of iperf 2 metrics. The python flows codes has an example of how to measure distances between non-parametric histograms/distributions per the Kolmogorv-Smirnov test. Statistical process controls (SPC) per techniques like Hotelling T2 can be useful to identify regressions under controlled tests. In general, trying to use single scalars or even parametric distribution values, e.g. mean & variance, need to be thought out as normalizing distributions loses information. I think others have said network metric distributions exhibit chaotic behaviors so normalization may not work nor will predictions too far out. User experience is a multi-label classification problem and not easily defined - probably need a supervised system with user feedback/inputs into the model to come up with that. Bottom line to me is that network analysis techniques need to us machine learning or multivariate stats, linear algebra, etc to be useful and even with all this advanced math it still can be incorrect. https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution https://en.wikipedia.org/wiki/Multi-label_classification https://en.wikipedia.org/wiki/Chaos_theory https://sourceforge.net/projects/iperf2/ Bob > One of the best talks I've ever seen on how to measure customer > satisfaction properly just went up after the P99 Conference. > > It's called Misery Metrics. > > After going through a deep dive as to why and how we think and act on > percentiles, bins, and other statistical methods as to how we use the > web and internet are *so wrong* (well worth watching and thinking > about if you are relying on or creating network metrics today), it > then points to the real metrics that matter to users and the ultimate > success of an internet business: Timeouts, retries, misses, failed > queries, angry phone calls, abandoned shopping carts and loss of > engagement. > > https://www.p99conf.io/session/misery-metrics-consequences/ > > The ending advice was - don't aim to make a specific percentile > acceptable, aim for an acceptable % of misery. > > I enjoyed the p99 conference more than any conference I've attended in > years. > > -- > This song goes out to all the folk that thought Stadia would work: > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz > Dave Täht CEO, TekLibre, LLC