From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 03DE13B29E for ; Tue, 11 Jan 2022 17:04:07 -0500 (EST) Received: by mail-ed1-x536.google.com with SMTP id z22so1796575edd.12 for ; Tue, 11 Jan 2022 14:04:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Un+pvGE8x5mEhC5wnkVlSA5s4r42n4iNwlvoOiVp2m4=; b=Hyo438YY6Xyzn6/dJwJLylON3QSlRj/+WE4C6IF4Z+VS0YtfMKcLVsYcyN3GI6Rb7x Z1TbBU6Pf5EQkldDoV3xeMCwxtKruW4QDs+Bozo9li/4jOrYxgvUFgJM/Ju4zCUv+4cI KxbFnXi+j4SjI8LyJj0q5k8rJevphtMt31SHDBqPCLa4r59cIWog/Gu/GIK2R7e23a12 4fDkEdJykoM58YSeBwZ47i5XdKh2i98p/iQrcJbMd6eXXPqHpClYD9GLdVpf9BDivf1X iUw4i+6W+yrFj2IputFfFMR+nelKLuxuhrypxuZEn8mouSSuotN1ARSAqjKf7OMBNvcR 2lug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Un+pvGE8x5mEhC5wnkVlSA5s4r42n4iNwlvoOiVp2m4=; b=gWtNj5BYNKef/1yr1+TWn3Pm8CDk2/dcVWyYBXsoA62VIL1Hp5KuoIaYd0iBt7jlrc CzTKvWUM8oPQLYuhwG7OFzYTs8My2P2hhGNtFAvxuiQjPh1aOZYuAGBG3fFCIfXFBvEB nROBm8qOmdRyJRPj7R+Y+qUXXoLZlT+g48PrUE60UMMifUKa0quq29agawMOKr8muHMW AT0GeRAAiOP1lEZjyC49fA3ayjR+4DnldzibA/6QEut+ZaqytOl604RTl7GHU+Ca/Iw7 N2sVZdsOHvb8wbu9EOOG6Ur1O9KZrhkngTPbJJbErBtCZq5X2OIiR5zCCGqn5uThebbp lzzA== X-Gm-Message-State: AOAM5335yPqTvEQzXxELVJAMQ3H+wOROqr8j847GiH6lNeq0SNRLFyb6 iKtowyC6754cQQr5JNfTtEusa4ppGgW176oBX5Q= X-Google-Smtp-Source: ABdhPJwNdeUHvdFz637Suy3HLhHKHPf4eTsYa7DdkvWXBEedS6IqTIaGCq5NS+rx9/cYCEBowcHdT8Zho0He0sRZDP0= X-Received: by 2002:a17:907:97ca:: with SMTP id js10mr5023163ejc.645.1641938646763; Tue, 11 Jan 2022 14:04:06 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Tue, 11 Jan 2022 14:03:54 -0800 Message-ID: To: Aaron Wood Cc: Christoph Paasch , Rpm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Rpm] apm metric - annoyance per minute X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Jan 2022 22:04:08 -0000 On Tue, Jan 11, 2022 at 11:44 AM Dave Taht wrote: > > On Tue, Jan 11, 2022 at 9:51 AM Aaron Wood wrote: > > > > I read it as the number of events per minute (not say the number of fra= mes longer than 20ms, but the number of events that took at least 20ms long= er than the base RTT, which I think is what Dave meant by "latency excursio= n"). > > yes. thx for reading my tea leaves. The big thing to me was > "annoyance" or "glitch" per minute somewhat in line of rpm's concept. > > Hey, this network does 2500RPM but with .5APM: > https://blog.cerowrt.org/post/disabling_channel_scans/ > > On my holiday trip, mostly staying in cheap hotels, not *one* hotel > out of 6 could sustain a quality videoconference. 10-20GPM, > but web pages and netflix loaded fine. > > > > > E.g. if you're doing DNS queries, and they usually return in 50ms, and = 3 of them take 83, 94, and 106 ms respectively, in a given minute, than tha= t would be an APM of 3? ( or maybe an APM rate of 30% if you were doing 10= /minute) > > Yes. You count the excursions from the (semi-smoothed) baseline, not > the size of the excursion. A "glitch" happened. > > > I've found the tricky thing for metrics is sorting out the event-count = vs. events-rate differences. A lot of tests that are isochronous give a co= nstant event-rate to base on (e.g. ping's default of once per second), but = other tests, like the UDP and ICMP pings in flent (at least in the past), h= ave a rate that's based on the RTT, so as RTT goes up, the rate of events g= oes down, which means that it oversamples the "fast" events, and undersampl= es "slow" events. > > The original rrul spec had an isochronous voip like flow, not ping rtt > test here., which has the annoying flaws you describe above. Which we > now have in the irtt tool, but most of our tests still use ping. At > the time (when we were shooting for reductions of latency from seconds > to 10s of ms) using "ping" wasn't as much of a problem as it is today. > We need a rrul_v2 and a tcp_nup, tcp_ndown tests that just do > isochronous flows at fixed (and ideally high frequency, irtt works > well to about 3ms, opus codec can do 2.7ms) > > > Further, retries for failed events muddy the waters, as the events aren= 't independent measurements. If a momentary drop in connectivity causes re= tries to happen, and each failed retry is counted, is that N failures? Or = just 1 failure? I've split those out as separate metrics in some systems I= 've built, so that I can tease them apart. I've also done things like the = distribution (histogram) of "attempts before success" or "attempts before o= peration failed". Usually those are dominated by "1 attempt before success= ", and "N attempts before operation failed" where N is the number of total = attempts before just giving up. > > Histograms are great. I kind of wanted to separate the concepts that a > "glitch" happened, and also measure the glitch duration (so X retries > turns into a duration rather than a count), and (sigh) whether the > glitch mattered or not. It doesn't matter to a web page if you have an > 250ms RTO on one flow but it takes 3sec to load anyway. > > glitches matter more for videoconferencing and gaming. I don't know if > there is any human factors research on this, but once I find my flow > in an application, a 20ms 'glitch' is roughly as annoying as a 3second > long one. Glitches matter even more for music and video playback. Anyone who's ever experienced a pink floyd record as a whole rather than ripped into tracks is an example, very few players get the transitions between songs correct (all my floyd records I ripped as a single song). Coincidentally I was trying to work out a rube goldberg machine for a web page load and st ubmedl ed https://www.youtube.com/watch?v=3DWyOSqjIABe0 ... stumbled across this wonderful ok-go video about what a 90% and 99% reliability rate meant for the successive success of 130 events. And while I was typing this, my lte connection glitched and I ended up typing "st ubmedl ed" whilst it caught up. I don't know how many people suffer from glitches as badly as I do - as one example I turn spellchekcing off when working with a real editor and there's no way to do that with gmail, I think - but finding and keeping my flow doesn't last very long when my services "glitch" like this. > > > > > On Tue, Jan 11, 2022 at 9:34 AM Christoph Paasch via Rpm wrote: > >> > >> Hi Dave! > >> > >> > On Jan 9, 2022, at 6:57 PM, Dave Taht via Rpm wrote: > >> > > >> > or gpm - glitch per minute > >> > > >> > defined as a latency excursion of more than 20ms. > >> > >> I kinda find that interesting :) Can you give an example? Would it cou= nt the number of times we miss a "20ms-deadline"? So, if the RTT is 100ms, = GPM would be 5 ? > >> > >> > >> Christoph > >> > >> > > >> > ? > >> > > >> > > >> > -- > >> > I tried to build a better future, a few times: > >> > https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > >> > > >> > Dave T=C3=A4ht CEO, TekLibre, LLC > >> > _______________________________________________ > >> > Rpm mailing list > >> > Rpm@lists.bufferbloat.net > >> > https://lists.bufferbloat.net/listinfo/rpm > >> > >> _______________________________________________ > >> Rpm mailing list > >> Rpm@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/rpm > > > > -- > I tried to build a better future, a few times: > https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > > Dave T=C3=A4ht CEO, TekLibre, LLC --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC