Re: [Rpm] apm metric - annoyance per minute

[Dave Taht <dave.taht@gmail.com>]

On Tue, Jan 11, 2022 at 9:51 AM Aaron Wood <woody77@gmail.com> wrote:
>
> I read it as the number of events per minute (not say the number of frames longer than 20ms, but the number of events that took at least 20ms longer than the base RTT, which I think is what Dave meant by "latency excursion").

yes. thx for reading my tea leaves. The big thing to me was
"annoyance" or "glitch" per minute somewhat in line of rpm's concept.

Hey, this network does 2500RPM but with .5APM:
https://blog.cerowrt.org/post/disabling_channel_scans/

On my holiday trip, mostly staying in cheap hotels, not *one* hotel
out of 6 could sustain a quality videoconference. 10-20GPM,
 but web pages and netflix loaded fine.

>
> E.g. if you're doing DNS queries, and they usually return in 50ms, and 3 of them take 83, 94, and 106 ms respectively, in a given minute, than that would be an APM of 3?  ( or maybe an APM rate of 30% if you were doing 10/minute)

Yes. You count the excursions from the (semi-smoothed) baseline, not
the size of the excursion. A "glitch" happened.

> I've found the tricky thing for metrics is sorting out the event-count vs. events-rate differences.  A lot of tests that are isochronous give a constant event-rate to base on (e.g. ping's default of once per second), but other tests, like the UDP and ICMP pings in flent (at least in the past), have a rate that's based on the RTT, so as RTT goes up, the rate of events goes down, which means that it oversamples the "fast" events, and undersamples "slow" events.

The original rrul spec had an isochronous voip like flow, not ping rtt
test here., which has the annoying flaws you describe above. Which we
now have in the irtt tool, but most of our tests still use ping. At
the time (when we were shooting for reductions of latency from seconds
to 10s of ms) using "ping" wasn't as much of a problem as it is today.
We need a rrul_v2 and a tcp_nup, tcp_ndown tests that just do
isochronous flows at fixed (and ideally high frequency, irtt works
well to about 3ms, opus codec can do 2.7ms)

> Further, retries for failed events muddy the waters, as the events aren't independent measurements.  If a momentary drop in connectivity causes retries to happen, and each failed retry is counted, is that N failures?  Or just 1 failure?  I've split those out as separate metrics in some systems I've built, so that I can tease them apart.  I've also done things like the distribution (histogram) of "attempts before success" or "attempts before operation failed".  Usually those are dominated by "1 attempt before success", and "N attempts before operation failed" where N is the number of total attempts before just giving up.

Histograms are great. I kind of wanted to separate the concepts that a
"glitch" happened, and also measure the glitch duration (so X retries
turns into a duration rather than a count), and (sigh) whether the
glitch mattered or not. It doesn't matter to a web page if you have an
250ms RTO on one flow but it takes 3sec to load anyway.

glitches matter more for videoconferencing and gaming. I don't know if
there is any human factors research on this, but once I find my flow
in an application, a 20ms 'glitch' is roughly as annoying as a 3second
long one.

>
> On Tue, Jan 11, 2022 at 9:34 AM Christoph Paasch via Rpm <rpm@lists.bufferbloat.net> wrote:
>>
>> Hi Dave!
>>
>> > On Jan 9, 2022, at 6:57 PM, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>> >
>> > or gpm - glitch per minute
>> >
>> > defined as a latency excursion of more than 20ms.
>>
>> I kinda find that interesting :) Can you give an example? Would it count the number of times we miss a "20ms-deadline"? So, if the RTT is 100ms, GPM would be 5 ?
>>
>>
>> Christoph
>>
>> >
>> > ?
>> >
>> >
>> > --
>> > I tried to build a better future, a few times:
>> > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> >
>> > Dave Täht CEO, TekLibre, LLC
>> > _______________________________________________
>> > Rpm mailing list
>> > Rpm@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/rpm
>>
>> _______________________________________________
>> Rpm mailing list
>> Rpm@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/rpm



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC