[Rpm] final farewell.

[Sebastian Moeller]

Mmmh,

so I recompiled and tested goresponsiveness after a fix for a cosmetic issue was anounced (issue is fixed, thanks)

But then I stumbled over a new line in the output:

"bash-3.2$ time ./networkQuality --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config  --rpmtimeout 60 --extended-stats --logger-filename go_networkQuality_$(date +%Y%m%d_%H%M%S)
04-16-2023 10:02:21 UTC Go Responsiveness to mensura.cdn-apple.com:443...
RPM:    39 (P90)
RPM:   207 (Double-Sided 10% Trimmed Mean)
Download:  80.994 Mbps ( 10.124 MBps), using 12 parallel connections.
Upload:    23.125 Mbps (  2.891 MBps), using 12 parallel connections.
Extended Statistics:
	Maximum Segment Size: 1208
	Total Bytes Retransmitted: 4268
	Retransmission Ratio: 0.54%
	Total Bytes Reordered: 98003004
	Average RTT: 33.46153846153846


real	0m12.842s
user	0m1.010s
sys	0m1.625s"


RPM:   207 (Double-Sided 10% Trimmed Mean) ?


	[SM] Trimmed mean? That got my whiskers in attention position, looking at the current draft I found the following "rationale":


"Over the timeframe of these intervals a potentially large number of samples has been collected. These may be affected by noise in the measurements, and outliers. Thus, to aggregate these we suggest to use a trimmed mean at the TMP (Trimmed Mean Percentage - default to 95%) percentile, thus providing the following numbers: TM(tcp_f), TM(tls_f), TM(http_f), TM(http_l).

The responsiveness is then calculated as the weighted mean:

Responsiveness = 60000 /
(1/6*(TM(tcp_f) + TM(tls_f) + TM(http_f)) + 1/2*TM(http_s))

This responsiveness value presents round-trips per minute (RPM)."


[SM] First a question TMP 95% what does that mean, the mean from 2.5% to 97.5% or from 0% to 95% 0r fro 5% to 95%? I am sure there are clear definitions but I think the draft should spell that out explicitly.


	[SM] But honestly, I am not 100% convinced this is actually a good idea... I think I understand the motivation here, boiling down a somewhat complex issue into a single number, but "trimmed mean" seems exceptionally un-useful for the type of distribution we are dealing with here:
a) we have zero means to differentiate "noise in the measurements, and outliers (which too me are more or less the same, any "outlier" not assigned to a random noise process is data)" from veridical measurements
b) RTT/OWD distributions are IMHO almost never symmetric (there is a hard lower limit, but more slack on the other end)
c) doing a trimmed mean of a skewed distribution will likely result in a higher RPM estimate than using a non-trimmed mean, which given a) seems problematic


The core of the issue is the desire for a single number... but I believe that is non-negotiable by now (similar the use of the inverse of the actually relevant delay data, few folks actually intuitively grasps inverse values and how to operate on them)

	[SM] Also this:

"The more probes that are sent, the more data available for calculation. In order to generate as much data as possible, the Responsiveness Test specifies that a client issue these probes regularly. There is, however, a risk that on low-capacity networks the responsiveness probes themselves will consume a significant amount of the capacity. Because the test mandates first saturating capacity before probing for responsiveness, we are able to accurately estimate how much of the capacity the responsiveness probes will consume and never send more probes than the network can handle."


	[SM] seems convincing, but in my case the reported working load stayed well below the actual capacity, so the "we are able to accurately estimate how much of the capacity the responsiveness probes will consume" seems a bit optimistic...


I understand posting here will not change anything, but I thought is appropriate to make my last post on-topic, for a change ;)

Regards
	Sebastian