<div dir="ltr">Some time recently I read a casual paper (on Medium I think) that made the point that deep diving into outliers and understanding them has led to a half dozen Nobel prizes, because they lead to discoveries of phenomena that nobody else had even noticed. See for instinance the Holmdel Horn <a href="https://en.wikipedia.org/wiki/Holmdel_Horn_Antenna">https://en.wikipedia.org/wiki/Holmdel_Horn_Antenna</a><div><br></div><div>To keep sane, I tend to keep outliers and clip them as last as possible, e.g. by choice of graph axis. This way I have the opportunity to notice otherwise hidden patterns.</div><div><br></div><div>In mlab data we sometimes see outliers that suggest "out of bounds" data rates. e.g. a repeated test that clearly has a max rate of 50 Mb/s or something, and then every so often a one test at 200 Mb/s or higher. My assumption is that these are from software managed shapers that occasionally fail to properly load their configurations. (I admit to not having looked hard enough to prove this hypnosis).<br><div><br></div><div>Thanks,<div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr">--MM--<br>The best way to predict the future is to create it. - Alan Kay<br><br>We must not tolerate intolerance;</div><div dir="ltr"> however our response must be carefully measured: </div><div> too strong would be hypocritical and risks spiraling out of control;</div><div> too weak risks being mistaken for tacit approval.</div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 17, 2021 at 7:25 AM Nick Buraglio <<a href="mailto:buraglio@forwardingplane.net">buraglio@forwardingplane.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">This is much more common in the high performance computing and networking space (i.e. perfsonar, TWAMP, and OWAMP). I have also been pushing "gather and store all the data" for ....since I was an engineer working on the Teragrid (which is where I first saw Matt's MTU talk around 2002 or 03, BTW). =) <div>High fidelity plots of everything that can be gathered is laborious to curate but is invaluable for so many reasons. Now we just need a way to make it happen everywhere for everyone in a way that's easy. </div><div><br></div><div>nb</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 17, 2021 at 8:57 AM Dave Taht <<a href="mailto:davet@teklibre.net" target="_blank">davet@teklibre.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Capturing and plotting *all* the data is often revealing. <br>
<br>
Sometimes plotting the data you are discarding (for what seems like sane reasons) is quite revealing. Saw this on slashdot this morning, it’s good...<br>
<br>
<a href="https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death" rel="noreferrer" target="_blank">https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death</a><br>
<br>
In the bufferbloat effort I’ve fought time and time again for folk to stop throwing out data above the 95 percentile, and at the very least plot everything they threw out to find patterns...<br>
<br>
dslreports’ graphing tools, for example, throws out a ton of “outliers" … and the only reason why there is no data past 4 sec here, is that the test doesn’t run long enough. <br>
<br>
<a href="http://www.dslreports.com/speedtest/results/bufferbloat?up=1" rel="noreferrer" target="_blank">http://www.dslreports.com/speedtest/results/bufferbloat?up=1</a><br>
<br>
(been trying to get ahold of someone over there to buy their raw data for years now. They have the biggest - 8 years worth - collection)<br>
<br>
mlabs has a similar data reduction issue that they haven’t got around to fixing. <br>
<br>
And more recently we encountered a smoothing problem in wireshark that made a halt in packet processing look more like a normal tcp cwnd cut….<br>
<br>
_______________________________________________<br>
Starlink mailing list<br>
<a href="mailto:Starlink@lists.bufferbloat.net" target="_blank">Starlink@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/starlink" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/starlink</a><br>
</blockquote></div>
</blockquote></div>