[Starlink] plotting all the data

[Starlink] plotting all the data

[Dave Taht]

Capturing and plotting *all* the data is often revealing. 

Sometimes plotting the data you are discarding (for what seems like sane reasons) is quite revealing.  Saw this on slashdot this morning, it’s good...

https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death

In the bufferbloat effort I’ve fought time and time again for folk to stop throwing out data above the 95 percentile, and at the very least plot everything they threw out to find patterns...

dslreports’ graphing tools, for example, throws out a ton of “outliers" … and the only reason why there is no data past 4 sec here, is that the test doesn’t run long enough. 

http://www.dslreports.com/speedtest/results/bufferbloat?up=1

(been trying to get ahold of someone over there to buy their raw data for years now. They have the biggest - 8 years worth - collection)

mlabs has a similar data reduction issue that they haven’t got around to fixing. 

And more recently we encountered a smoothing problem in wireshark that made a halt in packet processing look more like a normal tcp cwnd cut….

Re: [Starlink] plotting all the data

[Nick Buraglio]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

This is much more common in the high performance computing and networking
space (i.e. perfsonar, TWAMP, and OWAMP). I have also been pushing "gather
and store all the data" for ....since I was an engineer working on the
Teragrid (which is where I first saw Matt's MTU talk around 2002 or 03,
BTW).  =)
High fidelity plots of everything that can be gathered is laborious to
curate but is invaluable for so many reasons. Now we just need a way to
make it happen everywhere for everyone in a way that's easy.

nb


On Thu, Jun 17, 2021 at 8:57 AM Dave Taht <davet@teklibre.net> wrote:

> Capturing and plotting *all* the data is often revealing.
>
> Sometimes plotting the data you are discarding (for what seems like sane
> reasons) is quite revealing.  Saw this on slashdot this morning, it’s
> good...
>
>
> https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death
>
> In the bufferbloat effort I’ve fought time and time again for folk to stop
> throwing out data above the 95 percentile, and at the very least plot
> everything they threw out to find patterns...
>
> dslreports’ graphing tools, for example, throws out a ton of “outliers" …
> and the only reason why there is no data past 4 sec here, is that the test
> doesn’t run long enough.
>
> http://www.dslreports.com/speedtest/results/bufferbloat?up=1
>
> (been trying to get ahold of someone over there to buy their raw data for
> years now. They have the biggest - 8 years worth - collection)
>
> mlabs has a similar data reduction issue that they haven’t got around to
> fixing.
>
> And more recently we encountered a smoothing problem in wireshark that
> made a halt in packet processing look more like a normal tcp cwnd cut….
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 2717 bytes --]

Re: [Starlink] plotting all the data

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

I once wrote a song about how the challenger disaster reshaped my life, and made me as stubborn as I am about getting and presenting good data and taking sane actions on it.

https://science.slashdot.org/comments.pl?sid=19123104&cid=61496334

I only get the urge to play it nowadays when things somewhere else are going badly wrong. Played it every darn day for the past couple weeks.



> On Jun 17, 2021, at 7:24 AM, Nick Buraglio <buraglio@forwardingplane.net> wrote:
> 
> This is much more common in the high performance computing and networking space (i.e. perfsonar, TWAMP, and OWAMP). I have also been pushing "gather and store all the data" for ....since I was an engineer working on the Teragrid (which is where I first saw Matt's MTU talk around 2002 or 03, BTW).  =) 
> High fidelity plots of everything that can be gathered is laborious to curate but is invaluable for so many reasons. Now we just need a way to make it happen everywhere for everyone in a way that's easy. 
> 
> nb
> 
> 
> On Thu, Jun 17, 2021 at 8:57 AM Dave Taht <davet@teklibre.net <mailto:davet@teklibre.net>> wrote:
> Capturing and plotting *all* the data is often revealing. 
> 
> Sometimes plotting the data you are discarding (for what seems like sane reasons) is quite revealing.  Saw this on slashdot this morning, it’s good...
> 
> https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death <https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death>
> 
> In the bufferbloat effort I’ve fought time and time again for folk to stop throwing out data above the 95 percentile, and at the very least plot everything they threw out to find patterns...
> 
> dslreports’ graphing tools, for example, throws out a ton of “outliers" … and the only reason why there is no data past 4 sec here, is that the test doesn’t run long enough. 
> 
> http://www.dslreports.com/speedtest/results/bufferbloat?up=1 <http://www.dslreports.com/speedtest/results/bufferbloat?up=1>
> 
> (been trying to get ahold of someone over there to buy their raw data for years now. They have the biggest - 8 years worth - collection)
> 
> mlabs has a similar data reduction issue that they haven’t got around to fixing. 
> 
> And more recently we encountered a smoothing problem in wireshark that made a halt in packet processing look more like a normal tcp cwnd cut….
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net <mailto:Starlink@lists.bufferbloat.net>
> https://lists.bufferbloat.net/listinfo/starlink <https://lists.bufferbloat.net/listinfo/starlink>


[-- Attachment #2: Type: text/html, Size: 4134 bytes --]

Re: [Starlink] plotting all the data

[Matt Mathis]

[-- Attachment #1: Type: text/plain, Size: 3366 bytes --]

Some time recently I read a casual paper (on Medium I think) that made the
point that deep diving into outliers and understanding them has led to a
half dozen Nobel prizes, because they lead to discoveries of phenomena that
nobody else had even noticed.  See for instinance the Holmdel Horn
https://en.wikipedia.org/wiki/Holmdel_Horn_Antenna

To keep sane, I tend to keep outliers and clip them as last as possible,
e.g. by choice of graph axis.  This way I have the opportunity to notice
otherwise hidden patterns.

In mlab data we sometimes see outliers that suggest "out of bounds" data
rates.  e.g. a repeated test that clearly has a max rate of 50 Mb/s or
something, and then every so often a one test at 200 Mb/s or higher.   My
assumption is that these are from software managed shapers that
occasionally fail to properly load their configurations.      (I admit to
not having looked hard enough to prove this hypnosis).

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.


On Thu, Jun 17, 2021 at 7:25 AM Nick Buraglio <buraglio@forwardingplane.net>
wrote:

> This is much more common in the high performance computing and networking
> space (i.e. perfsonar, TWAMP, and OWAMP). I have also been pushing "gather
> and store all the data" for ....since I was an engineer working on the
> Teragrid (which is where I first saw Matt's MTU talk around 2002 or 03,
> BTW).  =)
> High fidelity plots of everything that can be gathered is laborious to
> curate but is invaluable for so many reasons. Now we just need a way to
> make it happen everywhere for everyone in a way that's easy.
>
> nb
>
>
> On Thu, Jun 17, 2021 at 8:57 AM Dave Taht <davet@teklibre.net> wrote:
>
>> Capturing and plotting *all* the data is often revealing.
>>
>> Sometimes plotting the data you are discarding (for what seems like sane
>> reasons) is quite revealing.  Saw this on slashdot this morning, it’s
>> good...
>>
>>
>> https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death
>>
>> In the bufferbloat effort I’ve fought time and time again for folk to
>> stop throwing out data above the 95 percentile, and at the very least plot
>> everything they threw out to find patterns...
>>
>> dslreports’ graphing tools, for example, throws out a ton of “outliers" …
>> and the only reason why there is no data past 4 sec here, is that the test
>> doesn’t run long enough.
>>
>> http://www.dslreports.com/speedtest/results/bufferbloat?up=1
>>
>> (been trying to get ahold of someone over there to buy their raw data for
>> years now. They have the biggest - 8 years worth - collection)
>>
>> mlabs has a similar data reduction issue that they haven’t got around to
>> fixing.
>>
>> And more recently we encountered a smoothing problem in wireshark that
>> made a halt in packet processing look more like a normal tcp cwnd cut….
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>

[-- Attachment #2: Type: text/html, Size: 4735 bytes --]

Re: [Starlink] plotting all the data

[George Burdell]

On Thu, Jun 17, 2021 at 08:49:22AM -0700, Matt Mathis wrote:
> Some time recently I read a casual paper (on Medium I think) that made the
> point that deep diving into outliers and understanding them has led to a
> half dozen Nobel prizes, because they lead to discoveries of phenomena that

Regrettably there are no Nobel prizes for networking. The closest thing
we have is the jon postel award:

https://www.internetsociety.org/blog/2021/05/nominations-open-jonathan-b-postel-service-award-2021/

> nobody else had even noticed.  See for instinance the Holmdel Horn
> https://en.wikipedia.org/wiki/Holmdel_Horn_Antenna

We use a stylized version of that image for the 

"cosmic background bufferbloat detector array". :)

We're pretty close to getting that fully online, with just some backend
software left to write, and some front end code to port to iOS, and
to add more clouds to the array.

If anyone here has sufficient go chops to get irtt ported to ios it would help.

postgres skils also.

> 
> To keep sane, I tend to keep outliers and clip them as last as possible,
> e.g. by choice of graph axis.  This way I have the opportunity to notice
> otherwise hidden patterns.

A combination of the full dataset plotted linearly along with a cdf plot
works best for us.

> 
> In mlab data we sometimes see outliers that suggest "out of bounds" data
> rates.  e.g. a repeated test that clearly has a max rate of 50 Mb/s or
> something, and then every so often a one test at 200 Mb/s or higher.   My
> assumption is that these are from software managed shapers that
> occasionally fail to properly load their configurations.      (I admit to
> not having looked hard enough to prove this hypnosis).

I am mostly interested in detecting the "sloshiness" from the broken 
virtio-net driver (lacking bql, and being quite common in clouds), at
the moment.

> 
> Thanks,
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
> 
> We must not tolerate intolerance;
>        however our response must be carefully measured:
>             too strong would be hypocritical and risks spiraling out of
> control;
>             too weak risks being mistaken for tacit approval.
> 
> 
> On Thu, Jun 17, 2021 at 7:25 AM Nick Buraglio <buraglio@forwardingplane.net>
> wrote:
> 
> > This is much more common in the high performance computing and networking
> > space (i.e. perfsonar, TWAMP, and OWAMP). I have also been pushing "gather
> > and store all the data" for ....since I was an engineer working on the
> > Teragrid (which is where I first saw Matt's MTU talk around 2002 or 03,
> > BTW).  =)
> > High fidelity plots of everything that can be gathered is laborious to
> > curate but is invaluable for so many reasons. Now we just need a way to
> > make it happen everywhere for everyone in a way that's easy.
> >
> > nb
> >
> >
> > On Thu, Jun 17, 2021 at 8:57 AM Dave Taht <davet@teklibre.net> wrote:
> >
> >> Capturing and plotting *all* the data is often revealing.
> >>
> >> Sometimes plotting the data you are discarding (for what seems like sane
> >> reasons) is quite revealing.  Saw this on slashdot this morning, it’s
> >> good...
> >>
> >>
> >> https://www.newyorker.com/magazine/2021/06/21/when-graphs-are-a-matter-of-life-and-death
> >>
> >> In the bufferbloat effort I’ve fought time and time again for folk to
> >> stop throwing out data above the 95 percentile, and at the very least plot
> >> everything they threw out to find patterns...
> >>
> >> dslreports’ graphing tools, for example, throws out a ton of “outliers" …
> >> and the only reason why there is no data past 4 sec here, is that the test
> >> doesn’t run long enough.
> >>
> >> http://www.dslreports.com/speedtest/results/bufferbloat?up=1
> >>
> >> (been trying to get ahold of someone over there to buy their raw data for
> >> years now. They have the biggest - 8 years worth - collection)
> >>
> >> mlabs has a similar data reduction issue that they haven’t got around to
> >> fixing.
> >>
> >> And more recently we encountered a smoothing problem in wireshark that
> >> made a halt in packet processing look more like a normal tcp cwnd cut….
> >>
> >> _______________________________________________
> >> Starlink mailing list
> >> Starlink@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/starlink
> >>
> >

> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink