[Starlink] Researchers Seeking Probe Volunteers in USA

Tue Jan 3 17:58:32 EST 2023

A serious question: 

> ... The QMap Probe is
> preconfigured to automatically test various network metrics with little burden on
> your available bandwidth.

Those of us here, like me and Dave Taht, who have measured the big elephants in the room (esp. for Starlink) like "lag under load" and "fairness with respect to competing traffic on the same <link>" probably were not consulted, if the goal is "little burden on your available bandwidth".

I've spent many years reading papers about "low cost metrics" for lag and fairness in the Internet context, starting with the old "packet-pair" techniques, and basically they are almost impossible to interpret and compare between service provider architectures. It's hard enough to compare DOCSIS 3 setups with other DOCSIS 3 CMTS's in other cities, or LTE data networks between different cities.

Frankly, I expect the results will be treated like other "quality metrics" - J.D. Power comes to mind from consulting experience in the automotive industry - and be cherry-picked to distort the results.

> This data is used to estimate the quality of internet
> service.

I have NEVER seen a technical definition of "quality of service" that made sense in terms of how the users experience their use of the Internet. It would be wonderful if such a definition actually existed. So how does relative "estimation" of an undefined concept work?

What questions really matter to users, in particular? Well, "availability" might matter, but availability is relative to "momentary need". A network that is 99.9% available, but the 0.1% of the time that the user NEEDS it is what matters to the user, not the rest of the 86,400 seconds each day. 

[As an aside, in a proceeding I participated under the  CRTC, Canada's "FCC" regulator on Network Management that focused on "quality", we queried that since none of the operators in Canada actually measured "response time" of their networks in any way, so how could they know that they were improving service? The response on the record from some of the largest Broadband ISPs in Canada was discouraging. They said I was wrong, and that they constantly measured "utilization" of the network capacity at every router, and the *average* utilization was almost always < 85%. They then invoked Little's Lemma in queueing theory to say that proved that the quality of service was *perfect*.
This was in a legal regulatory proceeding, *under oath*. I just cannot understand how folks technical enough to invoke Little's Lemma could be so ignorant. Little's Lemma isn't at all good at converting "average utilization" to "user experienced lag under load", it's mathematically *invalid*. But what is worse is that they had no idea of what user experienced "quality" was. It's like a software vendor saying they have no bugs, that they know about, when they have no way for users to report bugs at all. OR in the modern context, where reporting bugs is a hassle and there's no "bug bounty" or expectation that the company will fix a reported bug in a timely manner.]

> A public report of our findings will be published on our website in 2023.

By all means participate if you want, but I suspect that the "raw data" will not be made available, and looking at the existing reports, it will be hard to extract meaningful comparisons relevant to real user experience at the test sites.

> Please check out some of our existing reports to get a better feel of what we
> measure:
> https://www.netforecast.com/audit-reports/<https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>.
> 
> The QMap Probe requires no regular maintenance or monitoring by you. It may
> occasionally need rebooting (turning off and on), and we would contact you via
> email or text to request this. The device sends various test packets to the
> internet and records their response characteristics. Our device has no knowledge
> of -- and does not communicate out -- any information about you or any devices in
> your home.
> 
> We will include a prepaid shipping label for returning the QMap Probe in the
> original box (please keep this!). Once we receive the device back, we will send
> you a $200 Amazon gift card (one per household).
> 
> To volunteer, please fill out this relatively painless survey:
> https://www.surveymonkey.com/r/8VZSB3M<https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>.
> Thank you!
> 
> ///END///
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/aa1e6019/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Tue, 3 Jan 2023 15:57:30 -0500
> From: Vint Cerf <vint at google.com>
> To: "Livingood, Jason" <Jason_Livingood at comcast.com>
> Cc: Dave Taht via Starlink <starlink at lists.bufferbloat.net>
> Subject: Re: [Starlink] Researchers Seeking Probe Volunteers in USA
> Message-ID:
> <CAHxHgge6sTffAqaMLv7z1k0ZnYtWw7s+OgXBJOBnm5zAwHjR+w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> netforecast was started by a good friend of mine - they are first rate.
> 
> v
> 
> 
> On Tue, Jan 3, 2023 at 3:53 PM Livingood, Jason via Starlink <
> starlink at lists.bufferbloat.net> wrote:
> 
> > Forwarding on from a group doing some Starlink research. I am aware of at
> > least one other researcher that will soon do the same and will forward that
> > later (guessing a week or two).
> >
> >
> >
> > Jason
> >
> > ///FORWARD///
> >
> > We need volunteers! NetForecast, a leader in measuring the quality of
> > internet service, is conducting a performance study of several types of
> > internet delivery technologies, including low-earth-orbit satellite like
> > Starlink.
> >
> >
> >
> > As a volunteer, you will host one of our proprietary QMap Probes in your
> > home network. This will connect to the internet via an available ethernet
> > port on your gateway/router and plug into a standard power outlet. The QMap
> > Probe is preconfigured to automatically test various network metrics with
> > little burden on your available bandwidth. This data is used to estimate
> > the quality of internet service. A public report of our findings will be
> > published on our website in 2023. Please check out some of our existing
> > reports to get a better feel of what we measure:
> > https://www.netforecast.com/audit-reports/
> >
> <https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>
> > .
> >
> >
> >
> > The QMap Probe requires no regular maintenance or monitoring by you. It
> > may occasionally need rebooting (turning off and on), and we would contact
> > you via email or text to request this. The device sends various test
> > packets to the internet and records their response characteristics. Our
> > device has no knowledge of -- and does not communicate out -- any
> > information about you or any devices in your home.
> >
> >
> >
> > We will include a prepaid shipping label for returning the QMap Probe in
> > the original box (please keep this!). Once we receive the device back, we
> > will send you a $200 Amazon gift card (one per household).
> >
> >
> >
> > To volunteer, please fill out this relatively painless survey:
> > https://www.surveymonkey.com/r/8VZSB3M
> >
> <https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>.
> > Thank you!
> >
> >
> >
> > ///END///
> > _______________________________________________
> > Starlink mailing list
> > Starlink at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> >
> 
> 
> --
> Please send any postal/overnight deliveries to:
> Vint Cerf
> Google, LLC
> 1900 Reston Metro Plaza, 16th Floor
> Reston, VA 20190
> +1 (571) 213 1346
> 
> 
> until further notice
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/e49cfc9f/attachment.html>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 3995 bytes
> Desc: S/MIME Cryptographic Signature
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/e49cfc9f/attachment.bin>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Starlink mailing list
> Starlink at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> ------------------------------
> 
> End of Starlink Digest, Vol 22, Issue 6
> ***************************************
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/34f4bd1c/attachment-0001.html>