[Starlink] Researchers Seeking Probe Volunteers in USA

[Starlink] Researchers Seeking Probe Volunteers in USA

[Livingood, Jason]

[-- Attachment #1: Type: text/plain, Size: 2159 bytes --]

Forwarding on from a group doing some Starlink research. I am aware of at least one other researcher that will soon do the same and will forward that later (guessing a week or two).

Jason
///FORWARD///
We need volunteers! NetForecast, a leader in measuring the quality of internet service, is conducting a performance study of several types of internet delivery technologies, including low-earth-orbit satellite like Starlink.

As a volunteer, you will host one of our proprietary QMap Probes in your home network. This will connect to the internet via an available ethernet port on your gateway/router and plug into a standard power outlet. The QMap Probe is preconfigured to automatically test various network metrics with little burden on your available bandwidth. This data is used to estimate the quality of internet service. A public report of our findings will be published on our website in 2023. Please check out some of our existing reports to get a better feel of what we measure: https://www.netforecast.com/audit-reports/<https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>.

The QMap Probe requires no regular maintenance or monitoring by you. It may occasionally need rebooting (turning off and on), and we would contact you via email or text to request this. The device sends various test packets to the internet and records their response characteristics. Our device has no knowledge of -- and does not communicate out -- any information about you or any devices in your home.

We will include a prepaid shipping label for returning the QMap Probe in the original box (please keep this!). Once we receive the device back, we will send you a $200 Amazon gift card (one per household).

To volunteer, please fill out this relatively painless survey: https://www.surveymonkey.com/r/8VZSB3M<https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>. Thank you!

///END///

[-- Attachment #2: Type: text/html, Size: 6408 bytes --]

Re: [Starlink] Researchers Seeking Probe Volunteers in USA

[Vint Cerf]

[-- Attachment #1.1: Type: text/plain, Size: 2758 bytes --]

netforecast was started by a good friend of mine - they are first rate.

v


On Tue, Jan 3, 2023 at 3:53 PM Livingood, Jason via Starlink <
starlink@lists.bufferbloat.net> wrote:

> Forwarding on from a group doing some Starlink research. I am aware of at
> least one other researcher that will soon do the same and will forward that
> later (guessing a week or two).
>
>
>
> Jason
>
> ///FORWARD///
>
> We need volunteers! NetForecast, a leader in measuring the quality of
> internet service, is conducting a performance study of several types of
> internet delivery technologies, including low-earth-orbit satellite like
> Starlink.
>
>
>
> As a volunteer, you will host one of our proprietary QMap Probes in your
> home network. This will connect to the internet via an available ethernet
> port on your gateway/router and plug into a standard power outlet. The QMap
> Probe is preconfigured to automatically test various network metrics with
> little burden on your available bandwidth. This data is used to estimate
> the quality of internet service. A public report of our findings will be
> published on our website in 2023. Please check out some of our existing
> reports to get a better feel of what we measure:
> https://www.netforecast.com/audit-reports/
> <https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>
> .
>
>
>
> The QMap Probe requires no regular maintenance or monitoring by you. It
> may occasionally need rebooting (turning off and on), and we would contact
> you via email or text to request this. The device sends various test
> packets to the internet and records their response characteristics. Our
> device has no knowledge of -- and does not communicate out -- any
> information about you or any devices in your home.
>
>
>
> We will include a prepaid shipping label for returning the QMap Probe in
> the original box (please keep this!). Once we receive the device back, we
> will send you a $200 Amazon gift card (one per household).
>
>
>
> To volunteer, please fill out this relatively painless survey:
> https://www.surveymonkey.com/r/8VZSB3M
> <https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>.
> Thank you!
>
>
>
> ///END///
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Please send any postal/overnight deliveries to:
Vint Cerf
Google, LLC
1900 Reston Metro Plaza, 16th Floor
Reston, VA 20190
+1 (571) 213 1346


until further notice

[-- Attachment #1.2: Type: text/html, Size: 5842 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3995 bytes --]

Re: [Starlink] Researchers Seeking Probe Volunteers in USA

[Eric]

[-- Attachment #1: Type: text/plain, Size: 2732 bytes --]

Jason,

(Replying on-list in case anyone else has the same question.)

Questionnaire #4:
Is the "port available on the router" a hard requirement, or is a port on a switch acceptable? My gateway router only has three ports, so one to WAN, other two to LAN switches (which connect to servers, workstations, WAPs and another switch). Would connecting the probe to a single-hop switch cause any issues?

Thanks,
Eric

------- Original Message -------
On Tuesday, January 3rd, 2023 at 12:53, Livingood, Jason via Starlink <starlink@lists.bufferbloat.net> wrote:

> Forwarding on from a group doing some Starlink research. I am aware of at least one other researcher that will soon do the same and will forward that later (guessing a week or two).
>
> Jason
>
> ///FORWARD///
>
> We need volunteers! NetForecast, a leader in measuring the quality of internet service, is conducting a performance study of several types of internet delivery technologies, including low-earth-orbit satellite like Starlink.
>
> As a volunteer, you will host one of our proprietary QMap Probes in your home network. This will connect to the internet via an available ethernet port on your gateway/router and plug into a standard power outlet. The QMap Probe is preconfigured to automatically test various network metrics with little burden on your available bandwidth. This data is used to estimate the quality of internet service. A public report of our findings will be published on our website in 2023. Please check out some of our existing reports to get a better feel of what we measure:[https://www.netforecast.com/audit-reports/](https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$).
>
> The QMap Probe requires no regular maintenance or monitoring by you. It may occasionally need rebooting (turning off and on), and we would contact you via email or text to request this. The device sends various test packets to the internet and records their response characteristics. Our device has no knowledge of -- and does not communicate out -- any information about you or any devices in your home.
>
> We will include a prepaid shipping label for returning the QMap Probe in the original box (please keep this!). Once we receive the device back, we will send you a $200 Amazon gift card (one per household).
>
> To volunteer, please fill out this relatively painless survey:[https://www.surveymonkey.com/r/8VZSB3M](https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$). Thank you!
>
> ///END///

[-- Attachment #2: Type: text/html, Size: 6732 bytes --]

Re: [Starlink] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA

[Livingood, Jason]

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

Not sure – I would ask their lead researcher (mike.reynolds@netforecast.com<mailto:mike.reynolds@netforecast.com>), cc’d here. I wouldn’t think it was an issue but never know.

JL

From: Eric <evil.function@proton.me>
Date: Tuesday, January 3, 2023 at 16:24
To: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>
Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net>
Subject: [EXTERNAL] Re: [Starlink] Researchers Seeking Probe Volunteers in USA

Jason,

(Replying on-list in case anyone else has the same question.)

Questionnaire #4:
Is the "port available on the router" a hard requirement, or is a port on a switch acceptable?  My gateway router only has three ports, so one to WAN, other two to LAN switches (which connect to servers, workstations, WAPs and another switch).  Would connecting the probe to a single-hop switch cause any issues?

Thanks,
Eric


------- Original Message -------
On Tuesday, January 3rd, 2023 at 12:53, Livingood, Jason via Starlink <starlink@lists.bufferbloat.net> wrote:


Forwarding on from a group doing some Starlink research. I am aware of at least one other researcher that will soon do the same and will forward that later (guessing a week or two).

Jason
///FORWARD///
We need volunteers! NetForecast, a leader in measuring the quality of internet service, is conducting a performance study of several types of internet delivery technologies, including low-earth-orbit satellite like Starlink.

As a volunteer, you will host one of our proprietary QMap Probes in your home network. This will connect to the internet via an available ethernet port on your gateway/router and plug into a standard power outlet. The QMap Probe is preconfigured to automatically test various network metrics with little burden on your available bandwidth. This data is used to estimate the quality of internet service. A public report of our findings will be published on our website in 2023. Please check out some of our existing reports to get a better feel of what we measure: https://www.netforecast.com/audit-reports/<https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>.

The QMap Probe requires no regular maintenance or monitoring by you. It may occasionally need rebooting (turning off and on), and we would contact you via email or text to request this. The device sends various test packets to the internet and records their response characteristics. Our device has no knowledge of -- and does not communicate out -- any information about you or any devices in your home.

We will include a prepaid shipping label for returning the QMap Probe in the original box (please keep this!). Once we receive the device back, we will send you a $200 Amazon gift card (one per household).

To volunteer, please fill out this relatively painless survey: https://www.surveymonkey.com/r/8VZSB3M<https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>. Thank you!

///END///


[-- Attachment #2: Type: text/html, Size: 10885 bytes --]

Re: [Starlink] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA

[Mike Reynolds]

[-- Attachment #1: Type: text/plain, Size: 3877 bytes --]

Port on a switch is perfectly fine. Sorry about the confusion.

-Mike

From: Livingood, Jason <Jason_Livingood@comcast.com>
Sent: Monday, January 9, 2023 9:35 AM
To: Eric <evil.function@proton.me>
Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net>; Mike Reynolds <mike.reynolds@netforecast.com>
Subject: Re: [EXTERNAL] Re: [Starlink] Researchers Seeking Probe Volunteers in USA

CAUTION: This email originated from outside of NetForecast. Use caution when clicking links or opening attachments.
Not sure – I would ask their lead researcher (mike.reynolds@netforecast.com<mailto:mike.reynolds@netforecast.com>), cc’d here. I wouldn’t think it was an issue but never know.

JL

From: Eric <evil.function@proton.me<mailto:evil.function@proton.me>>
Date: Tuesday, January 3, 2023 at 16:24
To: "Livingood, Jason" <Jason_Livingood@cable.comcast.com<mailto:Jason_Livingood@cable.comcast.com>>
Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.net>>
Subject: [EXTERNAL] Re: [Starlink] Researchers Seeking Probe Volunteers in USA

Jason,

(Replying on-list in case anyone else has the same question.)

Questionnaire #4:
Is the "port available on the router" a hard requirement, or is a port on a switch acceptable?  My gateway router only has three ports, so one to WAN, other two to LAN switches (which connect to servers, workstations, WAPs and another switch).  Would connecting the probe to a single-hop switch cause any issues?

Thanks,
Eric


------- Original Message -------
On Tuesday, January 3rd, 2023 at 12:53, Livingood, Jason via Starlink <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.net>> wrote:

Forwarding on from a group doing some Starlink research. I am aware of at least one other researcher that will soon do the same and will forward that later (guessing a week or two).

Jason
///FORWARD///
We need volunteers! NetForecast, a leader in measuring the quality of internet service, is conducting a performance study of several types of internet delivery technologies, including low-earth-orbit satellite like Starlink.

As a volunteer, you will host one of our proprietary QMap Probes in your home network. This will connect to the internet via an available ethernet port on your gateway/router and plug into a standard power outlet. The QMap Probe is preconfigured to automatically test various network metrics with little burden on your available bandwidth. This data is used to estimate the quality of internet service. A public report of our findings will be published on our website in 2023. Please check out some of our existing reports to get a better feel of what we measure: https://www.netforecast.com/audit-reports/<https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>.

The QMap Probe requires no regular maintenance or monitoring by you. It may occasionally need rebooting (turning off and on), and we would contact you via email or text to request this. The device sends various test packets to the internet and records their response characteristics. Our device has no knowledge of -- and does not communicate out -- any information about you or any devices in your home.

We will include a prepaid shipping label for returning the QMap Probe in the original box (please keep this!). Once we receive the device back, we will send you a $200 Amazon gift card (one per household).

To volunteer, please fill out this relatively painless survey: https://www.surveymonkey.com/r/8VZSB3M<https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>. Thank you!

///END///


[-- Attachment #2: Type: text/html, Size: 12328 bytes --]

Re: [Starlink] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA

[Eric]

[-- Attachment #1: Type: text/plain, Size: 3884 bytes --]

Got it, thanks!

------- Original Message -------
On Monday, January 9th, 2023 at 06:38, Mike Reynolds <mike.reynolds@netforecast.com> wrote:

> Port on a switch is perfectly fine. Sorry about the confusion.
>
> -Mike
>
> From: Livingood, Jason <Jason_Livingood@comcast.com>
> Sent: Monday, January 9, 2023 9:35 AM
> To: Eric <evil.function@proton.me>
> Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net>; Mike Reynolds <mike.reynolds@netforecast.com>
> Subject: Re: [EXTERNAL] Re: [Starlink] Researchers Seeking Probe Volunteers in USA
>
> CAUTION: This email originated from outside of NetForecast. Use caution when clicking links or opening attachments.
>
> Not sure – I would ask their lead researcher (mike.reynolds@netforecast.com), cc’d here. I wouldn’t think it was an issue but never know.
>
> JL
>
> From: Eric <evil.function@proton.me>
> Date: Tuesday, January 3, 2023 at 16:24
> To: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>
> Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net>
> Subject: [EXTERNAL] Re: [Starlink] Researchers Seeking Probe Volunteers in USA
>
> Jason,
>
> (Replying on-list in case anyone else has the same question.)
>
> Questionnaire #4:
>
> Is the "port available on the router" a hard requirement, or is a port on a switch acceptable? My gateway router only has three ports, so one to WAN, other two to LAN switches (which connect to servers, workstations, WAPs and another switch). Would connecting the probe to a single-hop switch cause any issues?
>
> Thanks,
>
> Eric
>
> ------- Original Message -------
> On Tuesday, January 3rd, 2023 at 12:53, Livingood, Jason via Starlink <starlink@lists.bufferbloat.net> wrote:
>
>> Forwarding on from a group doing some Starlink research. I am aware of at least one other researcher that will soon do the same and will forward that later (guessing a week or two).
>>
>> Jason
>>
>> ///FORWARD///
>>
>> We need volunteers! NetForecast, a leader in measuring the quality of internet service, is conducting a performance study of several types of internet delivery technologies, including low-earth-orbit satellite like Starlink.
>>
>> As a volunteer, you will host one of our proprietary QMap Probes in your home network. This will connect to the internet via an available ethernet port on your gateway/router and plug into a standard power outlet. The QMap Probe is preconfigured to automatically test various network metrics with little burden on your available bandwidth. This data is used to estimate the quality of internet service. A public report of our findings will be published on our website in 2023. Please check out some of our existing reports to get a better feel of what we measure:[https://www.netforecast.com/audit-reports/](https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$).
>>
>> The QMap Probe requires no regular maintenance or monitoring by you. It may occasionally need rebooting (turning off and on), and we would contact you via email or text to request this. The device sends various test packets to the internet and records their response characteristics. Our device has no knowledge of -- and does not communicate out -- any information about you or any devices in your home.
>>
>> We will include a prepaid shipping label for returning the QMap Probe in the original box (please keep this!). Once we receive the device back, we will send you a $200 Amazon gift card (one per household).
>>
>> To volunteer, please fill out this relatively painless survey:[https://www.surveymonkey.com/r/8VZSB3M](https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$). Thank you!
>>
>> ///END///

[-- Attachment #2: Type: text/html, Size: 11057 bytes --]

Re: [Starlink] Researchers Seeking Probe Volunteers in USA

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 9233 bytes --]


A serious question: 
 

> ... The QMap Probe is
> preconfigured to automatically test various network metrics with little burden on
> your available bandwidth.
 
Those of us here, like me and Dave Taht, who have measured the big elephants in the room (esp. for Starlink) like "lag under load" and "fairness with respect to competing traffic on the same <link>" probably were not consulted, if the goal is "little burden on your available bandwidth".
 
I've spent many years reading papers about "low cost metrics" for lag and fairness in the Internet context, starting with the old "packet-pair" techniques, and basically they are almost impossible to interpret and compare between service provider architectures. It's hard enough to compare DOCSIS 3 setups with other DOCSIS 3 CMTS's in other cities, or LTE data networks between different cities.
 
Frankly, I expect the results will be treated like other "quality metrics" - J.D. Power comes to mind from consulting experience in the automotive industry - and be cherry-picked to distort the results.
 
> This data is used to estimate the quality of internet
> service.
 
 
I have NEVER seen a technical definition of "quality of service" that made sense in terms of how the users experience their use of the Internet. It would be wonderful if such a definition actually existed. So how does relative "estimation" of an undefined concept work?
 
What questions really matter to users, in particular? Well, "availability" might matter, but availability is relative to "momentary need". A network that is 99.9% available, but the 0.1% of the time that the user NEEDS it is what matters to the user, not the rest of the 86,400 seconds each day. 
 
[As an aside, in a proceeding I participated under the  CRTC, Canada's "FCC" regulator on Network Management that focused on "quality", we queried that since none of the operators in Canada actually measured "response time" of their networks in any way, so how could they know that they were improving service? The response on the record from some of the largest Broadband ISPs in Canada was discouraging. They said I was wrong, and that they constantly measured "utilization" of the network capacity at every router, and the *average* utilization was almost always < 85%. They then invoked Little's Lemma in queueing theory to say that proved that the quality of service was *perfect*.
This was in a legal regulatory proceeding, *under oath*. I just cannot understand how folks technical enough to invoke Little's Lemma could be so ignorant. Little's Lemma isn't at all good at converting "average utilization" to "user experienced lag under load", it's mathematically *invalid*. But what is worse is that they had no idea of what user experienced "quality" was. It's like a software vendor saying they have no bugs, that they know about, when they have no way for users to report bugs at all. OR in the modern context, where reporting bugs is a hassle and there's no "bug bounty" or expectation that the company will fix a reported bug in a timely manner.]
 
 
 
 
> A public report of our findings will be published on our website in 2023.
 
By all means participate if you want, but I suspect that the "raw data" will not be made available, and looking at the existing reports, it will be hard to extract meaningful comparisons relevant to real user experience at the test sites.
 

> Please check out some of our existing reports to get a better feel of what we
> measure:
> https://www.netforecast.com/audit-reports/<https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>.
> 
> The QMap Probe requires no regular maintenance or monitoring by you. It may
> occasionally need rebooting (turning off and on), and we would contact you via
> email or text to request this. The device sends various test packets to the
> internet and records their response characteristics. Our device has no knowledge
> of -- and does not communicate out -- any information about you or any devices in
> your home.
> 
> We will include a prepaid shipping label for returning the QMap Probe in the
> original box (please keep this!). Once we receive the device back, we will send
> you a $200 Amazon gift card (one per household).
> 
> To volunteer, please fill out this relatively painless survey:
> https://www.surveymonkey.com/r/8VZSB3M<https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>.
> Thank you!
> 
> ///END///
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/aa1e6019/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Tue, 3 Jan 2023 15:57:30 -0500
> From: Vint Cerf <vint@google.com>
> To: "Livingood, Jason" <Jason_Livingood@comcast.com>
> Cc: Dave Taht via Starlink <starlink@lists.bufferbloat.net>
> Subject: Re: [Starlink] Researchers Seeking Probe Volunteers in USA
> Message-ID:
> <CAHxHgge6sTffAqaMLv7z1k0ZnYtWw7s+OgXBJOBnm5zAwHjR+w@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> netforecast was started by a good friend of mine - they are first rate.
> 
> v
> 
> 
> On Tue, Jan 3, 2023 at 3:53 PM Livingood, Jason via Starlink <
> starlink@lists.bufferbloat.net> wrote:
> 
> > Forwarding on from a group doing some Starlink research. I am aware of at
> > least one other researcher that will soon do the same and will forward that
> > later (guessing a week or two).
> >
> >
> >
> > Jason
> >
> > ///FORWARD///
> >
> > We need volunteers! NetForecast, a leader in measuring the quality of
> > internet service, is conducting a performance study of several types of
> > internet delivery technologies, including low-earth-orbit satellite like
> > Starlink.
> >
> >
> >
> > As a volunteer, you will host one of our proprietary QMap Probes in your
> > home network. This will connect to the internet via an available ethernet
> > port on your gateway/router and plug into a standard power outlet. The QMap
> > Probe is preconfigured to automatically test various network metrics with
> > little burden on your available bandwidth. This data is used to estimate
> > the quality of internet service. A public report of our findings will be
> > published on our website in 2023. Please check out some of our existing
> > reports to get a better feel of what we measure:
> > https://www.netforecast.com/audit-reports/
> >
> <https://urldefense.com/v3/__https:/www.netforecast.com/audit-reports/__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFKpAJeQfQ$>
> > .
> >
> >
> >
> > The QMap Probe requires no regular maintenance or monitoring by you. It
> > may occasionally need rebooting (turning off and on), and we would contact
> > you via email or text to request this. The device sends various test
> > packets to the internet and records their response characteristics. Our
> > device has no knowledge of -- and does not communicate out -- any
> > information about you or any devices in your home.
> >
> >
> >
> > We will include a prepaid shipping label for returning the QMap Probe in
> > the original box (please keep this!). Once we receive the device back, we
> > will send you a $200 Amazon gift card (one per household).
> >
> >
> >
> > To volunteer, please fill out this relatively painless survey:
> > https://www.surveymonkey.com/r/8VZSB3M
> >
> <https://urldefense.com/v3/__https:/www.surveymonkey.com/r/8VZSB3M__;!!CQl3mcHX2A!FrL2Yijo-63gS4PMToq0adfntj2fhza8ekyba1EbS8-tCgsQpg5MsIAYAvP5xUzLdDRa667bslUTtw_s0WpvvBpJEFJ_EHoUnA$>.
> > Thank you!
> >
> >
> >
> > ///END///
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> >
> 
> 
> --
> Please send any postal/overnight deliveries to:
> Vint Cerf
> Google, LLC
> 1900 Reston Metro Plaza, 16th Floor
> Reston, VA 20190
> +1 (571) 213 1346
> 
> 
> until further notice
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/e49cfc9f/attachment.html>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 3995 bytes
> Desc: S/MIME Cryptographic Signature
> URL:
> <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230103/e49cfc9f/attachment.bin>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> ------------------------------
> 
> End of Starlink Digest, Vol 22, Issue 6
> ***************************************
> 

[-- Attachment #2: Type: text/html, Size: 13560 bytes --]

Re: [Starlink] Researchers Seeking Probe Volunteers in USA

[Livingood, Jason]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

> Those of us here, like me and Dave Taht, who have measured the big elephants in the room (esp. for Starlink) like "lag under load" and "fairness with respect to competing traffic on the same <link>" probably were not consulted, if the goal is "little burden on your available bandwidth".



I don’t have specifics for their test config, but most of the platforms would determine ‘little burden’ by looking for cross traffic (aka user demand on the connection) and if it is non-existent/low then running tests that can highly utilize the link capacity – whether for a working latency test or whatever.



> Frankly, I expect the results will be treated like other "quality metrics" - J.D. Power comes to mind from consulting experience in the automotive industry - and be cherry-picked to distort the results.



I dunno – I think the research & measurement community seems to be coalescing around certain types of working latency / responsiveness measures as being pretty good & predictive of real end user application QoE.



> By all means participate if you want, but I suspect that the "raw data" will not be made available, and looking at the existing reports, it will be hard to extract meaningful comparisons relevant to real user experience at the test sites.



Not sure if the raw data will be available. Even if not, they may publish the parameters of the tests themselves.



JL

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]

Re: [Starlink] Researchers Seeking Probe Volunteers in USA

[Dave Taht]

I have many kvetches about the new latency under load tests being
designed and distributed over the past year. I am delighted! that they
are happening, but most really need third party evaluation, and
calibration, and a solid explanation of what network pathologies they
do and don't cover. Also a RED team attitude towards them, as well as
thinking hard about what you are not measuring (operations research).

I actually rather love the new cloudflare speedtest, because it tests
a single TCP connection, rather than dozens, and at the same time folk
are complaining that it doesn't find the actual "speed!". yet... the
test itself more closely emulates a user experience than speedtest.net
does. I am personally pretty convinced that the fewer numbers of flows
that a web page opens improves the likelihood of a good user
experience, but lack data on it.

To try to tackle the evaluation and calibration part, I've reached out
to all the new test designers in the hope that we could get together
and produce a report of what each new test is actually doing. I've
tweeted, linked in, emailed, and spammed every measurement list I know
of, and only to some response, please reach out to other test designer
folks and have them join the rpm email list?

My principal kvetches in the new tests so far are:

0) None of the tests last long enough.

Ideally there should be a mode where they at least run to "time of
first loss", or periodically, just run longer than the
industry-stupid^H^H^H^H^H^Hstandard 20 seconds. There be dragons
there! It's really bad science to optimize the internet for 20
seconds. It's like optimizing a car, to handle well, for just 20
seconds.

1) Not testing up + down + ping at the same time

None of the new tests actually test the same thing that the infamous
rrul test does - all the others still test up, then down, and ping. It
was/remains my hope that the simpler parts of the flent test suite -
such as the tcp_up_squarewave tests, the rrul test, and the rtt_fair
tests would provide calibration to the test designers.

we've got zillions of flent results in the archive published here:
https://blog.cerowrt.org/post/found_in_flent/

The new tests have all added up + ping and down + ping, but not up +
down + ping. Why??

The behaviors of what happens in that case are really non-intuitive, I
know, but... it's just one more phase to add to any one of those new
tests. I'd be deliriously happy if someone(s) new to the field
started doing that, even optionally, and boggled at how it defeated
their assumptions.

Among other things that would show...

It's the home router industry's dirty secret than darn few "gigabit"
home routers can actually forward in both directions at a gigabit. I'd
like to smash that perception thoroughly, but given our starting point
is a gigabit router was a "gigabit switch" - and historically been
something that couldn't even forward at 200Mbit - we have a long way
to go there.

Only in the past year have non-x86 home routers appeared that could
actually do a gbit in both directions.

2) Few are actually testing within-stream latency

Apple's rpm project is making a stab in that direction. It looks
highly likely, that with a little more work, crusader and
go-responsiveness can finally start sampling the tcp RTT, loss and
markings, more directly. As for the rest... sampling TCP_INFO on
windows, and Linux, at least, always appeared simple to me, but I'm
discovering how hard it is by delving deep into the rust behind
crusader.

the goresponsiveness thing is also IMHO running WAY too many streams
at the same time, I guess motivated by an attempt to have the test
complete quickly?

B) To try and tackle the validation problem:

In the libreqos.io project we've established a testbed where tests can
be plunked through various ISP plan network emulations. It's here:
https://payne.taht.net (run bandwidth test for what's currently hooked
up)

We could rather use an AS number and at least a ipv4/24 and ipv6/48 to
leverage with that, so I don't have to nat the various emulations.
(and funding, anyone got funding?) Or, as the code is GPLv2 licensed,
to see more test designers setup a testbed like this to calibrate
their own stuff.

Presently we're able to test:
flent
netperf
iperf2
iperf3
speedtest-cli
crusader
the broadband forum udp based test:
https://github.com/BroadbandForum/obudpst
trexx

There's also a virtual machine setup that we can remotely drive a web
browser from (but I didn't want to nat the results to the world) to
test other web services.

Re: [Starlink] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA

[Livingood, Jason]

> 0) None of the tests last long enough.

The user-initiated ones tend to be shorter - likely because the average user does not want to wait several minutes for a test to complete. But IMO this is where a test platform like SamKnows, Ookla's embedded client, NetMicroscope, and others can come in - since they run in the background on some randomized schedule w/o user intervention. Thus, the user's time-sensitivity is no longer a factor and a longer duration test can be performed.

> 1) Not testing up + down + ping at the same time

You should consider publishing a LUL BCP I-D in the IRTF/IETF - like in IPPM...

JL

Re: [Starlink] [EXTERNAL] Re: Researchers Seeking Probe Volunteers in USA

[Dave Taht]

On Mon, Jan 9, 2023 at 10:54 AM Livingood, Jason
<Jason_Livingood@comcast.com> wrote:
>
> > 0) None of the tests last long enough.
>
> The user-initiated ones tend to be shorter - likely because the average user does not want to wait several minutes for a test to complete. But IMO this is where a test platform like SamKnows, Ookla's embedded client, NetMicroscope, and others can come in - since they run in the background on some randomized schedule w/o user intervention. Thus, the user's time-sensitivity is no longer a factor and a longer duration test can be performed.

I would be so happy if someone independent ( and not necessarily me!)
was validating those more private tests, and retaining reference
packet captures of the various behaviors observed.

Bloat is just one problem among many, that shows up on a speedtest. I
have, for example, been working for months, on a very difficult
problem occurring on at least one wifi6 chipset... where the block-ack
window is being violated, leading to a ton of jitter under certain
loads, and a bandwidth reduction, that doesn't show up in summary
data.

Samknows published a really good blog recently,
here: https://samknows.com/blog/testing-principles

about how they are going about things... however...

>
> > 1) Not testing up + down + ping at the same time
>
> You should consider publishing a LUL BCP I-D in the IRTF/IETF - like in IPPM...

I have cc'd ippm a few times on these threads, and am on that mailing
list. It's pretty moribund compared to around here.

I am primarily interested in correct code (be it from a specification,
or not), and in looking at packet captures, to validate that it is
doing what it says on the tin, and moreover getting more stuff on
those tin, that I already know, should be tested for.

I agree that someday trying to nail down what latency under load
means, would be good to do. I'd settle at the moment, for single flow
simultaneous, tcp up, down, ping, and within stream latencies... all
plotted on the same chart.

> JL
>


-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC