[Rpm] Fwd: [M-Lab-Discuss] Download sppeds desreprencies when running Speed test

Fri Apr 15 09:06:06 EDT 2022

some choice comments on ndt's testing...

---------- Forwarded message ---------
From: Nick Feamster <feamster at gmail.com>
Date: Fri, Apr 15, 2022 at 5:23 AM
Subject: Re: [M-Lab-Discuss] Download sppeds desreprencies when
running Speed test
To: discuss <discuss at measurementlab.net>
Cc: etha... at gmail.com <ethanbkb at gmail.com>, pza... at logitech.com
<pzahra at logitech.com>, discuss <discuss at measurementlab.net>,
la... at measurementlab.net <laiyi at measurementlab.net>

A few thoughts:

1. All of these tests are subject to severe sampling bias, particular
under-representation of samples in communities where connectivity gaps
are most dire.  This I believe is a fundamental issue that is common
across *all* of today’s methodologies. Client-based tests face severe
sampling limitations; even the MBA program stratifies its sample by
ISP and thereby severely undersamples most geographies, making it not
very useful for many types of studies.  See our TPRC paper for some
initial discussion of these issues:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3786158

2. All of these tests are subject to various issues with client-based
measurements, from WiFi bottlenecks to limitations of the radios on
(old) client devices. A particular issue we discovered at one point
was that many NDT tests on higher speed links were being performed
from old iPhones whose radios did not support speeds of greater than
100 Mbps. In short, the NDT test was measuring the radio of the mobile
device, not the ISP. See our CACM paper for more discussion on that.
https://dl.acm.org/doi/pdf/10.1145/3372135

3. I am not sure it is accurate to say that NDT answers the question
of “how well is you pipe transferring data”. A more precise statement
would be that it measures the transfer rate across a single transport
connection—at one point a fairly outmoded version of TCP, and now a
single BBR stream.  And so it might represent how well the connection
transfers data across that transport, but it’s also worth pointing out
that no modern application that seeks to maximize capacity relies on a
single transport connection—everything from browsers to streaming
clients on smart TVs use multiple parallel connections (some do so in
rather extreme fashion, but that’s another topic).

4. Off-net measurements have their own caveats. One notable one—which
came to light in none other than an M-Lab report!—was that as a result
of an end-to-end path to an off-net server that crossed Cogent, the
report mis-diagnosed the location of congestion (and its resolution).
Cogent’s CEO also acknowledged this particular issue (read my FCC
filing on the issue for more about that:
https://www.fcc.gov/ecfs/file/download/DOC-578d040705800000-A.pdf).

5. To the point about the Google search “one box”—characterizing the
loss of NDT measurements as a “great loss to the research community”
presumes that “more data is better” and “open data is
better"—independent of the *quality* of those measurements.  But, I’d
argue we need to rethink that. As Jim and Jason have both pointed out,
the data has been actively misused and mischaracterized over many
years—and this continues to happen. Part of the reason I believe this
to be the case is the reason Lai Yi mentions: there are “so few
alternatives”.  It’s time to change that because regardless of the
facts, people will take the path of least resistance. Casting it in
the most charitable light—while some stakeholders may be willfully
misrepresenting the data, I suspect others simply don’t have the time
or interest in understanding the nuance that this group is familiar
with. Taking shortcuts like that is sloppy and unscientific—something
that has bothered me for quite some time—but I think some groups have
gotten away with it because there’s little denying that there *are*
gaps in infrastructure, connectivity, etc.—nevermind that the data
itself doesn’t actually tell you much about the specific nature of any
particular problem, people are happy to talk in broad strokes and
handwave if the data conveniently speaks to an agenda. But now, as we
think about actually *solving* problems, this becomes a more serious
problem—the existing tools and methods—measurement techniques, data,
sampling approaches—do very little in helping anyone actually work
towards fixing these problems.  I believe that’s where we should be
turning our focus in the next 5-10 years.

On Thursday, April 14, 2022 at 7:26:28 PM UTC-5 etha... at gmail.com wrote:
>
> I think this is a great summary of key differences!
>
> However, as an additional caveat in interpreting the data, it's not clear to me what off-net vs on-net measurements meaningfully represent in terms of "the complete path across the Internet from user to content." For many (probably most) Internet users, much (probably most) of their traffic comes either from an on-net Content Delivery Network (CDN) node or across a dedicated network interconnection to the content provider. So it's not clear to me that traffic crossing one or a few particular network interconnections between me and the assigned MLab server is a more meaningful measure of the performance I'd get to content than a measurement to an on-net server -- I suspect that neither exercises any interdomain interconnections that my traffic encounters when using YouTube, Netflix, Facebook, services hosted on the major cloud providers, services hosted on a number of widely-used CDNs, etc. If the NDT measurement is constrained by issues at the interconnection, the issues could apply to any services I use that happen to share that interconnection, but that's probably not the case for most services I use, and it requires a good amount of Internet measurement to understand which ones do share it.
>
> Ethan
>
>
> On Wed, Apr 13, 2022 at 6:15 PM Lai Yi Ohlsen <la... at measurementlab.net> wrote:
>>
>> Hi Paul,
>>
>> Thanks for your question. The simplest way to answer, is that M-Lab’s NDT and Ookla’s SpeedTest measure fundamentally different things. They differ in three key ways: 1. M-Lab measures to servers in off-net locations, Ookla measures to on-net locations. 2. NDT measures bulk transport capacity, SpeedTest attempts to measure link capacity and 3. NDT is a single stream test, SpeedTest is a multi-stream  Here is a more in depth explanation of each characteristic:
>>
>> 1. Off-net vs. on-net measurement: All of M-Lab’s measurement services, including NDT, are hosted on our off-net platform. “On-net” refers to measurements performed on the same network as the network it is measuring, such as an Internet Service Provider (ISP) measuring itself. It only captures one segment of any path that data is likely to be traversing. In contrast, “off-net” measurements extend beyond a user’s access provider’s network to measure the complete path across the Internet from user to content including interconnections. By definition, on-net measurement can not even detect the effects of any performance limitations at interconnects between ISPs. All of the measurement services hosted by M-Lab inherit the off-net platform methodology for nearly all users.
>> 2. Link capacity vs. bulk transport: When using NDT tests specifically, Internet users are sometimes confused when their individual results don’t confirm the speeds promised by their Internet service provider. “Speed” is often associated with “link capacity,” which is the maximum bitrate of a link; in other words, the best performance possible. However, NDT measures “bulk transport capacity” — the rate that TCP can deliver data across the end-to-end path; in other words, the reliability of that connection. It is important to note that many link problems (such as low level packet loss and reordering) typically adversely impact both MLab measurements and real application performance.  These two ways of measuring performance, link capacity and bulk transport capacity, are different and are often conflated when both concepts are referred to as “Internet speed.” When using NDT data to discuss speed, it is important to clarify these terms to have more effective conversations about Internet speed.
>> 3. Single-stream vs. multi-stream tests: NDT measures the single-stream performance of bulk transport capacity. While modern web browsers will use multiple streams of data, testing for multiple streams can compensate for data delivery problems that are exposed by a single stream. A multi-stream test can return measurements closer to link capacity but it would not represent the adverse performance impact of low-level packet loss. By testing for single-stream performance, NDT is an effective baseline for measuring a user’s Internet performance.
>>
>> In short, NDT does not answer the question “how big is your pipe” but rather “how well is your pipe transferring data.” We consider both metrics to be important for understanding connectivity. For more reading, you can reference How fast is my Internet? Speed Tests, Accuracy, NDT & M-Lab which digs more into the definition of “speed” and NDT Data in NTIA Indicators of Broadband Need, which though focuses on the US federal mapping resource, helps describe the difference between the aggregated datasets for the purpose of research.
>>
>> Additionally, any given test result is impacted by a number of factors, including the network management and topology of the path, the traffic on the network at the time of the test, and other environmental factors. One of the reasons M-Lab finds it so important to publish this data openly, is so we can study these factors in a more statistically rigorous way as part of a sample vs. in the one-off context of individual tests.
>>
>> I hope this answer addresses your question, but please let me know if I can explain anything further.
>>
>>
>> On Wed, Apr 13, 2022 at 12:38 PM Paul Zahra <pza... at logitech.com> wrote:
>>>
>>> I noticed when I run your speed test at times the dpwnload speeds are much, much lower then when using OOkla,   This moing test showed  sub 1 MB , but OOKLA showed at least 40MB  .Can you explain the possible differences?   My connection does seem slow.
>>>
>>> Thaks for any insight you can provide,
>>>
>>> Regards,
>>> Paul Zahra
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u... at measurementlab.net.
>>> To view this discussion on the web visit https://groups.google.com/a/measurementlab.net/d/msgid/discuss/a609ce0e-7be6-4b4e-a231-227d7cd4106bn%40measurementlab.net.
>>
>>
>>
>> --
>> Lai Yi Ohlsen
>> Director, Measurement Lab
>> Code for Science & Society
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u... at measurementlab.net.
>>
>> To view this discussion on the web visit https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAD3WrcMg5JOuFWy4Fd6G4CKnE3CQN0m0nWXCbCGoZg_S1vcuxw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google
Groups "discuss" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to discuss+unsubscribe at measurementlab.net.
To view this discussion on the web visit
https://groups.google.com/a/measurementlab.net/d/msgid/discuss/843ec51a-bc85-48ad-9a24-7ba56c952197n%40measurementlab.net.

-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC