From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B01413B29E for ; Fri, 15 Apr 2022 09:06:19 -0400 (EDT) Received: by mail-ed1-x52a.google.com with SMTP id z12so9890191edl.2 for ; Fri, 15 Apr 2022 06:06:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=tF6m/AVbixAzb3kVJ9GX7Ecr2o62azmLhfuvY3kOT0Q=; b=Z+5rTvv4Y8XXYXTzpXEobMq4IXWerfap9gpW1aVlj+kaiwHflA97p7Del+hm6J/rWY vtV0F9jDC0TDb/fmnQ/WnjVqZ6ngZ3ioidXsGexwSFwYEFn72b9oykpE5/N33PMSBgAS f9kka3kw0yNpwJ2leoNJoxWULpO+WgxfOiZlKwCMr6O5XgtM1ugpuzRgpWIobRXJ7pVb mvH6HjRsG5Ymw6dtmn9wQ6dKfG/Bf9iSgU16bs9sP+1wvM9sz7/vtZtJLvUjxXonViP8 ZtriCM7MflgfIKBhX0OeFwjnaN8CoOGp4X94ic9SwW+4C3mMLPnK5H3BByn4RN0L28IL I49Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=tF6m/AVbixAzb3kVJ9GX7Ecr2o62azmLhfuvY3kOT0Q=; b=Cw+03CNg6vRrl5/tPf5r2BmcD81QS49g41UGZUGuClJdGVvb0pUfqt3i6JrTtkYhAc BrmIT68t0SSSZTd/uXWHEE1kz3tSPdaLDpy4ayY+xrcgRCmav+bEhOtHx5Sv6t0CB9nw cNBFlXiSuN0FiCcFQzWQ6jADk+Od2OfFc19xwD/dmZLmm/8H+s78HLzUzn86rVsg3u33 3ujUhIHqtLPCPWEhPlEKeDqiLns3Az0fsLN4JGoNJSFSo66PmOPgDTx/SNuwt218yWd0 rBI+RlM74ZGQGo9lBINBGyHTS7TFBOrkSDG71zcsAGGtG1jjRjN8hhcVX+k4tDkxFCj2 K9GQ== X-Gm-Message-State: AOAM532KVMa2pKnS0dorPZPa76I2EidxECfDjvnTcv3TNQuzPAH5eoiX SJRy1edpGLQx6AOy8hXcs01bEclHRwHY2hunBFrDX5mLk4k= X-Google-Smtp-Source: ABdhPJxP4V2AD8Eh5kyxyVXZVijhNxed4gzPSkFY/foHuHx/LhK0TezMN+hGvAU2jj54Nvh2RYd+4i0aapr5jl4aX6g= X-Received: by 2002:aa7:cac5:0:b0:41c:c1fb:f5cc with SMTP id l5-20020aa7cac5000000b0041cc1fbf5ccmr7978975edt.219.1650027978228; Fri, 15 Apr 2022 06:06:18 -0700 (PDT) MIME-Version: 1.0 References: <843ec51a-bc85-48ad-9a24-7ba56c952197n@measurementlab.net> In-Reply-To: <843ec51a-bc85-48ad-9a24-7ba56c952197n@measurementlab.net> From: Dave Taht Date: Fri, 15 Apr 2022 06:06:06 -0700 Message-ID: To: Rpm Cc: Nick Feamster Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [Rpm] Fwd: [M-Lab-Discuss] Download sppeds desreprencies when running Speed test X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Apr 2022 13:06:19 -0000 some choice comments on ndt's testing... ---------- Forwarded message --------- From: Nick Feamster Date: Fri, Apr 15, 2022 at 5:23 AM Subject: Re: [M-Lab-Discuss] Download sppeds desreprencies when running Speed test To: discuss Cc: etha...@gmail.com , pza...@logitech.com , discuss , la...@measurementlab.net A few thoughts: 1. All of these tests are subject to severe sampling bias, particular under-representation of samples in communities where connectivity gaps are most dire. This I believe is a fundamental issue that is common across *all* of today=E2=80=99s methodologies. Client-based tests face seve= re sampling limitations; even the MBA program stratifies its sample by ISP and thereby severely undersamples most geographies, making it not very useful for many types of studies. See our TPRC paper for some initial discussion of these issues: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3D3786158 2. All of these tests are subject to various issues with client-based measurements, from WiFi bottlenecks to limitations of the radios on (old) client devices. A particular issue we discovered at one point was that many NDT tests on higher speed links were being performed from old iPhones whose radios did not support speeds of greater than 100 Mbps. In short, the NDT test was measuring the radio of the mobile device, not the ISP. See our CACM paper for more discussion on that. https://dl.acm.org/doi/pdf/10.1145/3372135 3. I am not sure it is accurate to say that NDT answers the question of =E2=80=9Chow well is you pipe transferring data=E2=80=9D. A more precise= statement would be that it measures the transfer rate across a single transport connection=E2=80=94at one point a fairly outmoded version of TCP, and now a single BBR stream. And so it might represent how well the connection transfers data across that transport, but it=E2=80=99s also worth pointing = out that no modern application that seeks to maximize capacity relies on a single transport connection=E2=80=94everything from browsers to streaming clients on smart TVs use multiple parallel connections (some do so in rather extreme fashion, but that=E2=80=99s another topic). 4. Off-net measurements have their own caveats. One notable one=E2=80=94whi= ch came to light in none other than an M-Lab report!=E2=80=94was that as a res= ult of an end-to-end path to an off-net server that crossed Cogent, the report mis-diagnosed the location of congestion (and its resolution). Cogent=E2=80=99s CEO also acknowledged this particular issue (read my FCC filing on the issue for more about that: https://www.fcc.gov/ecfs/file/download/DOC-578d040705800000-A.pdf). 5. To the point about the Google search =E2=80=9Cone box=E2=80=9D=E2=80=94c= haracterizing the loss of NDT measurements as a =E2=80=9Cgreat loss to the research community= =E2=80=9D presumes that =E2=80=9Cmore data is better=E2=80=9D and =E2=80=9Copen data = is better"=E2=80=94independent of the *quality* of those measurements. But, I= =E2=80=99d argue we need to rethink that. As Jim and Jason have both pointed out, the data has been actively misused and mischaracterized over many years=E2=80=94and this continues to happen. Part of the reason I believe th= is to be the case is the reason Lai Yi mentions: there are =E2=80=9Cso few alternatives=E2=80=9D. It=E2=80=99s time to change that because regardless= of the facts, people will take the path of least resistance. Casting it in the most charitable light=E2=80=94while some stakeholders may be willfully misrepresenting the data, I suspect others simply don=E2=80=99t have the ti= me or interest in understanding the nuance that this group is familiar with. Taking shortcuts like that is sloppy and unscientific=E2=80=94somethi= ng that has bothered me for quite some time=E2=80=94but I think some groups ha= ve gotten away with it because there=E2=80=99s little denying that there *are* gaps in infrastructure, connectivity, etc.=E2=80=94nevermind that the data itself doesn=E2=80=99t actually tell you much about the specific nature of = any particular problem, people are happy to talk in broad strokes and handwave if the data conveniently speaks to an agenda. But now, as we think about actually *solving* problems, this becomes a more serious problem=E2=80=94the existing tools and methods=E2=80=94measurement techniqu= es, data, sampling approaches=E2=80=94do very little in helping anyone actually work towards fixing these problems. I believe that=E2=80=99s where we should be turning our focus in the next 5-10 years. On Thursday, April 14, 2022 at 7:26:28 PM UTC-5 etha...@gmail.com wrote: > > I think this is a great summary of key differences! > > However, as an additional caveat in interpreting the data, it's not clear= to me what off-net vs on-net measurements meaningfully represent in terms = of "the complete path across the Internet from user to content." For many (= probably most) Internet users, much (probably most) of their traffic comes = either from an on-net Content Delivery Network (CDN) node or across a dedic= ated network interconnection to the content provider. So it's not clear to = me that traffic crossing one or a few particular network interconnections b= etween me and the assigned MLab server is a more meaningful measure of the = performance I'd get to content than a measurement to an on-net server -- I = suspect that neither exercises any interdomain interconnections that my tra= ffic encounters when using YouTube, Netflix, Facebook, services hosted on t= he major cloud providers, services hosted on a number of widely-used CDNs, = etc. If the NDT measurement is constrained by issues at the interconnection= , the issues could apply to any services I use that happen to share that in= terconnection, but that's probably not the case for most services I use, an= d it requires a good amount of Internet measurement to understand which one= s do share it. > > Ethan > > > On Wed, Apr 13, 2022 at 6:15 PM Lai Yi Ohlsen = wrote: >> >> Hi Paul, >> >> Thanks for your question. The simplest way to answer, is that M-Lab=E2= =80=99s NDT and Ookla=E2=80=99s SpeedTest measure fundamentally different t= hings. They differ in three key ways: 1. M-Lab measures to servers in off-n= et locations, Ookla measures to on-net locations. 2. NDT measures bulk tran= sport capacity, SpeedTest attempts to measure link capacity and 3. NDT is a= single stream test, SpeedTest is a multi-stream Here is a more in depth e= xplanation of each characteristic: >> >> 1. Off-net vs. on-net measurement: All of M-Lab=E2=80=99s measurement se= rvices, including NDT, are hosted on our off-net platform. =E2=80=9COn-net= =E2=80=9D refers to measurements performed on the same network as the netwo= rk it is measuring, such as an Internet Service Provider (ISP) measuring it= self. It only captures one segment of any path that data is likely to be tr= aversing. In contrast, =E2=80=9Coff-net=E2=80=9D measurements extend beyond= a user=E2=80=99s access provider=E2=80=99s network to measure the complete= path across the Internet from user to content including interconnections. = By definition, on-net measurement can not even detect the effects of any pe= rformance limitations at interconnects between ISPs. All of the measurement= services hosted by M-Lab inherit the off-net platform methodology for near= ly all users. >> 2. Link capacity vs. bulk transport: When using NDT tests specifically, = Internet users are sometimes confused when their individual results don=E2= =80=99t confirm the speeds promised by their Internet service provider. =E2= =80=9CSpeed=E2=80=9D is often associated with =E2=80=9Clink capacity,=E2=80= =9D which is the maximum bitrate of a link; in other words, the best perfor= mance possible. However, NDT measures =E2=80=9Cbulk transport capacity=E2= =80=9D =E2=80=94 the rate that TCP can deliver data across the end-to-end p= ath; in other words, the reliability of that connection. It is important to= note that many link problems (such as low level packet loss and reordering= ) typically adversely impact both MLab measurements and real application pe= rformance. These two ways of measuring performance, link capacity and bulk= transport capacity, are different and are often conflated when both concep= ts are referred to as =E2=80=9CInternet speed.=E2=80=9D When using NDT data= to discuss speed, it is important to clarify these terms to have more effe= ctive conversations about Internet speed. >> 3. Single-stream vs. multi-stream tests: NDT measures the single-stream = performance of bulk transport capacity. While modern web browsers will use = multiple streams of data, testing for multiple streams can compensate for d= ata delivery problems that are exposed by a single stream. A multi-stream t= est can return measurements closer to link capacity but it would not repres= ent the adverse performance impact of low-level packet loss. By testing for= single-stream performance, NDT is an effective baseline for measuring a us= er=E2=80=99s Internet performance. >> >> In short, NDT does not answer the question =E2=80=9Chow big is your pipe= =E2=80=9D but rather =E2=80=9Chow well is your pipe transferring data.=E2= =80=9D We consider both metrics to be important for understanding connectiv= ity. For more reading, you can reference How fast is my Internet? Speed Tes= ts, Accuracy, NDT & M-Lab which digs more into the definition of =E2=80=9Cs= peed=E2=80=9D and NDT Data in NTIA Indicators of Broadband Need, which thou= gh focuses on the US federal mapping resource, helps describe the differenc= e between the aggregated datasets for the purpose of research. >> >> Additionally, any given test result is impacted by a number of factors, = including the network management and topology of the path, the traffic on t= he network at the time of the test, and other environmental factors. One of= the reasons M-Lab finds it so important to publish this data openly, is so= we can study these factors in a more statistically rigorous way as part of= a sample vs. in the one-off context of individual tests. >> >> I hope this answer addresses your question, but please let me know if I = can explain anything further. >> >> >> On Wed, Apr 13, 2022 at 12:38 PM Paul Zahra wrote: >>> >>> I noticed when I run your speed test at times the dpwnload speeds are m= uch, much lower then when using OOkla, This moing test showed sub 1 MB ,= but OOKLA showed at least 40MB .Can you explain the possible differences?= My connection does seem slow. >>> >>> Thaks for any insight you can provide, >>> >>> Regards, >>> Paul Zahra >>> >>> -- >>> You received this message because you are subscribed to the Google Grou= ps "discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send = an email to discuss+u...@measurementlab.net. >>> To view this discussion on the web visit https://groups.google.com/a/me= asurementlab.net/d/msgid/discuss/a609ce0e-7be6-4b4e-a231-227d7cd4106bn%40me= asurementlab.net. >> >> >> >> -- >> Lai Yi Ohlsen >> Director, Measurement Lab >> Code for Science & Society >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Group= s "discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send a= n email to discuss+u...@measurementlab.net. >> >> To view this discussion on the web visit https://groups.google.com/a/mea= surementlab.net/d/msgid/discuss/CAD3WrcMg5JOuFWy4Fd6G4CKnE3CQN0m0nWXCbCGoZg= _S1vcuxw%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@measurementlab.net. To view this discussion on the web visit https://groups.google.com/a/measurementlab.net/d/msgid/discuss/843ec51a-bc8= 5-48ad-9a24-7ba56c952197n%40measurementlab.net. --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC