[LibreQoS] Fwd: project validating speedtest results and test tools

[LibreQoS] Fwd: project validating speedtest results and test tools

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 4000 bytes --]

---------- Forwarded message ---------
From: Elizabeth Belding <ebelding@cs.ucsb.edu>
Date: Thu, Jan 12, 2023 at 9:07 AM
Subject: Re: project validating speedtest results and test tools
To: Dave Taht <dave.taht@gmail.com>
Cc: National Broadband Mapping Coalition <bbcoalition@marconisociety.org>


Hi Dave,

Thanks for letting the group know about this really important work.

I am attaching a paper my group wrote, that recently received a Best
Paper Award at the ACM Internet Measurement Conference.  In it, we
examine Ookla and M-Lab speed test results, and make the case that in
order to be properly interpreted, context about the device type, last
hop link, and most importantly, the speed of the plan the user is
subscribed to must be included (and we introduce a nifty way to
reverse engineer subscribed plans for individual speed tests when
starting from bulk crowdsourced data points).  The set of graphs in
figure 13 specifically compares what Ookla measures vs. M-Lab, based
on the subscribed plan, and shows some important differences, in this
case for download speed.

I’d be happy to hear more about the specifics of what you are doing.

Elizabeth



___________________________________________________________

Elizabeth Belding
Professor, Dept. of Computer Science
Associate Dean - Diversity, Equity and Inclusion, College of Engineering
University of California, Santa Barbara
http://ebelding.cs.ucsb.edu

On Jan 12, 2023, at 8:10 AM, Dave Taht <dave.taht@gmail.com> wrote:

While I know the focus here is on the "maps", my concern has long been
about the quality of the "speedtest" data being reported. We've
started a project to validate the correctness of those results in
measuring up and download "speed", and built up a testbed that can
emulate various types of link technology.

If you have a favorite test, or test site, or (especially) anything
the FCC or NTIA is feeding into their pipeline, please let me know? A
virtualized set of web clients will go up soon, so we can use those.

Our testbed not formally organized into a reporting structure yet
(what would people like to see?), we (the geeks), at the moment, are
mostly just pounding data through various "plans" with all the tests
at our disposal to see if the results line up. To see a demo of what
the testbed can do, click on "run bandwidth test" off of this link,
and then click on a plan:

https://payne.taht.net/

(this is an obscure joke, the clients we are all emulating are called zergXX)

In this recent blog post, I took apart the new cloudflare and
speedtest.net tests:

https://blog.cerowrt.org/post/speedtests/

In that post I listed the other tests that we know how to test in that
environment. I would enjoy a chance to take apart all the speedtests,
from everyone out there - to see if it is doing any of the useful
metrics I outlined above, and reporting accurate statistics.

(as I write the quartile estimator is broken, and the tests we do
change from day to day - if you are interested, please join us in the
#libreqos:matrix.org channel on matrix.org - and the code is GPLv2ed!)

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

-- 
You received this message because you are subscribed to the Google
Groups "National Broadband Mapping Coalition" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to BBCoalition+unsubscribe@marconisociety.org.
To view this discussion on the web visit
https://groups.google.com/a/marconisociety.org/d/msgid/BBCoalition/CAA93jw5eMisj6a5yz4wvpECM_jXrtkUi1fc%3D14BuWAZkM3XBzg%40mail.gmail.com.




-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

[-- Attachment #2: belding_imc22.pdf --]
[-- Type: application/pdf, Size: 946040 bytes --]