---------- Forwarded message --------- From: Elizabeth Belding Date: Thu, Jan 12, 2023 at 9:07 AM Subject: Re: project validating speedtest results and test tools To: Dave Taht Cc: National Broadband Mapping Coalition Hi Dave, Thanks for letting the group know about this really important work. I am attaching a paper my group wrote, that recently received a Best Paper Award at the ACM Internet Measurement Conference. In it, we examine Ookla and M-Lab speed test results, and make the case that in order to be properly interpreted, context about the device type, last hop link, and most importantly, the speed of the plan the user is subscribed to must be included (and we introduce a nifty way to reverse engineer subscribed plans for individual speed tests when starting from bulk crowdsourced data points). The set of graphs in figure 13 specifically compares what Ookla measures vs. M-Lab, based on the subscribed plan, and shows some important differences, in this case for download speed. I’d be happy to hear more about the specifics of what you are doing. Elizabeth ___________________________________________________________ Elizabeth Belding Professor, Dept. of Computer Science Associate Dean - Diversity, Equity and Inclusion, College of Engineering University of California, Santa Barbara http://ebelding.cs.ucsb.edu On Jan 12, 2023, at 8:10 AM, Dave Taht wrote: While I know the focus here is on the "maps", my concern has long been about the quality of the "speedtest" data being reported. We've started a project to validate the correctness of those results in measuring up and download "speed", and built up a testbed that can emulate various types of link technology. If you have a favorite test, or test site, or (especially) anything the FCC or NTIA is feeding into their pipeline, please let me know? A virtualized set of web clients will go up soon, so we can use those. Our testbed not formally organized into a reporting structure yet (what would people like to see?), we (the geeks), at the moment, are mostly just pounding data through various "plans" with all the tests at our disposal to see if the results line up. To see a demo of what the testbed can do, click on "run bandwidth test" off of this link, and then click on a plan: https://payne.taht.net/ (this is an obscure joke, the clients we are all emulating are called zergXX) In this recent blog post, I took apart the new cloudflare and speedtest.net tests: https://blog.cerowrt.org/post/speedtests/ In that post I listed the other tests that we know how to test in that environment. I would enjoy a chance to take apart all the speedtests, from everyone out there - to see if it is doing any of the useful metrics I outlined above, and reporting accurate statistics. (as I write the quartile estimator is broken, and the tests we do change from day to day - if you are interested, please join us in the #libreqos:matrix.org channel on matrix.org - and the code is GPLv2ed!) -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC -- You received this message because you are subscribed to the Google Groups "National Broadband Mapping Coalition" group. To unsubscribe from this group and stop receiving emails from it, send an email to BBCoalition+unsubscribe@marconisociety.org. To view this discussion on the web visit https://groups.google.com/a/marconisociety.org/d/msgid/BBCoalition/CAA93jw5eMisj6a5yz4wvpECM_jXrtkUi1fc%3D14BuWAZkM3XBzg%40mail.gmail.com. -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC