Agreed - average jitter is useless, needs to be peak/maximum. Disallowing a very small number of exception points is ok. Simon On February 24, 2021 2:10:55 PM Dave Taht wrote: > So I've taken a tiny amount of time to run a few tests. For starters, > thank you very much > for your dedication and time into creating such a usable website, and faq. > > I have several issues though I really haven't had time to delve deep > into the packet captures. (others, please try taking em, and put them > somewhere?) > > 0) "average" jitter is a meaningless number. In the case of a > videoconferencing application, > what matters most is max jitter, where the app will choose to ride the > top edge of that, rather than follow it. I'd prefer using a 98% > number, rather than 75% number, to weight where the typical delay in a > videoconfernce might end up. > > 1) The worst case scenario of bloat affecting a users experience is > during a simultaneous up and download, and I'd rather you did that > rather than test them separately. Also you get > a more realistic figure for the actual achievable bandwidth under > contention and can expose problems like strict priority queuing in one > direction or another locking out further flows. > > 2) I get absurdly great results from it with or without sqm on on a > reasonably modern cablemodem (buffercontrol and pie and a cmts doing > the right things) > > This points to any of number of problems (features!) It's certainly my > hope that all the cdn makers at this point have installed bufferbloat > mitigations. Testing a cdn's tcp IS a great idea, but as a > bufferbloated test, maybe not so much. > > The packet capture of the tcp flows DOES show about 60ms jitter... but > no loss. Your test shows: > > https://www.waveform.com/tools/bufferbloat?test-id=6fc7dd95-8bfa-4b76-b141-ed423b6580a9 > > And is very jittery in the beginning of the test on its estimates. I > really should be overjoyed at knowing a cdn is doing more of the right > things, but in terms of a test... and linux also has got a ton of > mitigations on the client side. > > 3) As a side note, ecn actually is negotiated on the upload, if it's > enabled on your system. > Are you tracking an ecn statistics at this point (ecnseen)? It is not > negotiated on the download (which is fine by me). > > I regrettable at this precise moment am unable to test a native > cablemodem at the same speed as a sqm box, hope to get further on this > tomorrow. > > Again, GREAT work so far, and I do think a test tool for all these > cdns - heck, one that tested all of them at the same time, is very, > very useful. > > On Wed, Feb 24, 2021 at 10:22 AM Sina Khanifar wrote: >> >> Hi all, >> >> A couple of months ago my co-founder Sam posted an early beta of the >> Bufferbloat test that we’ve been working on, and Dave also linked to >> it a couple of weeks ago. >> >> Thank you all so much for your feedback - we almost entirely >> redesigned the tool and the UI based on the comments we received. >> We’re almost ready to launch the tool officially today at this URL, >> but wanted to show it to the list in case anyone finds any last bugs >> that we might have overlooked: >> >> https://www.waveform.com/tools/bufferbloat >> >> If you find a bug, please share the "Share Your Results" link with us >> along with what happened. We capture some debugging information on the >> backend, and having a share link allows us to diagnose any issues. >> >> This is really more of a passion project than anything else for us – >> we don’t anticipate we’ll try to commercialize it or anything like >> that. We're very thankful for all the work the folks on this list have >> done to identify and fix bufferbloat, and hope this is a useful >> contribution. I’ve personally been very frustrated by bufferbloat on a >> range of devices, and decided it might be helpful to build another >> bufferbloat test when the DSLReports test was down at some point last >> year. >> >> Our goals with this project were: >> * To build a second solid bufferbloat test in case DSLReports goes down again. >> * Build a test where bufferbloat is front and center as the primary >> purpose of the test, rather than just a feature. >> * Try to explain bufferbloat and its effect on a user's connection >> as clearly as possible for a lay audience. >> >> A few notes: >> * On the backend, we’re using Cloudflare’s CDN to perform the actual >> download and upload speed test. I know John Graham-Cunning has posted >> to this list in the past; if he or anyone from Cloudflare sees this, >> we’d love some help. Our Cloudflare Workers are being >> bandwidth-throttled due to having a non-enterprise grade account. >> We’ve worked around this in a kludgy way, but we’d love to get it >> resolved. >> * We have lots of ideas for improvements, e.g. simultaneous >> upload/downloads, trying different file size chunks, time-series >> latency graphs, using WebRTC to test UDP traffic etc, but in the >> interest of getting things launched we're sticking with the current >> featureset. >> * There are a lot of browser-specific workarounds that we had to >> implement, and latency itself is measured in different ways on >> Safari/Webkit vs Chromium/Firefox due to limitations of the >> PerformanceTiming APIs. You may notice that latency is different on >> different browsers, however the actual bufferbloat (relative increase >> in latency) should be pretty consistent. >> >> In terms of some of the changes we made based on the feedback we >> receive on this list: >> >> Based on Toke’s feedback: >> https://lists.bufferbloat.net/pipermail/bloat/2020-November/015960.html >> https://lists.bufferbloat.net/pipermail/bloat/2020-November/015976.html >> * We changed the way the speed tests run to show an instantaneous >> speed as the test is being run. >> * We moved the bufferbloat grade into the main results box. >> * We tried really hard to get as close to saturating gigabit >> connections as possible. We redesigned completely the way we chunk >> files, added a “warming up” period, and spent quite a bit optimizing >> our code to minimize CPU usage, as we found that was often the >> limiting factor to our speed test results. >> * We changed the shield grades altogether and went through a few >> different iterations of how to show the effect of bufferbloat on >> connectivity, and ended up with a “table view” to try to show the >> effect that bufferbloat specifically is having on the connection >> (compared to when the connection is unloaded). >> * We now link from the results table view to the FAQ where the >> conditions for each type of connection are explained. >> * We also changed the way we measure latency and now use the faster >> of either Google’s CDN or Cloudflare at any given location. We’re also >> using the WebTiming APIs to get a more accurate latency number, though >> this does not work on some mobile browsers (e.g. iOS Safari) and as a >> result we show a higher latency on mobile devices. Since our test is >> less a test of absolute latency and more a test of relative latency >> with and without load, we felt this was workable. >> * Our jitter is now an average (was previously RMS). >> * The “before you start” text was rewritten and moved above the start button. >> * We now spell out upload and download instead of having arrows. >> * We hugely reduced the number of cross-site scripts. I was a bit >> embarrassed by this if I’m honest - I spent a long time building web >> tools for the EFF, where we almost never allowed any cross-site >> scripts. * Our site is hosted on Shopify, and adding any features via >> their app store ends up adding a whole lot of gunk. But we uninstalled >> some apps, rewrote our template, and ended up removing a whole lot of >> the gunk. There’s still plenty of room for improvement, but it should >> be a lot better than before. >> >> Based on Dave Collier-Brown’s feedback: >> https://lists.bufferbloat.net/pipermail/bloat/2020-November/015966.html >> * We replaced the “unloaded” and “loaded” language with “unloaded” >> and then “download active” and “upload active.” In the grade box we >> indicate that, for example, “Your latency increased moderately under >> load.” >> * We tried to generally make it easier for non-techie folks to >> understand by emphasizing the grade and adding the table showing how >> bufferbloat affects some commonly-used services. >> * We didn’t really change the candle charts too much - they’re >> mostly just to give a basic visual - we focused more on the actual >> meat of the results above that. >> >> Based on Sebastian Moeller’s feedback: >> https://lists.bufferbloat.net/pipermail/bloat/2020-November/015963.html >> * We considered doing a bidirectional saturating load, but decided >> to skip on implementing it for now. * It’s definitely something we’d >> like to experiment with more in the future. >> * We added a “warming up” period as well as a “draining” period to >> help fill and empty the buffer. We haven’t added the option for an >> extended test, but have this on our list of backlog changes to make in >> the future. >> >> Based on Y’s feedback (link): >> https://lists.bufferbloat.net/pipermail/bloat/2020-November/015962.html >> * We actually ended up removing the grades, but we explained our >> criteria for the new table in the FAQ. >> >> Based on Greg White's feedback (shared privately): >> * We added an FAQ answer explaining jitter and how we measure it. >> >> We’d love for you all to play with the new version of the tool and >> send over any feedback you might have. We’re going to be in a feature >> freeze before launch but we'd love to get any bugs sorted out. We'll >> likely put this project aside after we iron out a last round of bugs >> and launch, and turn back to working on projects that help us pay the >> bills, but we definitely hope to revisit and improve the tool over >> time. >> >> Best, >> >> Sina, Arshan, and Sam. >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat > > > > -- > "For a successful technology, reality must take precedence over public > relations, for Mother Nature cannot be fooled" - Richard Feynman > > dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat