Why not write an RFC on internet metrics? Then evangelize customers to rely on metrics compliant with the RFC. On Mon, Sep 26, 2022 at 5:36 PM Dave Taht wrote: > On Mon, Sep 26, 2022 at 2:45 PM Bruce Perens via Starlink > wrote: > > > > That's a good maxim: Don't believe a speed test that is hosted by your > own ISP. > > A network designed for speedtest.net, is a network... designed for > speedtest. Starlink seemingly was designed for speedtest - the 15 > second "cycle" to sense/change their bandwidth setting is just within > the 20s cycle speedtest terminates at, and speedtest returns the last > number for the bandwidth. It is a brutal test - using 8 or more flows > - much harder on the network than your typical web page load which, > while that is often 15 or so, most never run long enough to get out of > slow start. At least some of qualifying for the RDOF money was > achieving 100mbits down on "speedtest". > > A knowledgeable user concerned about web PLT should be looking a the > first 3 s of a given test, and even then once the bandwidth cracks > 20Mbit, it's of no help for most web traffic ( we've been citing mike > belshe's original work here a lot, > and more recent measurements still show that ) > > Speedtest also does nothing to measure how well a given > videoconference or voip session might go. There isn't a test (at least > not when last I looked) in the FCC broadband measurements for just > videoconferencing, and their latency under load test for many years > now, is buried deep in the annual report. > > I hope that with both ookla and samknows more publicly recording and > displaying latency under load (still, sigh, I think only displaying > the last number and only sampling every 250ms) that we can shift the > needle on this, but I started off this thread complaining nobody was > picking up on those numbers... and neither service tests the worst > case scenario of a simultaneous up/download, which was the principal > scenario we explored with the flent "rrul" series of tests, which were > originally designed to emulate and deeply understand what bittorrent > was doing to networks, and our principal tool in designing new fq and > aqm and transport CCs, along with the rtt_fair test for testing near > and far destinations at the same time. > > My model has always been a family of four, one person uploading, > another doing web, one doing videoconferencing, > and another doing voip or gaming, and no test anyone has emulates > that. With 16 wifi devices > per household, the rrul scenario is actually not "worst case", but > increasingly the state of things "normally". > > Another irony about speedtest is that users are inspired^Wtrained to > use it when the "network feels slow", and self-initiate something that > makes it worse, for both them and their portion of the network. > > Since the internet architecture board met last year, ( > https://www.iab.org/activities/workshops/network-quality/ ) there > seems to be an increasing amount of work on better metrics and tests > for QoE, with stuff like apple's responsiveness test, etc. > > I have a new one - prototyped in some starlink tests so far, and > elsewhere - called "SPOM" - steady packets over milliseconds, which, > when run simultaneously with capacity seeking traffic, might be a > better predictor of videoconferencing performance. > > There's also a really good "P99" conference coming up for those, that > like me, are OCD about a few sigmas. > > > > > On Mon, Sep 26, 2022 at 2:36 PM Eugene Y Chang via Starlink < > starlink@lists.bufferbloat.net> wrote: > >> > >> Thank you for the dialog,. > >> This discussion with regards to Starlink is interesting as it confirms > my guesses about the gap between Starlinks overly simplified, over > optimistic marketing and the reality as they acquire subscribers. > >> > >> I am actually interested in a more perverse issue. I am seeing latency > and bufferbloat as a consequence from significant under provisioning. It > doesn’t matter that the ISP is selling a fiber drop, if (parts) of their > network is under provisioned. Two end points can be less than 5 mile apart > and realize 120+ ms latency. Two Labor Days ago (a holiday) the max latency > was 230+ ms. The pattern I see suggest digital redlining. The older > communities appear to have much more severe under provisioning. > >> > >> Another observation. Running speedtest appears to go from the edge of > the network by layer 2 to the speedtest host operated by the ISP. Yup, > bypasses the (suspected overloaded) routers. > >> > >> Anyway, just observing. > >> > >> Gene > >> ---------------------------------------------- > >> Eugene Chang > >> IEEE Senior Life Member > >> eugene.chang@ieee.org > >> 781-799-0233 (in Honolulu) > >> > >> > >> > >> On Sep 26, 2022, at 11:20 AM, Sebastian Moeller > wrote: > >> > >> Hi Gene, > >> > >> > >> On Sep 26, 2022, at 23:10, Eugene Y Chang > wrote: > >> > >> Comments inline below. > >> > >> Gene > >> ---------------------------------------------- > >> Eugene Chang > >> IEEE Senior Life Member > >> eugene.chang@ieee.org > >> 781-799-0233 (in Honolulu) > >> > >> > >> > >> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller > wrote: > >> > >> Hi Eugene, > >> > >> > >> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink < > starlink@lists.bufferbloat.net> wrote: > >> > >> Ok, we are getting into the details. I agree. > >> > >> Every node in the path has to implement this to be effective. > >> > >> > >> Amazingly the biggest bang for the buck is gotten by fixing those nodes > that actually contain a network path's bottleneck. Often these are pretty > stable. So yes for fully guaranteed service quality all nodes would need to > participate, but for improving things noticeably it is sufficient to > improve the usual bottlenecks, e.g. for many internet access links the home > gateway is a decent point to implement better buffer management. (In short > the problem are over-sized and under-managed buffers, and one of the best > solution is better/smarter buffer management). > >> > >> > >> This is not completely true. > >> > >> > >> [SM] You are likely right, trying to summarize things leads to > partially incorrect generalizations. > >> > >> > >> Say the bottleneck is at node N. During the period of congestion, the > upstream node N-1 will have to buffer. When node N recovers, the > bufferbloat at N-1 will be blocking until the bufferbloat drains. Etc. > etc. Making node N better will reduce the extent of the backup at N-1, but > N-1 should implement the better code. > >> > >> > >> [SM] It is the node that builds up the queue that profits most from > better queue management.... (again I generalize, the node with the queue > itself probably does not care all that much, but the endpoints will profit > if the queue experiencing node deals with that queue more gracefully). > >> > >> > >> > >> > >> > >> In fact, every node in the path has to have the same prioritization or > the scheme becomes ineffective. > >> > >> > >> Yes and no, one of the clearest winners has been flow queueing, IMHO > not because it is the most optimal capacity sharing scheme, but because it > is the least pessimal scheme, allowing all (or none) flows forward > progress. You can interpret that as a scheme in which flows below their > capacity share are prioritized, but I am not sure that is the best way to > look at these things. > >> > >> > >> The hardest part is getting competing ISPs to implement and coordinate. > >> > >> > >> [SM] Yes, but it turned out even with non-cooperating ISPs there is a > lot end-users can do unilaterally on their side to improve both ingress and > egress congestion. Admittedly especially ingress congestion would be even > better handled with cooperation of the ISP. > >> > >> Bufferbloat and handoff between ISPs will be hard. The only way to fix > this is to get the unwashed public to care. Then they can say “we don’t > care about the technical issues, just fix it.” Until then ….. > >> > >> > >> [SM] Well we do this one home network at a time (not because that is > efficient or ideal, but simply because it is possible). Maybe, if you have > not done so already try OpenWrt with sqm-scripts (and maybe cake-autorate > in addition) on your home internet access link for say a week and let us > know ih/how your experience changed? > >> > >> Regards > >> Sebastian > >> > >> > >> > >> > >> > >> > >> Regards > >> Sebastian > >> > >> > >> > >> Gene > >> ---------------------------------------------- > >> Eugene Chang > >> IEEE Senior Life Member > >> eugene.chang@ieee.org > >> 781-799-0233 (in Honolulu) > >> > >> > >> > >> On Sep 26, 2022, at 10:48 AM, David Lang wrote: > >> > >> software updates can do far more than just improve recovery. > >> > >> In practice, large data transfers are less sensitive to latency than > smaller data transfers (i.e. downloading a CD image vs a video conference), > software can ensure better fairness in preventing a bulk transfer from > hurting the more latency sensitive transfers. > >> > >> (the example below is not completely accurate, but I think it gets the > point across) > >> > >> When buffers become excessivly large, you have the situation where a > video call is going to generate a small amount of data at a regular > interval, but a bulk data transfer is able to dump a huge amount of data > into the buffer instantly. > >> > >> If you just do FIFO, then you get a small chunk of video call, then > several seconds worth of CD transfer, followed by the next small chunk of > the video call. > >> > >> But the software can prevent the one app from hogging so much of the > connection and let the chunk of video call in sooner, avoiding the impact > to the real time traffic. Historically this has required the admin classify > all traffic and configure equipment to implement different treatment based > on the classification (and this requires trust in the classification > process), the bufferbloat team has developed options (fq_codel and cake) > that can ensure fairness between applications/servers with little or no > configuration, and no trust in other systems to properly classify their > traffic. > >> > >> The one thing that Cake needs to work really well is to be able to know > what the data rate available is. With Starlink, this changes frequently and > cake integrated into the starlink dish/router software would be far better > than anything that can be done externally as the rate changes can be fed > directly into the settings (currently they are only indirectly detected) > >> > >> David Lang > >> > >> > >> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: > >> > >> You already know this. Bufferbloat is a symptom and not the cause. > Bufferbloat grows when there are (1) periods of low or no bandwidth or (2) > periods of insufficient bandwidth (aka network congestion). > >> > >> If I understand this correctly, just a software update cannot make > bufferbloat go away. It might improve the speed of recovery (e.g. throw > away all time sensitive UDP messages). > >> > >> Gene > >> ---------------------------------------------- > >> Eugene Chang > >> IEEE Senior Life Member > >> eugene.chang@ieee.org > >> 781-799-0233 (in Honolulu) > >> > >> > >> > >> On Sep 26, 2022, at 10:04 AM, Bruce Perens wrote: > >> > >> Please help to explain. Here's a draft to start with: > >> > >> Starlink Performance Not Sufficient for Military Applications, Say > Scientists > >> > >> The problem is not availability: Starlink works where nothing but > another satellite network would. It's not bandwidth, although others have > questions about sustaining bandwidth as the customer base grows. It's > latency and jitter. As load increases, latency, the time it takes for a > packet to get through, increases more than it should. The scientists who > have fought bufferbloat, a major cause of latency on the internet, know > why. SpaceX needs to upgrade their system to use the scientist's Open > Source modifications to Linux to fight bufferbloat, and thus reduce > latency. This is mostly just using a newer version, but there are some > tunable parameters. Jitter is a change in the speed of getting a packet > through the network during a connection, which is inevitable in satellite > networks, but will be improved by making use of the bufferbloat-fighting > software, and probably with the addition of more satellites. > >> > >> We've done all of the work, SpaceX just needs to adopt it by upgrading > their software, said scientist Dave Taht. Jim Gettys, Taht's collaborator > and creator of the X Window System, chimed in: > >> Open Source luminary Bruce Perens said: sometimes Starlink's latency > and jitter make it inadequate to remote-control my ham radio station. But > the military is experimenting with remote-control of vehicles on the > battlefield and other applications that can be demonstrated, but won't > happen at scale without adoption of bufferbloat-fighting strategies. > >> > >> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang < > eugene.chang@alum.mit.edu> wrote: > >> The key issue is most people don’t understand why latency matters. They > don’t see it or feel it’s impact. > >> > >> First, we have to help people see the symptoms of latency and how it > impacts something they care about. > >> - gamers care but most people may think it is frivolous. > >> - musicians care but that is mostly for a hobby. > >> - business should care because of productivity but they don’t know how > to “see” the impact. > >> > >> Second, there needs to be a “OMG, I have been seeing the action of > latency all this time and never knew it! I was being shafted.” Once you > have this awakening, you can get all the press you want for free. > >> > >> Most of the time when business apps are developed, “we” hide the impact > of poor performance (aka latency) or they hide from the discussion because > the developers don’t have a way to fix the latency. Maybe businesses don’t > care because any employees affected are just considered poor performers. > (In bad economic times, the poor performers are just laid off.) For > employees, if they happen to be at a location with bad latency, they don’t > know that latency is hurting them. Unfair but most people don’t know the > issue is latency. > >> > >> Talking and explaining why latency is bad is not as effective as > showing why latency is bad. Showing has to be with something that has a > person impact. > >> > >> Gene > >> ----------------------------------- > >> Eugene Chang > >> eugene.chang@alum.mit.edu > >> +1-781-799-0233 (in Honolulu) > >> > >> > >> > >> > >> > >> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink < > starlink@lists.bufferbloat.net> > wrote: > >> > >> If you want to get attention, you can get it for free. I can place > articles with various press if there is something interesting to say. Did > this all through the evangelism of Open Source. All we need to do is write, > sign, and publish a statement. What they actually write is less relevant if > they publish a link to our statement. > >> > >> Right now I am concerned that the Starlink latency and jitter is going > to be a problem even for remote controlling my ham station. The US Military > is interested in doing much more, which they have demonstrated, but I don't > see happening at scale without some technical work on the network. Being > able to say this isn't ready for the government's application would be an > attention-getter. > >> > >> Thanks > >> > >> Bruce > >> > >> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink < > starlink@lists.bufferbloat.net> > wrote: > >> These days, if you want attention, you gotta buy it. A 50k half page > >> ad in the wapo or NYT riffing off of It's the latency, Stupid!", > >> signed by the kinds of luminaries we got for the fcc wifi fight, would > >> go a long way towards shifting the tide. > >> > >> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht dave.taht@gmail.com>> wrote: > >> > >> > >> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason > >> > > wrote: > >> > >> > >> The awareness & understanding of latency & impact on QoE is nearly > unknown among reporters. IMO maybe there should be some kind of background > briefings for reporters - maybe like a simple YouTube video explainer that > is short & high level & visual? Otherwise reporters will just continue to > focus on what they know... > >> > >> > >> That's a great idea. I have visions of crashing the washington > >> correspondents dinner, but perhaps > >> there is some set of gatherings journalists regularly attend? > >> > >> > >> On 9/21/22, 14:35, "Starlink on behalf of Dave Taht via Starlink" < > starlink-bounces@lists.bufferbloat.net starlink-bounces@lists.bufferbloat.net> on behalf of > starlink@lists.bufferbloat.net > > wrote: > >> > >> I still find it remarkable that reporters are still missing the > >> meaning of the huge latencies for starlink, under load. > >> > >> > >> > >> > >> -- > >> FQ World Domination pending: > https://blog.cerowrt.org/post/state_of_fq_codel/< > https://blog.cerowrt.org/post/state_of_fq_codel/> > >> Dave Täht CEO, TekLibre, LLC > >> > >> > >> > >> > >> -- > >> FQ World Domination pending: > https://blog.cerowrt.org/post/state_of_fq_codel/< > https://blog.cerowrt.org/post/state_of_fq_codel/> > >> Dave Täht CEO, TekLibre, LLC > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/starlink < > https://lists.bufferbloat.net/listinfo/starlink> > >> > >> > >> -- > >> Bruce Perens K6BP > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/starlink < > https://lists.bufferbloat.net/listinfo/starlink> > >> > >> > >> > >> > >> -- > >> Bruce Perens K6BP > >> > >> > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/starlink > >> > >> > >> _______________________________________________ > >> Starlink mailing list > >> Starlink@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/starlink > > > > > > > > -- > > Bruce Perens K6BP > > _______________________________________________ > > Starlink mailing list > > Starlink@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/starlink > > > > -- > FQ World Domination pending: > https://blog.cerowrt.org/post/state_of_fq_codel/ > Dave Täht CEO, TekLibre, LLC > -- Bruce Perens K6BP