[Starlink] The "reasons" that bufferbloat isn't a problem

Wed May 8 04:01:08 EDT 2024

Sorry - I meant ~ 4,5 % of the ISPs, not 2 :)

All the best,

Frank

Frantisek (Frank) Borsik

https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik at gmail.com

On Wed, May 8, 2024 at 9:58 AM Frantisek Borsik <frantisek.borsik at gmail.com>
wrote:

> Just to add the latest numbers from our (LibreQoS) ongoing "QoE
> competitive landscape research":
>
> Out of 66k plus ISPs worldwide, barely 3k use some QoE middle-box. Preseem
> is the market leader, with well over 400 T2 & T3 ISPs (number shared in
> their wonderful Fixed Wireless Network Report 2024 Q1 Edition
> <https://preseem.com/wp-content/uploads/2024/01/Preseem-Fixed-Wireless-Network-Report-2024-Q1.pdf>),
> Bequant seems to be 2nd, in terms of market penetration, claiming 500+ ISPs
> worldwide. Then we assume Cambium Networks and Paraqum - numbers of their
> users are not know, but we can expect something similar, in the Preseem and
> Bequant ballpark. Then there is LibreQoS. All in all, it's safe to say that
> somewhere between 2,5 and 3 thousand Internet Service Providers worldwide
> are using QoE middle-box of sort. So barely 2% of the ISPs worldwide are
> using it, we are still in the "innovator" stage of the whole "crossing the
> chasm" paradigm.
>
> We are all still very early on, working on it, and I'm lovin' it.
>
>
> All the best,
>
> Frank
>
> Frantisek (Frank) Borsik
>
>
>
> https://www.linkedin.com/in/frantisekborsik
>
> Signal, Telegram, WhatsApp: +421919416714
>
> iMessage, mobile: +420775230885
>
> Skype: casioa5302ca
>
> frantisek.borsik at gmail.com
>
>
> On Wed, May 8, 2024 at 4:16 AM Dave Taht via Starlink <
> starlink at lists.bufferbloat.net> wrote:
>
>> This was a wonderful post, rich!
>>
>> I note that preseem, paraqum,  bequant and libreqos (a bufferbloat.net
>> backed project) are in the fq codel or cake Middlebox for isps *Qoe) market
>> and all of us have made a substantial dent in the problem for oh, call it
>> 1000 isps worldwide total between us. Comcast also has done a pretty good
>> job but it seems yhe rest of the cable industry is asleep at the switch.
>>
>> The wisps totally got it with fq codel and cake arriving native for
>> mikeotiks entire product line and much of ubnts gear prior to that.
>>
>>
>> Qoe is still a pretty hard sell. Libreqos has a ton of free users and we
>> think over a million devices managed by it but not enough paid users to
>> justify even 1/10th the investment we have made so far into it (something
>> that I hope turns around with the upcoming v1.5 lts release and some
>> outputs from the nlnet and Comcast funded cakemaint and nqb projects)
>>
>> Thing is, at higher and fiber rates all the bloat moves to the wifi, and
>> a ton of that, like eero especially was long ago fq codeled and so I think
>> several major players have also (except for those stuck with broadcom).
>>
>> That said there are a lot of defective wifi aps left to replace. Nearly
>> every coffee shop I have been in with the exception of Starbucks has really
>> lousy wifi.
>>
>> I am so thrilled to see what starlink has accomplished so far with their
>> rollout of bufferbloat.net stuff and look forward to more. They are
>> still missing a few tricks... but are aware of what tricks they are
>> missing...
>>
>> Lack of knowledge of which regrettably remains the case for 97% of the
>> market and 99.99$ user base. Still ar apps will drive this rventually... I
>> think starlink is nicely positioned now to meet their demanding growth
>> goals and humanity's future in space assured, so there's that. ( i still
>> would rather like elone to send over a nice pair of tesla motors and
>> battery pack for my sailboat)
>>
>> I did have a nice jam with ajit Pai last week who is now well on his way
>> towards getting it. (See my twitter for the pics)
>>
>> On Mon, May 6, 2024, 4:25 AM Rich Brown via Starlink <
>> starlink at lists.bufferbloat.net> wrote:
>>
>>> Hi Gene,
>>>
>>> I've been vacillating on whether to send this note, but have decided to
>>> pull the trigger. I apologize in advance for the "Debbie Downer" nature of
>>> this message. I also apologize for any errors, omissions, or
>>> over-simplifications of the "birth of bufferbloat" story and its fixes.
>>> Corrections welcome.
>>>
>>> Rich
>>> ------
>>>
>>> If we are going to take a shot at opening people's eyes to bufferbloat,
>>> we should know some of the "objections" we'll run up against. Even though
>>> there's terrific technical data to back it up, people seem especially
>>> resistant to thinking that bufferbloat might affect their network, even
>>> when they're seeing problems that sound exactly like bufferbloat symptoms.
>>> But first, some history:
>>>
>>> The very idea of bufferbloat is simply unbelievable. Jim Gettys in 2011
>>> [1] couldn't believe it, and he's a smart networking guy,. At the time, it
>>> seemed incredible (that is "not credible" == impossible) that something
>>> could induce 1.2 seconds of latency into his home network connection. He
>>> called in favors from technical contacts at his ISP and at Bell Labs who
>>> went over everything with a fine-toothed comb. It was all exactly as
>>> spec'd. But he still had the latency.
>>>
>>> This led Jim and Dave Täht to start the investigation into the
>>> phenomenon known today as "bufferbloat" - the undesirable latency that
>>> comes from a router or other network equipment buffering too much data.
>>> Over several years, a group of smart people made huge improvements:
>>> fq_codel was released 14 May 2012 [3]; it was incorporated into the Linux
>>> kernel shortly afterward. CAKE came in 2015, and the fixes that minimize
>>> bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate [4] arrived to
>>> handle varying speed ISP links. All these techniques work great: in 2014,
>>> my 7mbps DSL link was quite usable. And when the pandemic hit, fq_codel on
>>> my OpenWrt router allowed me to use that same 7mbps DSL line for two
>>> simultaneous zoom calls.
>>>
>>> As one of the authors of [2], I am part of the team that has tried over
>>> the years to explain bufferbloat and how to fix it. We've spoken with
>>> vendors. We've spent untold hours responding to posts on assorted boards
>>> and forums with the the bufferbloat story.
>>>
>>> With these technical fixes in hand, we cockily set about to tell the
>>> world about how to fix bufferbloat. Our efforts have been met with
>>> skepticism at best, or stony silence. What are the objections?
>>>
>>> - This is just the ordinary behavior: I would expect things to be slower
>>> when there's more traffic (Willfully ignoring orders of magnitude increase
>>> in delay.)
>>> - Besides, I'm the only one using the internet. (Except when my phone
>>> uploads photos. Or my computer kicks off some automated process. Or I
>>> browse the web. Or ...)
>>> - It only happens some of the time. (Exactly. That's probably when
>>> something's uploading photos, or your computer is doing stuff in the
>>> background.)
>>> - Those bufferbloat tests you hear about are bogus. They artificially
>>> add load, which isn't a realistic test. (...and if you actually are
>>> downloading a file?)
>>> - Bufferbloat only happens when the network is 100% loaded. (True. But
>>> when you open a web page, your browser briefly uses 100% of the link. Is
>>> this enough to cause momentary lag?)
>>> - It's OK. I just tell my kids/spouse not to use the internet when I'm
>>> gaming. (Huh?)
>>> - I have gigabit service from my ISP. (That helps, but if you're
>>> complaining about "slowness" you still need to rule out bufferbloat in your
>>> router.)
>>> - I can't believe that router manufacturers would ever allow such a
>>> thing to happen in their gear. (See the Jim Gettys story above.)
>>> - I mean... wouldn't router vendors want to provide the best for their
>>> customers? (No - implementing this (new-ish) code requires engineering
>>> effort. They're selling plenty of routers with decade-old software. The
>>> Boss says, "would we sell more if they made these changes? Probably not.")
>>> - Why would my ISP provision/sell me a router that gave crappy service?
>>> They're a big company, they must know about this stuff. (Maybe. We have
>>> reached out to all the vendors. But remember they profit if you decide your
>>> network is too slow and you upgrade to a faster device/plan.)
>>> - But couldn't I just tweak the QoS on my router? (Maybe. But see [5])
>>> - Besides, I just spent $300 on a "gaming router". Obviously, I bought
>>> the most expensive/best possible solution on the market (But I still have
>>> lag...)
>>> - You're telling me that a bunch of pointy-headed academics are smarter
>>> than commercial router developers - who sold me that $300 router? (I can't
>>> believe it.)
>>> - And then you say that I should throw away that gaming router and
>>> install some "open source firmware"? (What the heck is that? And why should
>>> I believe you?)
>>> - What if it doesn't solve the problem? Who will give me support? And
>>> how will I get back to a vendor-supported system? (Valid point - the first
>>> valid point)
>>> - Aren't there any commercial solutions I can just buy? (Not at the
>>> moment. IQrouter was a shining light here - available from Amazon, simple
>>> setup, worked a treat - but they have gone out of business. And of course,
>>> for the skeptic, this is proof that the "fq_codel-stuff" isn't really a
>>> solution - it seems just like snake oil.)
>>>
>>> So... All these hurdles make it hard to convince people that bufferbloat
>>> could be the problem, or that they can fix for themselves.
>>>
>>> A couple of us have reached out to Consumer Reports, wondering if they
>>> would like a story about how vendors would prefer to sell you a new, faster
>>> router (or new faster ISP plan) than fix your bufferbloat. This kind of
>>> story seemed to be straight up their alley, but we never heard back after
>>> an initial contact. Maybe they deserve another call...
>>>
>>> The recent latency results from Starlink give me a modicum of hope.
>>> They're a major player. They (and their customers) can point to an order of
>>> magnitude reduction in latency over other solutions. It still requires
>>> enough "regular customers" to tell their current ISP that they are
>>> switching to Starlink (and spend $600 to purchase a Dishy plus $100/month)
>>> to provide a market incentive.
>>>
>>> Despite all this doom and gloom, I remain hopeful that things will get
>>> better. We know the technology exists for people to take control of their
>>> network and solve the problem for themselves. We can continue to respond on
>>> forums where people express their dismay at the crummy performance and
>>> suggest a solution. We can hope that a major vendor will twig to this
>>> effect and bring out a mass-market solution.
>>>
>>> I think your suggestion of speaking to eSports people is intriguing.
>>> They're highly motivated to make their personal networks better. And
>>> actually solving the problem would have a network effect of bringing in
>>> others with the same problem.
>>>
>>> Good luck, and thanks for thinking about this.
>>>
>>> Rich Brown
>>>
>>> [1]
>>> https://courses.cs.washington.edu/courses/cse550/21au/papers/bufferbloat.pdf
>>> [2]
>>> https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
>>> [3]
>>> https://lists.bufferbloat.net/pipermail/cerowrt-devel/2012-May/000233.html
>>> [4] https://github.com/lynxthecat/cake-autorate
>>> [5]
>>> https://www.bufferbloat.net/projects/bloat/wiki/More_about_Bufferbloat/#what-s-wrong-with-simply-configuring-qos
>>>
>>> On May 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink <
>>> starlink at lists.bufferbloat.net> wrote:
>>>
>>> Of course. For the gamers, the focus is managing latency. They have
>>> control of everything else.
>>>
>>> With our high latency and wide range of values, the eSports teams train
>>> on campus. It will be interesting to see how much improvements there can be
>>> for teams to be able to training from their homes.
>>>
>>> Gene
>>> ----------------------------------------------
>>> Eugene Chang
>>> IEEE Life Senior Member
>>> IEEE Communications Society & Signal Processing Society,
>>>     Hawaii Chapter Chair
>>> IEEE Life Member Affinity Group Hawaii Chair
>>> IEEE Entrepreneurship, Mentor
>>> eugene.chang at ieee.org
>>> m 781-799-0233 (in Honolulu)
>>>
>>>
>>> _______________________________________________
>>> Starlink mailing list
>>> Starlink at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/starlink
>>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20240508/d3ef65d5/attachment-0001.html>