From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D8C5B3B29D for ; Tue, 7 May 2024 22:16:29 -0400 (EDT) Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-4196c62bb4eso28967195e9.2 for ; Tue, 07 May 2024 19:16:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715134588; x=1715739388; darn=lists.bufferbloat.net; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0AMAvnLGbspOmqBcb60ERucdhQMtm/KzvE4UMEnZZEo=; b=JJROd2/VCst/vtDbD+5JfP2bAtpGLyXcLaqyg7xYGMp/v9WEpK+6+Xp+QVkMi4mt77 sheApBprdd0SkgBOCOczltBp4ggnd4TMtB7fR+fjyA6Nr7yMdGvZnH85QpV7kRSZYnJW EGzYmVz+Az4apN/f8+RoVWf7kyCKo05VBKX1ZAzCC2k9iWszdsY2aNlyHP/Kwyw+ikq5 k6O2KITXoQ0YI0E/fvYRduTSHqFDZNBl5W8XeNkWBxVslBI2glZiAOEiIBmzLjRmQvai 3V7fvW8Is6sQ6g99uzISp/3rIZGVGN+oBbbMBGePfSamnCamK0KXJa7bxPp7JOMzsva6 9Bbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715134588; x=1715739388; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0AMAvnLGbspOmqBcb60ERucdhQMtm/KzvE4UMEnZZEo=; b=qGDTixyXrToAEYibTRgMCJf5gqy02o7wFVhlza6UCYLqUAhiU3gOMICWuUkSq1m2Bg M7a0f4o578rg7BrNg0gpQFvTfqZUKQjzz8b9dnzCfy0HRG1XmCKxzRIqFMKBPGc6nLyF +zAmpBO09I2m96FS4guqJ7wSwKhQHfSUixsq3uG30s4jafdDRnhfvSxSRUt6QE/6uoqQ benedLBWKg6hnONSO3RRcHhHDf5fFGRburWQIqsK7rqXlgVRX1nGG1w3JJkvbbLNHpfZ 9Mw9rlJkQqxHMOoOJD5ajvh+vsYbeAq5t/Q3PtZVrwOZEV/gzzXqf3TRQD2psTVO21My IXgQ== X-Forwarded-Encrypted: i=1; AJvYcCWKEPrWbB+noGsLZANvJPyhb4uSx2ZKoFCeaQnqZlcUP2vCZVYv3dpGLkHXoDjzOgf6tJUK6XftjN1STudwfgz8KCAeosV5Ij3zVO5YEv8= X-Gm-Message-State: AOJu0YxZ1Z2lshMYgugnEsduwS9/NHmiSL9k9dW5Df30vI0yAffTby3S bnHqP6K6WmSLYdee8XNdGZae6GWR+/ztEL1ysNnSbgYRtKxbPze1y0xuaIkywTeth7JBXW5rAFD B4nbO1l3tqZUqxWtocJpzhYBBu7M= X-Google-Smtp-Source: AGHT+IHioY+rif41G+TtKf9dGcIKePFxK1nuCvKQBoYGRFK7BRcatygCx3O+dNMUEAwpk/AppdvXRnFuKjiQ/4qtdQc= X-Received: by 2002:a05:600c:511a:b0:41b:b77c:679 with SMTP id 5b1f17b1804b1-41f71accb88mr11625865e9.37.1715134588368; Tue, 07 May 2024 19:16:28 -0700 (PDT) MIME-Version: 1.0 References: <438B1BC4-D465-497A-B6BA-700E1D411036@ieee.org> <79C02ABB-B2A6-4B4D-98F4-6540D3F96EBB@ieee.org> <7E918B58-382A-4793-A144-13A7075CA56C@connectivitycap.com> <13rq2389-9012-p95n-s494-q3pp070s497n@ynat.uz> <6qop2p3o-351p-788q-q1q2-86sosnq3rn21@ynat.uz> <3FF32F52-4A93-496B-85FF-00020FA4A48B@gmx.de> <08F6942E-CC08-4956-B92E-CBEC091D86E4@ieee.org> In-Reply-To: From: Dave Taht Date: Tue, 7 May 2024 18:48:50 -0700 Message-ID: To: Rich Brown Cc: Sebastian Moeller , Colin_Higbie , Dave Taht via Starlink , Eugene Y Chang Content-Type: multipart/alternative; boundary="000000000000e1c06a0617e7e331" Subject: Re: [Starlink] The "reasons" that bufferbloat isn't a problem X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2024 02:16:30 -0000 --000000000000e1c06a0617e7e331 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable This was a wonderful post, rich! I note that preseem, paraqum, bequant and libreqos (a bufferbloat.net backed project) are in the fq codel or cake Middlebox for isps *Qoe) market and all of us have made a substantial dent in the problem for oh, call it 1000 isps worldwide total between us. Comcast also has done a pretty good job but it seems yhe rest of the cable industry is asleep at the switch. The wisps totally got it with fq codel and cake arriving native for mikeotiks entire product line and much of ubnts gear prior to that. Qoe is still a pretty hard sell. Libreqos has a ton of free users and we think over a million devices managed by it but not enough paid users to justify even 1/10th the investment we have made so far into it (something that I hope turns around with the upcoming v1.5 lts release and some outputs from the nlnet and Comcast funded cakemaint and nqb projects) Thing is, at higher and fiber rates all the bloat moves to the wifi, and a ton of that, like eero especially was long ago fq codeled and so I think several major players have also (except for those stuck with broadcom). That said there are a lot of defective wifi aps left to replace. Nearly every coffee shop I have been in with the exception of Starbucks has really lousy wifi. I am so thrilled to see what starlink has accomplished so far with their rollout of bufferbloat.net stuff and look forward to more. They are still missing a few tricks... but are aware of what tricks they are missing... Lack of knowledge of which regrettably remains the case for 97% of the market and 99.99$ user base. Still ar apps will drive this rventually... I think starlink is nicely positioned now to meet their demanding growth goals and humanity's future in space assured, so there's that. ( i still would rather like elone to send over a nice pair of tesla motors and battery pack for my sailboat) I did have a nice jam with ajit Pai last week who is now well on his way towards getting it. (See my twitter for the pics) On Mon, May 6, 2024, 4:25=E2=80=AFAM Rich Brown via Starlink < starlink@lists.bufferbloat.net> wrote: > Hi Gene, > > I've been vacillating on whether to send this note, but have decided to > pull the trigger. I apologize in advance for the "Debbie Downer" nature o= f > this message. I also apologize for any errors, omissions, or > over-simplifications of the "birth of bufferbloat" story and its fixes. > Corrections welcome. > > Rich > ------ > > If we are going to take a shot at opening people's eyes to bufferbloat, w= e > should know some of the "objections" we'll run up against. Even though > there's terrific technical data to back it up, people seem especially > resistant to thinking that bufferbloat might affect their network, even > when they're seeing problems that sound exactly like bufferbloat symptoms= . > But first, some history: > > The very idea of bufferbloat is simply unbelievable. Jim Gettys in 2011 > [1] couldn't believe it, and he's a smart networking guy,. At the time, i= t > seemed incredible (that is "not credible" =3D=3D impossible) that somethi= ng > could induce 1.2 seconds of latency into his home network connection. He > called in favors from technical contacts at his ISP and at Bell Labs who > went over everything with a fine-toothed comb. It was all exactly as > spec'd. But he still had the latency. > > This led Jim and Dave T=C3=A4ht to start the investigation into the pheno= menon > known today as "bufferbloat" - the undesirable latency that comes from a > router or other network equipment buffering too much data. Over several > years, a group of smart people made huge improvements: fq_codel was > released 14 May 2012 [3]; it was incorporated into the Linux kernel short= ly > afterward. CAKE came in 2015, and the fixes that minimize bufferbloat in > Wi-Fi arrived in 2018. In 2021 cake-autorate [4] arrived to handle varyin= g > speed ISP links. All these techniques work great: in 2014, my 7mbps DSL > link was quite usable. And when the pandemic hit, fq_codel on my OpenWrt > router allowed me to use that same 7mbps DSL line for two simultaneous zo= om > calls. > > As one of the authors of [2], I am part of the team that has tried over > the years to explain bufferbloat and how to fix it. We've spoken with > vendors. We've spent untold hours responding to posts on assorted boards > and forums with the the bufferbloat story. > > With these technical fixes in hand, we cockily set about to tell the worl= d > about how to fix bufferbloat. Our efforts have been met with skepticism a= t > best, or stony silence. What are the objections? > > - This is just the ordinary behavior: I would expect things to be slower > when there's more traffic (Willfully ignoring orders of magnitude increas= e > in delay.) > - Besides, I'm the only one using the internet. (Except when my phone > uploads photos. Or my computer kicks off some automated process. Or I > browse the web. Or ...) > - It only happens some of the time. (Exactly. That's probably when > something's uploading photos, or your computer is doing stuff in the > background.) > - Those bufferbloat tests you hear about are bogus. They artificially add > load, which isn't a realistic test. (...and if you actually are downloadi= ng > a file?) > - Bufferbloat only happens when the network is 100% loaded. (True. But > when you open a web page, your browser briefly uses 100% of the link. Is > this enough to cause momentary lag?) > - It's OK. I just tell my kids/spouse not to use the internet when I'm > gaming. (Huh?) > - I have gigabit service from my ISP. (That helps, but if you're > complaining about "slowness" you still need to rule out bufferbloat in yo= ur > router.) > - I can't believe that router manufacturers would ever allow such a thing > to happen in their gear. (See the Jim Gettys story above.) > - I mean... wouldn't router vendors want to provide the best for their > customers? (No - implementing this (new-ish) code requires engineering > effort. They're selling plenty of routers with decade-old software. The > Boss says, "would we sell more if they made these changes? Probably not."= ) > - Why would my ISP provision/sell me a router that gave crappy service? > They're a big company, they must know about this stuff. (Maybe. We have > reached out to all the vendors. But remember they profit if you decide yo= ur > network is too slow and you upgrade to a faster device/plan.) > - But couldn't I just tweak the QoS on my router? (Maybe. But see [5]) > - Besides, I just spent $300 on a "gaming router". Obviously, I bought th= e > most expensive/best possible solution on the market (But I still have > lag...) > - You're telling me that a bunch of pointy-headed academics are smarter > than commercial router developers - who sold me that $300 router? (I can'= t > believe it.) > - And then you say that I should throw away that gaming router and instal= l > some "open source firmware"? (What the heck is that? And why should I > believe you?) > - What if it doesn't solve the problem? Who will give me support? And how > will I get back to a vendor-supported system? (Valid point - the first > valid point) > - Aren't there any commercial solutions I can just buy? (Not at the > moment. IQrouter was a shining light here - available from Amazon, simple > setup, worked a treat - but they have gone out of business. And of course= , > for the skeptic, this is proof that the "fq_codel-stuff" isn't really a > solution - it seems just like snake oil.) > > So... All these hurdles make it hard to convince people that bufferbloat > could be the problem, or that they can fix for themselves. > > A couple of us have reached out to Consumer Reports, wondering if they > would like a story about how vendors would prefer to sell you a new, fast= er > router (or new faster ISP plan) than fix your bufferbloat. This kind of > story seemed to be straight up their alley, but we never heard back after > an initial contact. Maybe they deserve another call... > > The recent latency results from Starlink give me a modicum of hope. > They're a major player. They (and their customers) can point to an order = of > magnitude reduction in latency over other solutions. It still requires > enough "regular customers" to tell their current ISP that they are > switching to Starlink (and spend $600 to purchase a Dishy plus $100/month= ) > to provide a market incentive. > > Despite all this doom and gloom, I remain hopeful that things will get > better. We know the technology exists for people to take control of their > network and solve the problem for themselves. We can continue to respond = on > forums where people express their dismay at the crummy performance and > suggest a solution. We can hope that a major vendor will twig to this > effect and bring out a mass-market solution. > > I think your suggestion of speaking to eSports people is intriguing. > They're highly motivated to make their personal networks better. And > actually solving the problem would have a network effect of bringing in > others with the same problem. > > Good luck, and thanks for thinking about this. > > Rich Brown > > [1] > https://courses.cs.washington.edu/courses/cse550/21au/papers/bufferbloat.= pdf > [2] > https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Buffe= rbloat/ > [3] > https://lists.bufferbloat.net/pipermail/cerowrt-devel/2012-May/000233.htm= l > [4] https://github.com/lynxthecat/cake-autorate > [5] > https://www.bufferbloat.net/projects/bloat/wiki/More_about_Bufferbloat/#w= hat-s-wrong-with-simply-configuring-qos > > On May 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink < > starlink@lists.bufferbloat.net> wrote: > > Of course. For the gamers, the focus is managing latency. They have > control of everything else. > > With our high latency and wide range of values, the eSports teams train o= n > campus. It will be interesting to see how much improvements there can be > for teams to be able to training from their homes. > > Gene > ---------------------------------------------- > Eugene Chang > IEEE Life Senior Member > IEEE Communications Society & Signal Processing Society, > Hawaii Chapter Chair > IEEE Life Member Affinity Group Hawaii Chair > IEEE Entrepreneurship, Mentor > eugene.chang@ieee.org > m 781-799-0233 (in Honolulu) > > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > --000000000000e1c06a0617e7e331 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
This was a wonderful post, rich!

I note that preseem, paraqum,=C2=A0 bequant and libreq= os (a bufferbloat.net backed project= ) are in the fq codel or cake Middlebox for isps *Qoe) market and all of us= have made a substantial dent in the problem for oh, call it 1000 isps worl= dwide total between us. Comcast also has done a pretty good job but it seem= s yhe rest of the cable industry is asleep at the switch.

The wisps totally got it with fq codel an= d cake arriving native for mikeotiks entire product line and much of ubnts = gear prior to that.


=
Qoe is still a pretty hard sell. Libreqos has a ton= of free users and we think over a million devices managed by it but not en= ough paid users to justify even 1/10th the investment we have made so far i= nto it (something that I hope turns around with the upcoming v1.5 lts relea= se and some outputs from the nlnet and Comcast funded cakemaint and nqb pro= jects)

Thing is, at high= er and fiber rates all the bloat moves to the wifi, and a ton of that, like= eero especially was long ago fq codeled and so I think several major playe= rs have also (except for those stuck with broadcom).=C2=A0

That said there are a lot of defective w= ifi aps left to replace. Nearly every coffee shop I have been in with the e= xception of Starbucks has really lousy wifi.

I am so thrilled to see what starlink has accomplished= so far with their rollout of bufferbloa= t.net stuff and look forward to more. They are still missing a few tric= ks... but are aware of what tricks they are missing...

Lack of knowledge of which regrettably remai= ns the case for 97% of the market and 99.99$ user base. Still ar apps will = drive this rventually... I think starlink is nicely positioned now to meet = their demanding growth goals and humanity's future in space assured, so= there's that. ( i still would rather like elone to send over a nice pa= ir of tesla motors and battery pack for my sailboat)

I did have a nice jam with ajit Pai last week = who is now well on his way towards getting it. (See my twitter for the pics= )

On Mon, May 6, 2024, 4:25=E2=80=AFAM Rich Brown via Starlink <starlink@lists.bufferbloat.n= et> wrote:
Hi Gene,
I've been vacillating on whether to send this note, bu= t have decided to pull the trigger. I apologize in advance for the "De= bbie Downer" nature of this message. I also apologize for any errors, = omissions, or over-simplifications of the "birth of bufferbloat" = story and its fixes. Corrections welcome.

Ri= ch
------

If we are going to take a shot= at opening people's eyes to bufferbloat, we should know some of the &q= uot;objections" we'll run up against. Even though there's terr= ific technical data to back it up, people seem especially resistant to thin= king that bufferbloat might affect their network, even when they're see= ing problems that sound exactly like bufferbloat symptoms. But first, some = history:

The very idea of bufferbloat is simply unbelievable.= Jim Gettys in 2011 [1] couldn't believe it, and he's a smart netwo= rking guy,. At the time, it seemed incredible (that is "not=C2=A0credi= ble" =3D=3D impossible) that something could induce 1.2 seconds of lat= ency into his home network connection. He called in favors from technical c= ontacts at his ISP and=C2=A0at Bell Labs who went over everything with a fi= ne-toothed comb. It was all exactly as spec'd. But he still had the lat= ency.=C2=A0

This led Jim and Dave T=C3=A4ht to start the investigati= on into the phenomenon known today as "bufferbloat" - the undesir= able latency that comes from a router or other network=C2=A0equipment buffe= ring too much data. Over several years, a group of smart people made huge i= mprovements: fq_codel was released 14 May 2012 [3]; it was incorporated=C2= =A0into the Linux kernel shortly afterward. CAKE came in 2015, and the fixe= s that minimize bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate= [4] arrived to handle=C2=A0varying speed ISP links. All these techniques w= ork great: in 2014, my 7mbps DSL link was quite usable. And when the pandem= ic hit, fq_codel on my OpenWrt router=C2=A0allowed me to use that same 7mbp= s DSL line for two simultaneous zoom calls.=C2=A0

= As one of the authors of [2], I am part of the team that has tried over the= years to explain bufferbloat and how to fix it. We've spoken with vend= ors. We've spent untold hours responding to posts on assorted boards an= d forums with the the bufferbloat story.=C2=A0

Wit= h these technical fixes in hand, we cockily set about to tell the world abo= ut how to fix bufferbloat. Our efforts have been met with skepticism at bes= t, or stony silence. What are the objections?=C2=A0

- This is just the ordinary behavior: I would expect things to be slower = when there's more traffic (Willfully ignoring orders of magnitude incre= ase in delay.)
- Besides, I'm the only one using the internet= . (Except when my phone uploads photos. Or my computer kicks off some autom= ated process. Or I browse the web. Or ...)
- It only happens some= of the time. (Exactly. That's probably when something's uploading = photos, or your computer is doing stuff in the background.)
- Tho= se bufferbloat tests you hear about are bogus. They artificially add load, = which isn't a realistic test. (...and if you actually are downloading a= file?)
- Bufferbloat only happens when the network is 100% loade= d. (True. But when you open a web page, your browser briefly uses 100% of t= he link. Is this enough to cause momentary lag?)
- It's OK. I= just tell my kids/spouse not to use the internet when I'm gaming. (Huh= ?)
- I have gigabit service from my ISP. (That helps, but if you&= #39;re complaining about "slowness" you still need to rule out bu= fferbloat in your router.)
- I can't believe that router manu= facturers would ever allow such a thing to happen in their gear. (See the J= im Gettys story above.)
- I mean... wouldn't router vendors w= ant to provide the best for their customers? (No - implementing this (new-i= sh) code requires engineering effort. They're selling plenty of routers= with decade-old software. The Boss says, "would we sell more if they = made these changes? Probably not.")
- Why would my ISP provi= sion/sell me a router that gave crappy service? They're a big company, = they must know about this stuff. (Maybe. We have reached out to all the ven= dors. But remember they profit if you decide your network is too slow and y= ou upgrade to a faster device/plan.)
- But couldn't I just tw= eak the QoS on my router? (Maybe. But see [5])
- Besides, I just = spent $300 on a "gaming router". Obviously, I bought the most exp= ensive/best possible solution on the market (But I still have lag...)
=
- You're telling me that a bunch of pointy-headed academics are sm= arter than commercial router developers - who sold me that $300 router? (I = can't believe it.)
- And then you say that I should throw awa= y that gaming router and install some "open source firmware"? (Wh= at the heck is that? And why should I believe you?)=C2=A0
- What = if it doesn't solve the problem? Who will give me support? And how will= I get back to a vendor-supported system? (Valid point - the first valid po= int)
- Aren't there any commercial solutions I can just buy? = (Not at the moment. IQrouter was a shining light here - available from Amaz= on, simple setup, worked a treat - but they have gone out of business. And = of course, for the skeptic, this is proof that the "fq_codel-stuff&quo= t; isn't really a solution - it seems just like snake oil.)
<= br>
So... All these hurdles make it hard to convince people that = bufferbloat could be the problem, or that they can fix for themselves.

A couple of us have reached out to Consumer Reports, w= ondering if they would like a story about how vendors would prefer to sell = you a new, faster router (or new faster ISP plan) than fix your bufferbloat= . This kind of story seemed to be straight up their alley, but we never hea= rd back after an initial contact. Maybe they deserve another call...
<= div>
The recent latency results from Starlink give me a modic= um of hope. They're a major player. They (and their customers) can poin= t to an order of magnitude reduction in latency over other solutions. It st= ill requires enough "regular customers" to tell their current ISP= that they are switching to Starlink (and spend $600 to purchase a Dishy pl= us $100/month) to provide a market incentive.

Desp= ite all this doom and gloom, I remain hopeful that things will get better. = We know the technology exists for people to take control of their network a= nd solve the problem for themselves. We can continue to respond on forums w= here people express their dismay at the crummy performance and suggest a so= lution. We can hope that a major vendor will twig to this effect and bring = out a mass-market solution.

I think your suggestio= n of speaking to eSports people is intriguing. They're highly motivated= to make their personal networks better. And actually solving the problem w= ould have a network effect of bringing in others with the same problem.=C2= =A0

Good luck, and thanks for thinking about this.=

Rich Brown

[2]=C2=A0<= a href=3D"https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_abo= ut_Bufferbloat/" target=3D"_blank" rel=3D"noreferrer">https://www.bufferblo= at.net/projects/bloat/wiki/What_can_I_do_about_Bufferbloat/
<= div>[5]=C2=A0https://www.bufferbloat.net/projects/bloat/wiki/Mo= re_about_Bufferbloat/#what-s-wrong-with-simply-configuring-qos

On May 1, 2024, at 6:19 PM, Eugene Y C= hang via Starlink <starlink@lists.bufferbloat.net> wr= ote:

Of course. For the gamers, the focus is managing latency. They hav= e control of everything else.

With our high latency and = wide range of values, the eSports teams train on campus. It will be interes= ting to see how much improvements there can be for teams to be able to trai= ning from their homes.
=

Gene
----------------------------= ------------------
Eugene Chang
IEEE Life Senior Member
<= div>IEEE Communications Society & Signal Processing Society, =C2=A0 =C2= =A0
=C2=A0 =C2=A0 Hawaii Chapter Chair
IEEE Life Member Affini= ty Group Hawaii Chair
IEEE Entrepreneurship, Mentor
eugene.= chang@ieee.org
m 781-799-0233 (in Honolulu)

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/sta= rlink
--000000000000e1c06a0617e7e331--