From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 881443B2A4 for ; Mon, 6 May 2024 20:38:51 -0400 (EDT) Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6f44881ad9eso2023322b3a.3 for ; Mon, 06 May 2024 17:38:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; t=1715042330; x=1715647130; darn=lists.bufferbloat.net; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=GRSssVDFZ0In/NumdgBqqQoFT6mUoV2jqJMQxT7cg/0=; b=Luhcng2t3ISReGiNS3Q00z923XFg0jSakEiJnFcUQ0/HwnE2Lno7RAvSRntXV0ynwD pKgi85QI1E71KKmtcQ/1TLHrW2dd6AAWO8whBBsLWVJhovtMCTYJQg60AfA+0PYX4jbZ nP9h7uAdeCVW4Z1R9jKPExrHNRV/Ig6Lg7kDA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715042330; x=1715647130; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GRSssVDFZ0In/NumdgBqqQoFT6mUoV2jqJMQxT7cg/0=; b=Y6n1kOuHmtc9FySGURqDgUXbqMJwWnb2o1sa6j2HfeSjwLc0mC1WQkXxoYWrBIf3HT 40lIq3AiqVSqkeT//Bzt1P+sCk9yn9Y4wj/tPBdlTH2ClAk50R5vbcqOHiYNhGF2ghFM wJTmZ07OKsEfU5AB8YFGmTiTBfw3YpY6dYOPbmPktdaGilaVUBx5F9DugbJFwAmp923u B19mhAcrg+qFVtSjXuQbyc5Rx1dCdNfljCXOm140llXOlQfGI9EcjMqMB66Ch1Ep2HX7 jrUmzCJ+2RELS+QVBsf1AcpOhTbBm59pxE3QRbxvQ2xdtFwgKtKt8c1YSHmlAKz7NFRc T0Xg== X-Forwarded-Encrypted: i=1; AJvYcCXrWKakEsIEtKbezhUWu5HYT1+XZXQbzMtSWEee45KelqkfAq48uf5VBVio+4K3bAhXwE3gDVoLGMxCxmKXwsy4ShJYQibnpiGXWtwZcQQ= X-Gm-Message-State: AOJu0Yz8DQK28el8lYKub/6OJSiA8kx+xoHwx1TnGrINx/B5w7AoaHY5 mPls0C2GOEjTdg8XnCvnTfL9BBCmQgAq4xsvrhBDn78hvkDj+pj8An9H/k612g== X-Google-Smtp-Source: AGHT+IFsIzYZvn/0ZB95pcNlSVt/vHJ5f8l0nwSjOYCn9vgqQfqkvysGKXOu2hIxWU2GIkQME0O5wg== X-Received: by 2002:a05:6a00:1799:b0:6f3:ed26:e4f5 with SMTP id s25-20020a056a00179900b006f3ed26e4f5mr14242314pfg.7.1715042329734; Mon, 06 May 2024 17:38:49 -0700 (PDT) Received: from smtpclient.apple (dhcp-72-253-194-45.hawaiiantel.net. [72.253.194.45]) by smtp.gmail.com with ESMTPSA id b23-20020aa78717000000b006f451dfd7dcsm5714080pfo.70.2024.05.06.17.38.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 May 2024 17:38:49 -0700 (PDT) From: Eugene Y Chang Message-Id: <961EDB45-E91D-4D1D-B81B-9D032E8CFC6B@ieee.org> Content-Type: multipart/signed; boundary="Apple-Mail=_332A7A3D-3E06-4981-8E5F-ECEA403E4EF7"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Date: Mon, 6 May 2024 14:38:46 -1000 In-Reply-To: Cc: Eugene Y Chang , Sebastian Moeller , Colin_Higbie , Dave Taht via Starlink To: Rich Brown References: <438B1BC4-D465-497A-B6BA-700E1D411036@ieee.org> <79C02ABB-B2A6-4B4D-98F4-6540D3F96EBB@ieee.org> <7E918B58-382A-4793-A144-13A7075CA56C@connectivitycap.com> <13rq2389-9012-p95n-s494-q3pp070s497n@ynat.uz> <6qop2p3o-351p-788q-q1q2-86sosnq3rn21@ynat.uz> <3FF32F52-4A93-496B-85FF-00020FA4A48B@gmx.de> <08F6942E-CC08-4956-B92E-CBEC091D86E4@ieee.org> X-Mailer: Apple Mail (2.3696.120.41.1.8) Subject: Re: [Starlink] The "reasons" that bufferbloat isn't a problem X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 May 2024 00:38:51 -0000 --Apple-Mail=_332A7A3D-3E06-4981-8E5F-ECEA403E4EF7 Content-Type: multipart/alternative; boundary="Apple-Mail=_5243398B-D281-497F-A9F0-9D9F1137834F" --Apple-Mail=_5243398B-D281-497F-A9F0-9D9F1137834F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Rich, Thanks for the recap in the email. I have seen all of those bits. I will help with the marketing magic needed. We need a team of smart people engaged to help vouch for the technical = integrity. We need a simple case (call it a special case, if you must) that shows = the problem can be fixed. Never mind if it is not a universal fix. We only need to show one happy, very visible community. Give me something to work with that we can defend from the list of = do-nothing reasons you list. It seems like you signed off on this challenge. Don=E2=80=99t do that. = Help give me the tools to push this to the next level. An energetic, vocal community is very valuable. They aren=E2=80=99t = satisfying if we want to debate the technology. We shouldn=E2=80=99t = care. We just want to win adoption. Gene ---------------------------------------------- Eugene Chang IEEE Life Senior Member IEEE Communications Society & Signal Processing Society, Hawaii Chapter Chair IEEE Life Member Affinity Group Hawaii Chair IEEE Entrepreneurship, Mentor eugene.chang@ieee.org m 781-799-0233 (in Honolulu) > On May 6, 2024, at 1:25 AM, Rich Brown = wrote: >=20 > Hi Gene, >=20 > I've been vacillating on whether to send this note, but have decided = to pull the trigger. I apologize in advance for the "Debbie Downer" = nature of this message. I also apologize for any errors, omissions, or = over-simplifications of the "birth of bufferbloat" story and its fixes. = Corrections welcome. >=20 > Rich > ------ >=20 > If we are going to take a shot at opening people's eyes to = bufferbloat, we should know some of the "objections" we'll run up = against. Even though there's terrific technical data to back it up, = people seem especially resistant to thinking that bufferbloat might = affect their network, even when they're seeing problems that sound = exactly like bufferbloat symptoms. But first, some history: >=20 > The very idea of bufferbloat is simply unbelievable. Jim Gettys in = 2011 [1] couldn't believe it, and he's a smart networking guy,. At the = time, it seemed incredible (that is "not credible" =3D=3D impossible) = that something could induce 1.2 seconds of latency into his home network = connection. He called in favors from technical contacts at his ISP and = at Bell Labs who went over everything with a fine-toothed comb. It was = all exactly as spec'd. But he still had the latency. >=20 > This led Jim and Dave T=C3=A4ht to start the investigation into the = phenomenon known today as "bufferbloat" - the undesirable latency that = comes from a router or other network equipment buffering too much data. = Over several years, a group of smart people made huge improvements: = fq_codel was released 14 May 2012 [3]; it was incorporated into the = Linux kernel shortly afterward. CAKE came in 2015, and the fixes that = minimize bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate [4] = arrived to handle varying speed ISP links. All these techniques work = great: in 2014, my 7mbps DSL link was quite usable. And when the = pandemic hit, fq_codel on my OpenWrt router allowed me to use that same = 7mbps DSL line for two simultaneous zoom calls. >=20 > As one of the authors of [2], I am part of the team that has tried = over the years to explain bufferbloat and how to fix it. We've spoken = with vendors. We've spent untold hours responding to posts on assorted = boards and forums with the the bufferbloat story. >=20 > With these technical fixes in hand, we cockily set about to tell the = world about how to fix bufferbloat. Our efforts have been met with = skepticism at best, or stony silence. What are the objections? >=20 > - This is just the ordinary behavior: I would expect things to be = slower when there's more traffic (Willfully ignoring orders of magnitude = increase in delay.) > - Besides, I'm the only one using the internet. (Except when my phone = uploads photos. Or my computer kicks off some automated process. Or I = browse the web. Or ...) > - It only happens some of the time. (Exactly. That's probably when = something's uploading photos, or your computer is doing stuff in the = background.) > - Those bufferbloat tests you hear about are bogus. They artificially = add load, which isn't a realistic test. (...and if you actually are = downloading a file?) > - Bufferbloat only happens when the network is 100% loaded. (True. But = when you open a web page, your browser briefly uses 100% of the link. Is = this enough to cause momentary lag?) > - It's OK. I just tell my kids/spouse not to use the internet when I'm = gaming. (Huh?) > - I have gigabit service from my ISP. (That helps, but if you're = complaining about "slowness" you still need to rule out bufferbloat in = your router.) > - I can't believe that router manufacturers would ever allow such a = thing to happen in their gear. (See the Jim Gettys story above.) > - I mean... wouldn't router vendors want to provide the best for their = customers? (No - implementing this (new-ish) code requires engineering = effort. They're selling plenty of routers with decade-old software. The = Boss says, "would we sell more if they made these changes? Probably = not.") > - Why would my ISP provision/sell me a router that gave crappy = service? They're a big company, they must know about this stuff. (Maybe. = We have reached out to all the vendors. But remember they profit if you = decide your network is too slow and you upgrade to a faster = device/plan.) > - But couldn't I just tweak the QoS on my router? (Maybe. But see [5]) > - Besides, I just spent $300 on a "gaming router". Obviously, I bought = the most expensive/best possible solution on the market (But I still = have lag...) > - You're telling me that a bunch of pointy-headed academics are = smarter than commercial router developers - who sold me that $300 = router? (I can't believe it.) > - And then you say that I should throw away that gaming router and = install some "open source firmware"? (What the heck is that? And why = should I believe you?) > - What if it doesn't solve the problem? Who will give me support? And = how will I get back to a vendor-supported system? (Valid point - the = first valid point) > - Aren't there any commercial solutions I can just buy? (Not at the = moment. IQrouter was a shining light here - available from Amazon, = simple setup, worked a treat - but they have gone out of business. And = of course, for the skeptic, this is proof that the "fq_codel-stuff" = isn't really a solution - it seems just like snake oil.) >=20 > So... All these hurdles make it hard to convince people that = bufferbloat could be the problem, or that they can fix for themselves. >=20 > A couple of us have reached out to Consumer Reports, wondering if they = would like a story about how vendors would prefer to sell you a new, = faster router (or new faster ISP plan) than fix your bufferbloat. This = kind of story seemed to be straight up their alley, but we never heard = back after an initial contact. Maybe they deserve another call... >=20 > The recent latency results from Starlink give me a modicum of hope. = They're a major player. They (and their customers) can point to an order = of magnitude reduction in latency over other solutions. It still = requires enough "regular customers" to tell their current ISP that they = are switching to Starlink (and spend $600 to purchase a Dishy plus = $100/month) to provide a market incentive. >=20 > Despite all this doom and gloom, I remain hopeful that things will get = better. We know the technology exists for people to take control of = their network and solve the problem for themselves. We can continue to = respond on forums where people express their dismay at the crummy = performance and suggest a solution. We can hope that a major vendor will = twig to this effect and bring out a mass-market solution. >=20 > I think your suggestion of speaking to eSports people is intriguing. = They're highly motivated to make their personal networks better. And = actually solving the problem would have a network effect of bringing in = others with the same problem. >=20 > Good luck, and thanks for thinking about this. >=20 > Rich Brown >=20 > [1] = https://courses.cs.washington.edu/courses/cse550/21au/papers/bufferbloat.p= df = [2] = https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Buffer= bloat/ = > [3] = https://lists.bufferbloat.net/pipermail/cerowrt-devel/2012-May/000233.html= = > [4] https://github.com/lynxthecat/cake-autorate = > [5] = https://www.bufferbloat.net/projects/bloat/wiki/More_about_Bufferbloat/#wh= at-s-wrong-with-simply-configuring-qos = >=20 >> On May 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink = > = wrote: >>=20 >> Of course. For the gamers, the focus is managing latency. They have = control of everything else. >>=20 >> With our high latency and wide range of values, the eSports teams = train on campus. It will be interesting to see how much improvements = there can be for teams to be able to training from their homes. >>=20 >> Gene >> ---------------------------------------------- >> Eugene Chang >> IEEE Life Senior Member >> IEEE Communications Society & Signal Processing Society, >> Hawaii Chapter Chair >> IEEE Life Member Affinity Group Hawaii Chair >> IEEE Entrepreneurship, Mentor >> eugene.chang@ieee.org >> m 781-799-0233 (in Honolulu) >=20 --Apple-Mail=_5243398B-D281-497F-A9F0-9D9F1137834F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Rich,
Thanks for the recap in the = email.
I have seen all of those bits.

I will help with the = marketing magic needed.
We need a team of smart = people engaged to help vouch for the technical integrity.

We need a simple case = (call it a special case, if you must) that shows the problem can be = fixed.
Never mind if it is not a universal = fix. 
We only need to show one happy, very = visible community.
Give me something to work with = that we can defend from the list of do-nothing reasons you = list.

It seems = like you signed off on this challenge. Don=E2=80=99t do that. Help give = me the tools to push this to the next level.
An = energetic, vocal community is very valuable. They aren=E2=80=99t = satisfying if we want to debate the technology. We shouldn=E2=80=99t = care. We just want to win adoption.

Gene
----------------------------------------------
Eugene Chang
IEEE Life Senior Member
IEEE = Communications Society & Signal Processing Society,    
    Hawaii Chapter Chair
IEEE Life Member = Affinity Group Hawaii Chair
IEEE Entrepreneurship, Mentor
eugene.chang@ieee.org
m 781-799-0233 (in = Honolulu)



On May 6, 2024, at 1:25 AM, Rich Brown <richb.hanover@gmail.com> wrote:

Hi = Gene,

I've been vacillating on whether to send this note, but have = decided to pull the trigger. I apologize in advance for the "Debbie = Downer" nature of this message. I also apologize for any errors, = omissions, or over-simplifications of the "birth of bufferbloat" story = and its fixes. Corrections welcome.

Rich
------

If we are going to take a shot at opening people's eyes to = bufferbloat, we should know some of the "objections" we'll run up = against. Even though there's terrific technical data to back it up, = people seem especially resistant to thinking that bufferbloat might = affect their network, even when they're seeing problems that sound = exactly like bufferbloat symptoms. But first, some history:

The very idea of bufferbloat is simply = unbelievable. Jim Gettys in 2011 [1] couldn't believe it, and he's a = smart networking guy,. At the time, it seemed incredible (that is = "not credible" =3D=3D impossible) that something could induce 1.2 = seconds of latency into his home network connection. He called in favors = from technical contacts at his ISP and at Bell Labs who went over = everything with a fine-toothed comb. It was all exactly as spec'd. But = he still had the latency. 

This led = Jim and Dave T=C3=A4ht to start the investigation into the phenomenon = known today as "bufferbloat" - the undesirable latency that comes from a = router or other network equipment buffering too much data. Over = several years, a group of smart people made huge improvements: fq_codel = was released 14 May 2012 [3]; it was incorporated into the Linux = kernel shortly afterward. CAKE came in 2015, and the fixes that minimize = bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate [4] arrived = to handle varying speed ISP links. All these techniques work great: = in 2014, my 7mbps DSL link was quite usable. And when the pandemic hit, = fq_codel on my OpenWrt router allowed me to use that same 7mbps DSL = line for two simultaneous zoom calls. 

As one of the authors of [2], I am part = of the team that has tried over the years to explain bufferbloat and how = to fix it. We've spoken with vendors. We've spent untold hours = responding to posts on assorted boards and forums with the the = bufferbloat story. 

With these technical fixes in hand, we cockily set about to = tell the world about how to fix bufferbloat. Our efforts have been met = with skepticism at best, or stony silence. What are the = objections? 

- This is just the ordinary behavior: I would expect things = to be slower when there's more traffic (Willfully ignoring orders of = magnitude increase in delay.)
- Besides, I'm the = only one using the internet. (Except when my phone uploads photos. Or my = computer kicks off some automated process. Or I browse the web. Or = ...)
- It only happens some of the time. (Exactly. = That's probably when something's uploading photos, or your computer is = doing stuff in the background.)
- Those bufferbloat = tests you hear about are bogus. They artificially add load, which isn't = a realistic test. (...and if you actually are downloading a = file?)
- Bufferbloat only happens when the network = is 100% loaded. (True. But when you open a web page, your browser = briefly uses 100% of the link. Is this enough to cause momentary = lag?)
- It's OK. I just tell my kids/spouse not to = use the internet when I'm gaming. (Huh?)
- I have = gigabit service from my ISP. (That helps, but if you're complaining = about "slowness" you still need to rule out bufferbloat in your = router.)
- I can't believe that router = manufacturers would ever allow such a thing to happen in their gear. = (See the Jim Gettys story above.)
- I mean... = wouldn't router vendors want to provide the best for their customers? = (No - implementing this (new-ish) code requires engineering effort. = They're selling plenty of routers with decade-old software. The Boss = says, "would we sell more if they made these changes? Probably = not.")
- Why would my ISP provision/sell me a = router that gave crappy service? They're a big company, they must know = about this stuff. (Maybe. We have reached out to all the vendors. But = remember they profit if you decide your network is too slow and you = upgrade to a faster device/plan.)
- But couldn't I = just tweak the QoS on my router? (Maybe. But see [5])
- Besides, I just spent $300 on a "gaming router". Obviously, = I bought the most expensive/best possible solution on the market (But I = still have lag...)
- You're telling me that a bunch = of pointy-headed academics are smarter than commercial router developers = - who sold me that $300 router? (I can't believe it.)
- And then you say that I should throw away that gaming = router and install some "open source firmware"? (What the heck is that? = And why should I believe you?) 
- What if it = doesn't solve the problem? Who will give me support? And how will I get = back to a vendor-supported system? (Valid point - the first valid = point)
- Aren't there any commercial solutions I = can just buy? (Not at the moment. IQrouter was a shining light here - = available from Amazon, simple setup, worked a treat - but they have gone = out of business. And of course, for the skeptic, this is proof that the = "fq_codel-stuff" isn't really a solution - it seems just like snake = oil.)

So... = All these hurdles make it hard to convince people that bufferbloat could = be the problem, or that they can fix for themselves.

A couple of us have = reached out to Consumer Reports, wondering if they would like a story = about how vendors would prefer to sell you a new, faster router (or new = faster ISP plan) than fix your bufferbloat. This kind of story seemed to = be straight up their alley, but we never heard back after an initial = contact. Maybe they deserve another call...

The recent latency results from = Starlink give me a modicum of hope. They're a major player. They (and = their customers) can point to an order of magnitude reduction in latency = over other solutions. It still requires enough "regular customers" to = tell their current ISP that they are switching to Starlink (and spend = $600 to purchase a Dishy plus $100/month) to provide a market = incentive.

Despite all this doom and gloom, I remain hopeful that things = will get better. We know the technology exists for people to take = control of their network and solve the problem for themselves. We can = continue to respond on forums where people express their dismay at the = crummy performance and suggest a solution. We can hope that a major = vendor will twig to this effect and bring out a mass-market = solution.

I = think your suggestion of speaking to eSports people is intriguing. = They're highly motivated to make their personal networks better. And = actually solving the problem would have a network effect of bringing in = others with the same problem. 

Good luck, and thanks for thinking = about this.

Rich= Brown

[2] https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_a= bout_Bufferbloat/

On May 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink = <starlink@lists.bufferbloat.net> wrote:

Of course. For the = gamers, the focus is managing latency. They have control of everything = else.

With our high = latency and wide range of values, the eSports teams train on campus. It = will be interesting to see how much improvements there can be for teams = to be able to training from their homes.

Gene
----------------------------------------------
Eugene Chang
IEEE Life Senior = Member
IEEE Communications Society & Signal = Processing Society,    
    Hawaii = Chapter Chair
IEEE Life Member Affinity Group = Hawaii Chair
IEEE Entrepreneurship, Mentor
eugene.chang@ieee.org
m 781-799-0233 (in = Honolulu)


= --Apple-Mail=_5243398B-D281-497F-A9F0-9D9F1137834F-- --Apple-Mail=_332A7A3D-3E06-4981-8E5F-ECEA403E4EF7 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEERPTGiBqcibajhTSsv0/8FiYdKmAFAmY5eBYACgkQv0/8FiYd KmCdGBAAi9hjSW5PMh8D3BRNaNjTaP5qd0jIit+fFd4GAtVyysQfp5oBEVM0j1Os 7Wvu6/enkyvxHyL78lcQ1RCk2lzagrnLpT/mfv7WCD+TOzhM5y62Ii9KI3j0h6JC 9CRYFRMZDTksNyPZ2sc/Myocr9YysXzzI0JdUadnBiHzpN9lTnv4tXzxhJDwRrsv /BwCi5Ybole8Lfghtzy1yHQh3h/aJjJeypU0My4X44EOCmyqkIfoI89trX5ATMnG CnLvb8iZMioMNmJ0mH3z+dK2apWlxlf2Jqdm8zlOytN6tf65zADywyjvXOsSab6S WFaK46GRH/UHZ79aIzmm9hfbVBgoBIN4wJzX+SuHs0HXuZDaUBfkg6KPfbx5fvPT Y/sAvQI54tj7lCIk6/jW2WbUl/wS//wqKxmuWDAyUBI1wUhZ5DELc6/7IP3fxnje nKJUVrXfp5mFSqpgrCl7glcDcTkIHw6YO0u0Ykk+seMK2UUS3ZOR168p6Q6rp3H6 OQiE2be7fyj99FuqwhwKKbFpZ1Fq5mqHEXlnPxcWdffJ/UJH27AcD65SvW4ieabd moEBPU+m32ZiupG9Fc3MqnrXwlwM/kDNe1jOp6pTAjwF4uUSMX0+TUcOlFKvimO6 D+snvDaMZLrZavuvkDAagk9+5cFywVVeCgi62yWZMiZ3ZtF6Eec= =lu6V -----END PGP SIGNATURE----- --Apple-Mail=_332A7A3D-3E06-4981-8E5F-ECEA403E4EF7--