From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 413163CB62 for ; Mon, 6 May 2024 07:25:14 -0400 (EDT) Received: by mail-qt1-x833.google.com with SMTP id d75a77b69052e-439656c1b50so10972861cf.1 for ; Mon, 06 May 2024 04:25:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714994713; x=1715599513; darn=lists.bufferbloat.net; h=message-id:in-reply-to:to:references:date:subject:mime-version:from :from:to:cc:subject:date:message-id:reply-to; bh=6FDopxpQehU0k+rAnWTjWr+vUJUQDZ0CAWTTYqhX3MA=; b=AKx6jLNFXsiVLC7tv1Lon1UwIkWBJIgDBUp11vUR6ZkvR1cvVAqdnxQ7CB692V1NR3 beF+EJsBV+G1xwnBtjwrp5EweRQUvjSzP1DVSND+9/mxZWDzEP397VvQo6gZ0vKUMaJU 6FvjGak4L1pdt5EpXoVUX3lMRDywaXloecmv3E8jSQPhBfuek9RHijvRo3iwLGgLHYGY 4QwcwoDh6yEJjcbQQFGsxuV81G/KIb1B2yJ4FFItNGbS0tiVOpzUL/0F976mDfk8vXvS OdYB6yIgOX466mewaIzx5yKm4jBl1ITT0fI7iJLdGoaeh8IvBhQUKwyzRB0zFSV2R4VR Eu1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714994713; x=1715599513; h=message-id:in-reply-to:to:references:date:subject:mime-version:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6FDopxpQehU0k+rAnWTjWr+vUJUQDZ0CAWTTYqhX3MA=; b=D54F/K3kR0ylY3HRfMzbVW+MlJEUv0LgSs1R+BjhYcdD75ta4MuzY8PSlWOlBpZBgs c7OlJDEMNIyjzOJ6kM/7qh8J1pknWw+o3guLeRUEw/TdWpJWsR2IvDzC5WYUP9deVQc8 QRe8DhciPdaiJ+kdiw2fSukRKpSYFrUWkR3mJGYOJnyrjnQDnsB7cOXgnyU5L8fxidzS ejJDAgsRrdqq5yQ31x9+nUbfiIDq5KrYrA1LzVILHMM/DwPI+HlfJFJdkMIdkN64QXKb mALy9QnpOSj9ASq5pas5MTDdvn38ak9n44gptWm2cO4N9YLBHwE6L5bOJjw6ZQHANRPE gotA== X-Forwarded-Encrypted: i=1; AJvYcCXCfsdq+mMLdMJW3aH91nwVEBtoH7trZeGu+erigXPRP8HECg000Oj44UceIHHD+H1IcApgYhDbUY2Kjvo/sgGLo48JELXqZkIA3wXoc5I= X-Gm-Message-State: AOJu0YyEeUwAKbKzt9xOZqolAZ6oxCaqfBX3xt6z3jxPBHL3RvI/MUxI SS/wnyEa+oRPDeoQfQ2hKet87nR0V5xZyj1saBz9UOfbvuknXxYu X-Google-Smtp-Source: AGHT+IG5MlJZqjFD5g3D643ctPzOloPi/eMx5AdDnq0cRc0HnMH+92a6Rq2WbKMBO4PNB/ErZhZZOQ== X-Received: by 2002:a05:622a:246:b0:43a:ca31:8617 with SMTP id c6-20020a05622a024600b0043aca318617mr11560259qtx.44.1714994713070; Mon, 06 May 2024 04:25:13 -0700 (PDT) Received: from smtpclient.apple ([198.55.239.44]) by smtp.gmail.com with ESMTPSA id fh14-20020a05622a588e00b0043af8341b9dsm1735855qtb.95.2024.05.06.04.25.11 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 May 2024 04:25:12 -0700 (PDT) From: Rich Brown Content-Type: multipart/alternative; boundary="Apple-Mail=_8529389B-EF52-4C69-8C2B-1825AAD9611E" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.8\)) Date: Mon, 6 May 2024 07:25:10 -0400 References: <438B1BC4-D465-497A-B6BA-700E1D411036@ieee.org> <79C02ABB-B2A6-4B4D-98F4-6540D3F96EBB@ieee.org> <7E918B58-382A-4793-A144-13A7075CA56C@connectivitycap.com> <13rq2389-9012-p95n-s494-q3pp070s497n@ynat.uz> <6qop2p3o-351p-788q-q1q2-86sosnq3rn21@ynat.uz> <3FF32F52-4A93-496B-85FF-00020FA4A48B@gmx.de> <08F6942E-CC08-4956-B92E-CBEC091D86E4@ieee.org> To: Sebastian Moeller , Colin_Higbie , Dave Taht via Starlink , Eugene Y Chang In-Reply-To: <08F6942E-CC08-4956-B92E-CBEC091D86E4@ieee.org> Message-Id: X-Mailer: Apple Mail (2.3696.120.41.1.8) Subject: [Starlink] The "reasons" that bufferbloat isn't a problem X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 May 2024 11:25:14 -0000 --Apple-Mail=_8529389B-EF52-4C69-8C2B-1825AAD9611E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Gene, I've been vacillating on whether to send this note, but have decided to = pull the trigger. I apologize in advance for the "Debbie Downer" nature = of this message. I also apologize for any errors, omissions, or = over-simplifications of the "birth of bufferbloat" story and its fixes. = Corrections welcome. Rich ------ If we are going to take a shot at opening people's eyes to bufferbloat, = we should know some of the "objections" we'll run up against. Even = though there's terrific technical data to back it up, people seem = especially resistant to thinking that bufferbloat might affect their = network, even when they're seeing problems that sound exactly like = bufferbloat symptoms. But first, some history: The very idea of bufferbloat is simply unbelievable. Jim Gettys in 2011 = [1] couldn't believe it, and he's a smart networking guy,. At the time, = it seemed incredible (that is "not credible" =3D=3D impossible) that = something could induce 1.2 seconds of latency into his home network = connection. He called in favors from technical contacts at his ISP and = at Bell Labs who went over everything with a fine-toothed comb. It was = all exactly as spec'd. But he still had the latency.=20 This led Jim and Dave T=C3=A4ht to start the investigation into the = phenomenon known today as "bufferbloat" - the undesirable latency that = comes from a router or other network equipment buffering too much data. = Over several years, a group of smart people made huge improvements: = fq_codel was released 14 May 2012 [3]; it was incorporated into the = Linux kernel shortly afterward. CAKE came in 2015, and the fixes that = minimize bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate [4] = arrived to handle varying speed ISP links. All these techniques work = great: in 2014, my 7mbps DSL link was quite usable. And when the = pandemic hit, fq_codel on my OpenWrt router allowed me to use that same = 7mbps DSL line for two simultaneous zoom calls.=20 As one of the authors of [2], I am part of the team that has tried over = the years to explain bufferbloat and how to fix it. We've spoken with = vendors. We've spent untold hours responding to posts on assorted boards = and forums with the the bufferbloat story.=20 With these technical fixes in hand, we cockily set about to tell the = world about how to fix bufferbloat. Our efforts have been met with = skepticism at best, or stony silence. What are the objections?=20 - This is just the ordinary behavior: I would expect things to be slower = when there's more traffic (Willfully ignoring orders of magnitude = increase in delay.) - Besides, I'm the only one using the internet. (Except when my phone = uploads photos. Or my computer kicks off some automated process. Or I = browse the web. Or ...) - It only happens some of the time. (Exactly. That's probably when = something's uploading photos, or your computer is doing stuff in the = background.) - Those bufferbloat tests you hear about are bogus. They artificially = add load, which isn't a realistic test. (...and if you actually are = downloading a file?) - Bufferbloat only happens when the network is 100% loaded. (True. But = when you open a web page, your browser briefly uses 100% of the link. Is = this enough to cause momentary lag?) - It's OK. I just tell my kids/spouse not to use the internet when I'm = gaming. (Huh?) - I have gigabit service from my ISP. (That helps, but if you're = complaining about "slowness" you still need to rule out bufferbloat in = your router.) - I can't believe that router manufacturers would ever allow such a = thing to happen in their gear. (See the Jim Gettys story above.) - I mean... wouldn't router vendors want to provide the best for their = customers? (No - implementing this (new-ish) code requires engineering = effort. They're selling plenty of routers with decade-old software. The = Boss says, "would we sell more if they made these changes? Probably = not.") - Why would my ISP provision/sell me a router that gave crappy service? = They're a big company, they must know about this stuff. (Maybe. We have = reached out to all the vendors. But remember they profit if you decide = your network is too slow and you upgrade to a faster device/plan.) - But couldn't I just tweak the QoS on my router? (Maybe. But see [5]) - Besides, I just spent $300 on a "gaming router". Obviously, I bought = the most expensive/best possible solution on the market (But I still = have lag...) - You're telling me that a bunch of pointy-headed academics are smarter = than commercial router developers - who sold me that $300 router? (I = can't believe it.) - And then you say that I should throw away that gaming router and = install some "open source firmware"? (What the heck is that? And why = should I believe you?)=20 - What if it doesn't solve the problem? Who will give me support? And = how will I get back to a vendor-supported system? (Valid point - the = first valid point) - Aren't there any commercial solutions I can just buy? (Not at the = moment. IQrouter was a shining light here - available from Amazon, = simple setup, worked a treat - but they have gone out of business. And = of course, for the skeptic, this is proof that the "fq_codel-stuff" = isn't really a solution - it seems just like snake oil.) So... All these hurdles make it hard to convince people that bufferbloat = could be the problem, or that they can fix for themselves. A couple of us have reached out to Consumer Reports, wondering if they = would like a story about how vendors would prefer to sell you a new, = faster router (or new faster ISP plan) than fix your bufferbloat. This = kind of story seemed to be straight up their alley, but we never heard = back after an initial contact. Maybe they deserve another call... The recent latency results from Starlink give me a modicum of hope. = They're a major player. They (and their customers) can point to an order = of magnitude reduction in latency over other solutions. It still = requires enough "regular customers" to tell their current ISP that they = are switching to Starlink (and spend $600 to purchase a Dishy plus = $100/month) to provide a market incentive. Despite all this doom and gloom, I remain hopeful that things will get = better. We know the technology exists for people to take control of = their network and solve the problem for themselves. We can continue to = respond on forums where people express their dismay at the crummy = performance and suggest a solution. We can hope that a major vendor will = twig to this effect and bring out a mass-market solution. I think your suggestion of speaking to eSports people is intriguing. = They're highly motivated to make their personal networks better. And = actually solving the problem would have a network effect of bringing in = others with the same problem.=20 Good luck, and thanks for thinking about this. Rich Brown [1] = https://courses.cs.washington.edu/courses/cse550/21au/papers/bufferbloat.p= df [2] = https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_about_Buffer= bloat/ [3] = https://lists.bufferbloat.net/pipermail/cerowrt-devel/2012-May/000233.html= [4] https://github.com/lynxthecat/cake-autorate [5] = https://www.bufferbloat.net/projects/bloat/wiki/More_about_Bufferbloat/#wh= at-s-wrong-with-simply-configuring-qos > On May 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink = wrote: >=20 > Of course. For the gamers, the focus is managing latency. They have = control of everything else. >=20 > With our high latency and wide range of values, the eSports teams = train on campus. It will be interesting to see how much improvements = there can be for teams to be able to training from their homes. >=20 > Gene > ---------------------------------------------- > Eugene Chang > IEEE Life Senior Member > IEEE Communications Society & Signal Processing Society, =20 > Hawaii Chapter Chair > IEEE Life Member Affinity Group Hawaii Chair > IEEE Entrepreneurship, Mentor > eugene.chang@ieee.org > m 781-799-0233 (in Honolulu) --Apple-Mail=_8529389B-EF52-4C69-8C2B-1825AAD9611E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi Gene,

I've been vacillating on whether to send this = note, but have decided to pull the trigger. I apologize in advance for = the "Debbie Downer" nature of this message. I also apologize for any = errors, omissions, or over-simplifications of the "birth of bufferbloat" = story and its fixes. Corrections welcome.

Rich
------

If we are going to take a shot at opening people's eyes to = bufferbloat, we should know some of the "objections" we'll run up = against. Even though there's terrific technical data to back it up, = people seem especially resistant to thinking that bufferbloat might = affect their network, even when they're seeing problems that sound = exactly like bufferbloat symptoms. But first, some history:

The very idea of bufferbloat is simply = unbelievable. Jim Gettys in 2011 [1] couldn't believe it, and he's a = smart networking guy,. At the time, it seemed incredible (that is = "not credible" =3D=3D impossible) that something could induce 1.2 = seconds of latency into his home network connection. He called in favors = from technical contacts at his ISP and at Bell Labs who went over = everything with a fine-toothed comb. It was all exactly as spec'd. But = he still had the latency. 

This led = Jim and Dave T=C3=A4ht to start the investigation into the phenomenon = known today as "bufferbloat" - the undesirable latency that comes from a = router or other network equipment buffering too much data. Over = several years, a group of smart people made huge improvements: fq_codel = was released 14 May 2012 [3]; it was incorporated into the Linux = kernel shortly afterward. CAKE came in 2015, and the fixes that minimize = bufferbloat in Wi-Fi arrived in 2018. In 2021 cake-autorate [4] arrived = to handle varying speed ISP links. All these techniques work great: = in 2014, my 7mbps DSL link was quite usable. And when the pandemic hit, = fq_codel on my OpenWrt router allowed me to use that same 7mbps DSL = line for two simultaneous zoom calls. 

As one of the authors of [2], I am part = of the team that has tried over the years to explain bufferbloat and how = to fix it. We've spoken with vendors. We've spent untold hours = responding to posts on assorted boards and forums with the the = bufferbloat story. 

With these technical fixes in hand, we cockily set about to = tell the world about how to fix bufferbloat. Our efforts have been met = with skepticism at best, or stony silence. What are the = objections? 

- This is just the ordinary behavior: I would expect things = to be slower when there's more traffic (Willfully ignoring orders of = magnitude increase in delay.)
- Besides, I'm the = only one using the internet. (Except when my phone uploads photos. Or my = computer kicks off some automated process. Or I browse the web. Or = ...)
- It only happens some of the time. (Exactly. = That's probably when something's uploading photos, or your computer is = doing stuff in the background.)
- Those bufferbloat = tests you hear about are bogus. They artificially add load, which isn't = a realistic test. (...and if you actually are downloading a = file?)
- Bufferbloat only happens when the network = is 100% loaded. (True. But when you open a web page, your browser = briefly uses 100% of the link. Is this enough to cause momentary = lag?)
- It's OK. I just tell my kids/spouse not to = use the internet when I'm gaming. (Huh?)
- I have = gigabit service from my ISP. (That helps, but if you're complaining = about "slowness" you still need to rule out bufferbloat in your = router.)
- I can't believe that router = manufacturers would ever allow such a thing to happen in their gear. = (See the Jim Gettys story above.)
- I mean... = wouldn't router vendors want to provide the best for their customers? = (No - implementing this (new-ish) code requires engineering effort. = They're selling plenty of routers with decade-old software. The Boss = says, "would we sell more if they made these changes? Probably = not.")
- Why would my ISP provision/sell me a = router that gave crappy service? They're a big company, they must know = about this stuff. (Maybe. We have reached out to all the vendors. But = remember they profit if you decide your network is too slow and you = upgrade to a faster device/plan.)
- But couldn't I = just tweak the QoS on my router? (Maybe. But see [5])
- Besides, I just spent $300 on a "gaming router". Obviously, = I bought the most expensive/best possible solution on the market (But I = still have lag...)
- You're telling me that a bunch = of pointy-headed academics are smarter than commercial router developers = - who sold me that $300 router? (I can't believe it.)
- And then you say that I should throw away that gaming = router and install some "open source firmware"? (What the heck is that? = And why should I believe you?) 
- What if it = doesn't solve the problem? Who will give me support? And how will I get = back to a vendor-supported system? (Valid point - the first valid = point)
- Aren't there any commercial solutions I = can just buy? (Not at the moment. IQrouter was a shining light here - = available from Amazon, simple setup, worked a treat - but they have gone = out of business. And of course, for the skeptic, this is proof that the = "fq_codel-stuff" isn't really a solution - it seems just like snake = oil.)

So... = All these hurdles make it hard to convince people that bufferbloat could = be the problem, or that they can fix for themselves.

A couple of us have = reached out to Consumer Reports, wondering if they would like a story = about how vendors would prefer to sell you a new, faster router (or new = faster ISP plan) than fix your bufferbloat. This kind of story seemed to = be straight up their alley, but we never heard back after an initial = contact. Maybe they deserve another call...

The recent latency results from = Starlink give me a modicum of hope. They're a major player. They (and = their customers) can point to an order of magnitude reduction in latency = over other solutions. It still requires enough "regular customers" to = tell their current ISP that they are switching to Starlink (and spend = $600 to purchase a Dishy plus $100/month) to provide a market = incentive.

Despite all this doom and gloom, I remain hopeful that things = will get better. We know the technology exists for people to take = control of their network and solve the problem for themselves. We can = continue to respond on forums where people express their dismay at the = crummy performance and suggest a solution. We can hope that a major = vendor will twig to this effect and bring out a mass-market = solution.

I = think your suggestion of speaking to eSports people is intriguing. = They're highly motivated to make their personal networks better. And = actually solving the problem would have a network effect of bringing in = others with the same problem. 

Good luck, and thanks for thinking = about this.

Rich= Brown

[2] https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_do_a= bout_Bufferbloat/

On May = 1, 2024, at 6:19 PM, Eugene Y Chang via Starlink <starlink@lists.bufferbloat.net> wrote:

Of course. For the = gamers, the focus is managing latency. They have control of everything = else.

With our high = latency and wide range of values, the eSports teams train on campus. It = will be interesting to see how much improvements there can be for teams = to be able to training from their homes.

Gene
----------------------------------------------
Eugene Chang
IEEE Life Senior = Member
IEEE Communications Society & Signal = Processing Society,    
    Hawaii = Chapter Chair
IEEE Life Member Affinity Group = Hawaii Chair
IEEE Entrepreneurship, Mentor
eugene.chang@ieee.org
m 781-799-0233 (in = Honolulu)

= --Apple-Mail=_8529389B-EF52-4C69-8C2B-1825AAD9611E--