From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2F1703B29E for ; Mon, 26 Sep 2022 17:20:47 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1664227243; bh=MjrVucDLSBACNKjt4H3oiU565qEOvrgXdS1MurSDtO4=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=SjCCb0wPEFFAHFTElb+NcsXU+Dm7a/oVS/8VWmzN3+m6qHDwqx/jD6fqZ5kkmhlFE xBoBJ31gsgOejauHPyUxkk0F+0p3t9aLflJ3p9jItsUS/rb0qr+UHv614vcCUIVVAZ pQ4Y7dIuB3d4yIkH7bwG/bW6lX3fgENA0o5WpjbI= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([77.8.198.40]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MtwZ4-1pXEXf0OlA-00uMO4; Mon, 26 Sep 2022 23:20:43 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> Date: Mon, 26 Sep 2022 23:20:41 +0200 Cc: David Lang , Dave Taht via Starlink Content-Transfer-Encoding: quoted-printable Message-Id: <9D97FB02-58A5-48B0-9B43-6B7BD2A24099@gmx.de> References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> To: Eugene Y Chang X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:fK69CGDjCpGjW9rPovY933GHgBWLXcxpm5ziFkQMfpXfpd1W9gj MMAsIDtn75d5EGpk/P/vzI0JmfeqyDlEQNhCXYb8q6006lY7sT7i4AbcKUZUeapxBX5Xl29 3tRuiZARfp5swfFC1fQtgWQ4fXynDGphrzXDyk36PE4WrEB5AzQw3/Q2rgq5QcwXuC+42IS /E3Qmb2JLId5zD4jNsWIQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:6W7w+OtJv5M=:x+esZaUkVeVvnAuEHtMgj5 q6/pnQ7PIPAGOKqoPpOUC3Hyf5uFzYpiyGUd0gV3scPiKYNsGMQeKzWinbj8zb71vpOkMOmUV e7TAvoBD1L6fMG0AIyRP1ZgWpwNjwgdXR8PJszzV7oiZoXewPiDSbuRuMut+XsGo6FmA5ow15 zRgaxOIL5aJvbc8bXISljXiIPtRnIUy2/w+d+Wrhj1/aiZRZv0P9LaulOrgCdRzE2AP7R/Rsf diHjlsHwbAM0r5oxWPFOxfeLLWMMT0k5ru47OrTBmvhbiO5Npd3zFDv9XARGuFyjdQgLH1Xxj Ly6dj225hX3vqnnPQY8pzzIxRMmK+YZO/m1HEFPSkfddIupGg72DvXxPJxaa9cTAYs8LMw4Cv FsFwYjBa9Or4WuHITTiqG2YuX3F4FtqiHZrUDcb+EuaVfLy1T+ZMfMOU8XllBbSsS8apH27hI 9wwQbTi44yD5e+45RAXuKiB+v31MR3bmsWcfkq3fCOcJLPSI+KCfMSZYF0hWAbFqj4/z78rl3 +Ot32j9WfWu9mLBZuOdPZtbz817SgMgtdJXe+DxnR+rs9ll78jhq0Ond4YvdQZspv1KjUhzsV NwXHrZmbp5YZFQhNrAs8YuUzcd3Dp4Fsre1nzJUA+l8qaqJEK5Ymsw2yDzQTdiSJ04k59K+EB YXwfiluey45RP+I4dxHkpDdZONQejG4fySYeVMrjlDQrHcx8vG8ZgW2VzMWhIb1iBqo6Gb4Fm 7ISiko9mSQzJrXvQ9+jlGXmR294y/I8gf9tNo73G/7CD3CA92mHCFEadHLlwfuKrVP4I9M26k J+EdQD0vVacMC1bMKPySgfHw0M5AkdPSN12NGwPUMRk4v5LclMufuFDmJYSiAZLrUCCYeg1aO 2byGHFUf51ZzTQ+uP4+E7u76pVDEtOhli/pySYdi7Q0NNAjXphyOSyI3BuRRpKMEXmWYReQ30 7LILCYkLNqylbilYXaVMwffftV5gJy90+cv38lWc+4jr+o169ySfCr4w0CYIBhGF6fmGt7s6k k5y7OWtgFbovKgtmWJ97TL4b4U9NaOHX3RORvysUzHlgtEJ18offsgW6VJBeJ9fVOqDOMeIcA frUBPbMS3TLZ7PNMxx2Uh+PKiokV65/eOC9k1Iq0ydrCILmaoc1GA4rflKQyHkZNChEKYJpas 3wDClP553vlhKj9puh7auKs1uG Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Sep 2022 21:20:47 -0000 Hi Gene, > On Sep 26, 2022, at 23:10, Eugene Y Chang = wrote: >=20 > Comments inline below. >=20 > Gene > ---------------------------------------------- > Eugene Chang > IEEE Senior Life Member > eugene.chang@ieee.org > 781-799-0233 (in Honolulu) >=20 >=20 >=20 >> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>=20 >> Hi Eugene, >>=20 >>=20 >>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = wrote: >>>=20 >>> Ok, we are getting into the details. I agree. >>>=20 >>> Every node in the path has to implement this to be effective. >>=20 >> Amazingly the biggest bang for the buck is gotten by fixing = those nodes that actually contain a network path's bottleneck. Often = these are pretty stable. So yes for fully guaranteed service quality all = nodes would need to participate, but for improving things noticeably it = is sufficient to improve the usual bottlenecks, e.g. for many internet = access links the home gateway is a decent point to implement better = buffer management. (In short the problem are over-sized and = under-managed buffers, and one of the best solution is better/smarter = buffer management). >>=20 >=20 > This is not completely true. [SM] You are likely right, trying to summarize things leads to = partially incorrect generalizations. > Say the bottleneck is at node N. During the period of congestion, the = upstream node N-1 will have to buffer. When node N recovers, the = bufferbloat at N-1 will be blocking until the bufferbloat drains. Etc. = etc. Making node N better will reduce the extent of the backup at N-1, = but N-1 should implement the better code. [SM] It is the node that builds up the queue that profits most = from better queue management.... (again I generalize, the node with the = queue itself probably does not care all that much, but the endpoints = will profit if the queue experiencing node deals with that queue more = gracefully). >=20 >=20 >>=20 >>> In fact, every node in the path has to have the same prioritization = or the scheme becomes ineffective. >>=20 >> Yes and no, one of the clearest winners has been flow queueing, = IMHO not because it is the most optimal capacity sharing scheme, but = because it is the least pessimal scheme, allowing all (or none) flows = forward progress. You can interpret that as a scheme in which flows = below their capacity share are prioritized, but I am not sure that is = the best way to look at these things. >=20 > The hardest part is getting competing ISPs to implement and = coordinate.=20 [SM] Yes, but it turned out even with non-cooperating ISPs there = is a lot end-users can do unilaterally on their side to improve both = ingress and egress congestion. Admittedly especially ingress congestion = would be even better handled with cooperation of the ISP. > Bufferbloat and handoff between ISPs will be hard. The only way to fix = this is to get the unwashed public to care. Then they can say =E2=80=9Cwe = don=E2=80=99t care about the technical issues, just fix it.=E2=80=9D = Until then =E2=80=A6.. [SM] Well we do this one home network at a time (not because = that is efficient or ideal, but simply because it is possible). Maybe, = if you have not done so already try OpenWrt with sqm-scripts (and maybe = cake-autorate in addition) on your home internet access link for say a = week and let us know ih/how your experience changed? Regards Sebastian >=20 >=20 >=20 >>=20 >> Regards >> Sebastian >>=20 >>=20 >>>=20 >>> Gene >>> ---------------------------------------------- >>> Eugene Chang >>> IEEE Senior Life Member >>> eugene.chang@ieee.org >>> 781-799-0233 (in Honolulu) >>>=20 >>>=20 >>>=20 >>>> On Sep 26, 2022, at 10:48 AM, David Lang wrote: >>>>=20 >>>> software updates can do far more than just improve recovery. >>>>=20 >>>> In practice, large data transfers are less sensitive to latency = than smaller data transfers (i.e. downloading a CD image vs a video = conference), software can ensure better fairness in preventing a bulk = transfer from hurting the more latency sensitive transfers. >>>>=20 >>>> (the example below is not completely accurate, but I think it gets = the point across) >>>>=20 >>>> When buffers become excessivly large, you have the situation where = a video call is going to generate a small amount of data at a regular = interval, but a bulk data transfer is able to dump a huge amount of data = into the buffer instantly. >>>>=20 >>>> If you just do FIFO, then you get a small chunk of video call, then = several seconds worth of CD transfer, followed by the next small chunk = of the video call. >>>>=20 >>>> But the software can prevent the one app from hogging so much of = the connection and let the chunk of video call in sooner, avoiding the = impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>=20 >>>> The one thing that Cake needs to work really well is to be able to = know what the data rate available is. With Starlink, this changes = frequently and cake integrated into the starlink dish/router software = would be far better than anything that can be done externally as the = rate changes can be fed directly into the settings (currently they are = only indirectly detected) >>>>=20 >>>> David Lang >>>>=20 >>>>=20 >>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>=20 >>>>> You already know this. Bufferbloat is a symptom and not the cause. = Bufferbloat grows when there are (1) periods of low or no bandwidth or = (2) periods of insufficient bandwidth (aka network congestion). >>>>>=20 >>>>> If I understand this correctly, just a software update cannot make = bufferbloat go away. It might improve the speed of recovery (e.g. throw = away all time sensitive UDP messages). >>>>>=20 >>>>> Gene >>>>> ---------------------------------------------- >>>>> Eugene Chang >>>>> IEEE Senior Life Member >>>>> eugene.chang@ieee.org >>>>> 781-799-0233 (in Honolulu) >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>=20 >>>>>> Please help to explain. Here's a draft to start with: >>>>>>=20 >>>>>> Starlink Performance Not Sufficient for Military Applications, = Say Scientists >>>>>>=20 >>>>>> The problem is not availability: Starlink works where nothing but = another satellite network would. It's not bandwidth, although others = have questions about sustaining bandwidth as the customer base grows. = It's latency and jitter. As load increases, latency, the time it takes = for a packet to get through, increases more than it should. The = scientists who have fought bufferbloat, a major cause of latency on the = internet, know why. SpaceX needs to upgrade their system to use the = scientist's Open Source modifications to Linux to fight bufferbloat, and = thus reduce latency. This is mostly just using a newer version, but = there are some tunable parameters. Jitter is a change in the speed of = getting a packet through the network during a connection, which is = inevitable in satellite networks, but will be improved by making use of = the bufferbloat-fighting software, and probably with the addition of = more satellites. >>>>>>=20 >>>>>> We've done all of the work, SpaceX just needs to adopt it by = upgrading their software, said scientist Dave Taht. Jim Gettys, Taht's = collaborator and creator of the X Window System, chimed in: >>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's = latency and jitter make it inadequate to remote-control my ham radio = station. But the military is experimenting with remote-control of = vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>=20 >>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>> The key issue is most people don=E2=80=99t understand why latency = matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>=20 >>>>>> First, we have to help people see the symptoms of latency and how = it impacts something they care about. >>>>>> - gamers care but most people may think it is frivolous. >>>>>> - musicians care but that is mostly for a hobby. >>>>>> - business should care because of productivity but they don=E2=80=99= t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>=20 >>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing the = action of latency all this time and never knew it! I was being = shafted.=E2=80=9D Once you have this awakening, you can get all the = press you want for free. >>>>>>=20 >>>>>> Most of the time when business apps are developed, =E2=80=9Cwe=E2=80= =9D hide the impact of poor performance (aka latency) or they hide from = the discussion because the developers don=E2=80=99t have a way to fix = the latency. Maybe businesses don=E2=80=99t care because any employees = affected are just considered poor performers. (In bad economic times, = the poor performers are just laid off.) For employees, if they happen to = be at a location with bad latency, they don=E2=80=99t know that latency = is hurting them. Unfair but most people don=E2=80=99t know the issue is = latency. >>>>>>=20 >>>>>> Talking and explaining why latency is bad is not as effective as = showing why latency is bad. Showing has to be with something that has a = person impact. >>>>>>=20 >>>>>> Gene >>>>>> ----------------------------------- >>>>>> Eugene Chang >>>>>> eugene.chang@alum.mit.edu >>>>>> +1-781-799-0233 (in Honolulu) >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>=20 >>>>>>> If you want to get attention, you can get it for free. I can = place articles with various press if there is something interesting to = say. Did this all through the evangelism of Open Source. All we need to = do is write, sign, and publish a statement. What they actually write is = less relevant if they publish a link to our statement. >>>>>>>=20 >>>>>>> Right now I am concerned that the Starlink latency and jitter is = going to be a problem even for remote controlling my ham station. The US = Military is interested in doing much more, which they have demonstrated, = but I don't see happening at scale without some technical work on the = network. Being able to say this isn't ready for the government's = application would be an attention-getter. >>>>>>>=20 >>>>>>> Thanks >>>>>>>=20 >>>>>>> Bruce >>>>>>>=20 >>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>> These days, if you want attention, you gotta buy it. A 50k half = page >>>>>>> ad in the wapo or NYT riffing off of It's the latency, Stupid!", >>>>>>> signed by the kinds of luminaries we got for the fcc wifi fight, = would >>>>>>> go a long way towards shifting the tide. >>>>>>>=20 >>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht > wrote: >>>>>>>>=20 >>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>> > wrote: >>>>>>>>>=20 >>>>>>>>> The awareness & understanding of latency & impact on QoE is = nearly unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise reporters will = just continue to focus on what they know... >>>>>>>>=20 >>>>>>>> That's a great idea. I have visions of crashing the washington >>>>>>>> correspondents dinner, but perhaps >>>>>>>> there is some set of gatherings journalists regularly attend? >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave Taht = via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>=20 >>>>>>>>> I still find it remarkable that reporters are still missing = the >>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> -- >>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> -- >>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>> _______________________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net = >>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>=20 >>>>>>>=20 >>>>>>> -- >>>>>>> Bruce Perens K6BP >>>>>>> _______________________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net = >>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> -- >>>>>> Bruce Perens K6BP >>>=20 >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/starlink >=20