From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 0323B3B29E for ; Mon, 26 Sep 2022 17:29:27 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1664227764; bh=/j3FC2mDcJRUm8RyRg+uTTWzUy9YCIz0CoC/Fro5hXw=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=QGn8pixxda3QTO2hoKK5KtHgb6eKOowDJBJ69pHwiHPnG8m6EL0+i1eu6IGbDsGto Dj+M0aRXW9H24PutdIEzvYiVpu/USi37eRaDj6zQMOPVycD2531V2YzkVK8LJuozOA iUgxXyMG7U954fSwZFz2+2XNFm63b2No/IHA8iKU= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([77.8.198.40]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mzhj9-1pPmxJ2EyX-00vind; Mon, 26 Sep 2022 23:29:24 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Mon, 26 Sep 2022 23:29:23 +0200 Cc: Eugene Y Chang , Dave Taht via Starlink Content-Transfer-Encoding: quoted-printable Message-Id: <2086C010-B91E-450A-A77F-D0840BC5FCC1@gmx.de> References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> To: David Lang X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:hWJDzAfdZXr+04v1DgW0hFnfQ8Wt7oIc0U2ocwIrDkXwdxiTtoB LrGiigew/GPSuOwbY1bT2/WakxVbwJFf4ZI6qUwWwIjCQh1NxDEiLSB7G0lMYrVYh0mwge5 3VX7gD2PmnuBECI/eXpWSxyf4b37ccvWmdTnbwHpyc36Ff2fMOspOt4fbDHViByDHGqrmOq m+bk/GKasYG3v9+Owz53g== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:70WIs5MeISc=:o8jU/imYh45+H7Z0UMfsxJ RozwhxqmUaaEufgoMrH4ZHk9TzO56zlfE/qsITBDce9CY59sGATZS0tRs1v4mqObcX/MY673p 8izA6RY4xUzwV4+Gz0mA0cqEE9m1ti9GtnN5NJwXDAxcVNA7BXZPmOHb+7S/6z+GL4vIi3i77 EWOvBST+mXONexTZSI+Ax2h/gp9xYZ5RoB+ONYJtNW1nTaHog9ENT7YecO2/bm7at/8MPt97n D+rv6drki8Nrzhm0YRbnR4JRuB4VNnBNG+qhQ2XuPTyW84OV6oWnETiYvHHbDvyxJg887+E1o dAx15z4Dy/uKnnlk5hd2B7W4AJrVq2jDBcmwjUeZi17zYLOM/IRYpgmBo2RuP79RAlmriyDUX kho143jNxFE5Dy1rmV1IIMLkPkcVYaw9hy1bjOm7QiZdpPB696MjxMf/UrcWyiCDIFBWqID4w b5zG9gTmUhBYT2JZ+1U2eNmaVrOtZtLRySCNWudXacNo0AQk3AhYuaU0uXyYds1q5225NzTS+ n3VAqXbbu3B/6mGM8cWXsMp3aHoucGB2uwp8fRS2x2ObHpECmGWXJNgM3od/m9nBdoaWFCsKL eBjcbaxtmdOOCJEH3ptaivR5zTW5fXguCk5HNE+Dc+KBrYcYm/Fa3CPgBcA0a0ynFoqivgO/r Tphi+IACKvk9mbHKERCEEt1IV26IgjPWFVtrVuk2VXS3vpng6nzEhVTTUB0fbm+UoeluHl9me d8X+s18FQN1Fld+PMceUhHpgqEaQgIdJXDbcrBUI/0mO5xdpujf+W9J/rzkwVcCKUMRotpK0y 0Cch8z4niBqUI+CqpmyHqBXp/FsRJx4ZBwX0n4B7d3cFTBb2lepjq9yJHkxuuytDtC12D+TKC o+SJXC76pNsycGvt6Wq5WaI5tZtF58C5dimIYH+hLj4quQ/jAnXDxbiNEzeEin2kzOUliG8+d sYnVoy2F19bnr5jgtv4cD9pv0Hil0p/i7yfb7EjKBq+do/rVdxPI4o/agXDBTY2I9nAQxGPW1 hi2j3hZLqIDsw2TVOOWVvIdOBCSmitv87q2UzmTGWOE1Q8uBdFpyfbDUviyqVz4mn+O9wICyF ArcQtp2U3djlwDEa1H5XeFDegwYPu1tFu2piYKsvbeduurAwce3zdXk2WbFu2gdwBrHqcsglT Fvs85sfMU7+UyGxKsJ9Sp6JWRm Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Sep 2022 21:29:28 -0000 Hi David, > On Sep 26, 2022, at 23:22, David Lang wrote: >=20 > On Mon, 26 Sep 2022, Eugene Y Chang wrote: >=20 >>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>>=20 >>> Hi Eugene, >>>=20 >>>=20 >>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = > = wrote: >>>>=20 >>>> Ok, we are getting into the details. I agree. >>>>=20 >>>> Every node in the path has to implement this to be effective. >>>=20 >>> Amazingly the biggest bang for the buck is gotten by fixing = those nodes that actually contain a network path's bottleneck. Often = these are pretty stable. So yes for fully guaranteed service quality all = nodes would need to participate, but for improving things noticeably it = is sufficient to improve the usual bottlenecks, e.g. for many internet = access links the home gateway is a decent point to implement better = buffer management. (In short the problem are over-sized and = under-managed buffers, and one of the best solution is better/smarter = buffer management). >>>=20 >>=20 >> This is not completely true. Say the bottleneck is at node N. During = the period of congestion, the upstream node N-1 will have to buffer. = When node N recovers, the bufferbloat at N-1 will be blocking until the = bufferbloat drains. Etc. etc. Making node N better will reduce the = extent of the backup at N-1, but N-1 should implement the better code. >=20 > only if node N and node N-1 handle the same traffic with the same link = speeds. In practice this is almost never the case. [SM] I note that typically for ingress shaping a = post-true-bottleneck shaper will not work unless we create an artificial = bottleneck by shaping the traffic to below true bottleneck (thereby = creating a new true but artificial bottleneck so the queue develops at a = point where we can control it). Also if the difference between "true structural" and artificial = bottleneck is small in comparison to the traffic inrush we can see = "traffic back-spill" into the typically oversized and under-managed = upstream buffers, but for reasonably well behaved that happens = relatively rarely. Rarely enough that ingress traffic shaping noticeably = improves latency-under-load in spite of not beeing a guranteed solution. > Until you get to gigabit last-mile links, the last mile is almost = always the bottleneck from both sides, so implementing cake on the home = router makes a huge improvement (and if you can get it on the last-mile = ISP router, even better). Once you get into the Internet fabric, = bottlenecks are fairly rare, they do happen, but ISPs carefully watch = for those and add additional paths and/or increase bandwith to avoid = them. [SM] Well, sometimes such links are congested too for economic = reasons... Regards Sebastian >=20 > David Lang >=20 >>>=20 >>>> In fact, every node in the path has to have the same prioritization = or the scheme becomes ineffective. >>>=20 >>> Yes and no, one of the clearest winners has been flow queueing, = IMHO not because it is the most optimal capacity sharing scheme, but = because it is the least pessimal scheme, allowing all (or none) flows = forward progress. You can interpret that as a scheme in which flows = below their capacity share are prioritized, but I am not sure that is = the best way to look at these things. >>=20 >> The hardest part is getting competing ISPs to implement and = coordinate. Bufferbloat and handoff between ISPs will be hard. The only = way to fix this is to get the unwashed public to care. Then they can say = =E2=80=9Cwe don=E2=80=99t care about the technical issues, just fix = it.=E2=80=9D Until then =E2=80=A6.. >>=20 >>=20 >>=20 >>>=20 >>> Regards >>> Sebastian >>>=20 >>>=20 >>>>=20 >>>> Gene >>>> ---------------------------------------------- >>>> Eugene Chang >>>> IEEE Senior Life Member >>>> eugene.chang@ieee.org >>>> 781-799-0233 (in Honolulu) >>>>=20 >>>>=20 >>>>=20 >>>>> On Sep 26, 2022, at 10:48 AM, David Lang wrote: >>>>>=20 >>>>> software updates can do far more than just improve recovery. >>>>>=20 >>>>> In practice, large data transfers are less sensitive to latency = than smaller data transfers (i.e. downloading a CD image vs a video = conference), software can ensure better fairness in preventing a bulk = transfer from hurting the more latency sensitive transfers. >>>>>=20 >>>>> (the example below is not completely accurate, but I think it gets = the point across) >>>>>=20 >>>>> When buffers become excessivly large, you have the situation where = a video call is going to generate a small amount of data at a regular = interval, but a bulk data transfer is able to dump a huge amount of data = into the buffer instantly. >>>>>=20 >>>>> If you just do FIFO, then you get a small chunk of video call, = then several seconds worth of CD transfer, followed by the next small = chunk of the video call. >>>>>=20 >>>>> But the software can prevent the one app from hogging so much of = the connection and let the chunk of video call in sooner, avoiding the = impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>>=20 >>>>> The one thing that Cake needs to work really well is to be able to = know what the data rate available is. With Starlink, this changes = frequently and cake integrated into the starlink dish/router software = would be far better than anything that can be done externally as the = rate changes can be fed directly into the settings (currently they are = only indirectly detected) >>>>>=20 >>>>> David Lang >>>>>=20 >>>>>=20 >>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>>=20 >>>>>> You already know this. Bufferbloat is a symptom and not the = cause. Bufferbloat grows when there are (1) periods of low or no = bandwidth or (2) periods of insufficient bandwidth (aka network = congestion). >>>>>>=20 >>>>>> If I understand this correctly, just a software update cannot = make bufferbloat go away. It might improve the speed of recovery (e.g. = throw away all time sensitive UDP messages). >>>>>>=20 >>>>>> Gene >>>>>> ---------------------------------------------- >>>>>> Eugene Chang >>>>>> IEEE Senior Life Member >>>>>> eugene.chang@ieee.org >>>>>> 781-799-0233 (in Honolulu) >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>>=20 >>>>>>> Please help to explain. Here's a draft to start with: >>>>>>>=20 >>>>>>> Starlink Performance Not Sufficient for Military Applications, = Say Scientists >>>>>>>=20 >>>>>>> The problem is not availability: Starlink works where nothing = but another satellite network would. It's not bandwidth, although others = have questions about sustaining bandwidth as the customer base grows. = It's latency and jitter. As load increases, latency, the time it takes = for a packet to get through, increases more than it should. The = scientists who have fought bufferbloat, a major cause of latency on the = internet, know why. SpaceX needs to upgrade their system to use the = scientist's Open Source modifications to Linux to fight bufferbloat, and = thus reduce latency. This is mostly just using a newer version, but = there are some tunable parameters. Jitter is a change in the speed of = getting a packet through the network during a connection, which is = inevitable in satellite networks, but will be improved by making use of = the bufferbloat-fighting software, and probably with the addition of = more satellites. >>>>>>>=20 >>>>>>> We've done all of the work, SpaceX just needs to adopt it by = upgrading their software, said scientist Dave Taht. Jim Gettys, Taht's = collaborator and creator of the X Window System, chimed in: >>>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's = latency and jitter make it inadequate to remote-control my ham radio = station. But the military is experimenting with remote-control of = vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>>=20 >>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>>> The key issue is most people don=E2=80=99t understand why = latency matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>>=20 >>>>>>> First, we have to help people see the symptoms of latency and = how it impacts something they care about. >>>>>>> - gamers care but most people may think it is frivolous. >>>>>>> - musicians care but that is mostly for a hobby. >>>>>>> - business should care because of productivity but they don=E2=80=99= t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>>=20 >>>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing the = action of latency all this time and never knew it! I was being = shafted.=E2=80=9D Once you have this awakening, you can get all the = press you want for free. >>>>>>>=20 >>>>>>> Most of the time when business apps are developed, =E2=80=9Cwe=E2=80= =9D hide the impact of poor performance (aka latency) or they hide from = the discussion because the developers don=E2=80=99t have a way to fix = the latency. Maybe businesses don=E2=80=99t care because any employees = affected are just considered poor performers. (In bad economic times, = the poor performers are just laid off.) For employees, if they happen to = be at a location with bad latency, they don=E2=80=99t know that latency = is hurting them. Unfair but most people don=E2=80=99t know the issue is = latency. >>>>>>>=20 >>>>>>> Talking and explaining why latency is bad is not as effective as = showing why latency is bad. Showing has to be with something that has a = person impact. >>>>>>>=20 >>>>>>> Gene >>>>>>> ----------------------------------- >>>>>>> Eugene Chang >>>>>>> eugene.chang@alum.mit.edu >>>>>>> +1-781-799-0233 (in Honolulu) >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>>=20 >>>>>>>> If you want to get attention, you can get it for free. I can = place articles with various press if there is something interesting to = say. Did this all through the evangelism of Open Source. All we need to = do is write, sign, and publish a statement. What they actually write is = less relevant if they publish a link to our statement. >>>>>>>>=20 >>>>>>>> Right now I am concerned that the Starlink latency and jitter = is going to be a problem even for remote controlling my ham station. The = US Military is interested in doing much more, which they have = demonstrated, but I don't see happening at scale without some technical = work on the network. Being able to say this isn't ready for the = government's application would be an attention-getter. >>>>>>>>=20 >>>>>>>> Thanks >>>>>>>>=20 >>>>>>>> Bruce >>>>>>>>=20 >>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>>> These days, if you want attention, you gotta buy it. A 50k half = page >>>>>>>> ad in the wapo or NYT riffing off of It's the latency, = Stupid!", >>>>>>>> signed by the kinds of luminaries we got for the fcc wifi = fight, would >>>>>>>> go a long way towards shifting the tide. >>>>>>>>=20 >>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht > wrote: >>>>>>>>>=20 >>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>>> > wrote: >>>>>>>>>>=20 >>>>>>>>>> The awareness & understanding of latency & impact on QoE is = nearly unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise reporters will = just continue to focus on what they know... >>>>>>>>>=20 >>>>>>>>> That's a great idea. I have visions of crashing the washington >>>>>>>>> correspondents dinner, but perhaps >>>>>>>>> there is some set of gatherings journalists regularly attend? >>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave Taht = via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>>=20 >>>>>>>>>> I still find it remarkable that reporters are still missing = the >>>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> -- >>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> -- >>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>> _______________________________________________ >>>>>>>> Starlink mailing list >>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> -- >>>>>>>> Bruce Perens K6BP >>>>>>>> _______________________________________________ >>>>>>>> Starlink mailing list >>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> -- >>>>>>> Bruce Perens K6BP >>>>=20 >>>> _______________________________________________ >>>> Starlink mailing list >>>> Starlink@lists.bufferbloat.net = >>>> https://lists.bufferbloat.net/listinfo/starlink =