From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2F5983CB37 for ; Tue, 27 Sep 2022 03:09:55 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1664262593; bh=6SWuYz4eWkdyetRgD2neyrSOGNYmTY9X0qRxsGDceQ8=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=F3LurF8Rx+f3Mn8gZ3xlUKVazE0nxIqowacpd7Xuy0wZe4cHw7GZbGxedye4EnwBh Ck1ht0WmKggCpHa/qpA8d8gHbs8CS5QniTPDEF0o7Uc2pbRSDoCfEx+RDaLnSgW6x0 IfC211doF05Jx7ZDt5+Ivv8PogixsSXAiOs1acJ0= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MF3He-1oS4Fd1Vuq-00FOr2; Tue, 27 Sep 2022 09:09:53 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Tue, 27 Sep 2022 09:09:49 +0200 Cc: Bruce Perens , Dave Taht via Starlink Content-Transfer-Encoding: quoted-printable Message-Id: <3FD7CB2A-8DC8-47FD-82DF-D27B967CBF4F@gmx.de> References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> <9D97FB02-58A5-48B0-9B43-6B7BD2A24099@gmx.de> To: Eugene Y Chang X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:juEatGk+onopDMBh9TBaUXLgS8WUgKtXwOdGt60LT2BLwP/mkJx X3Ok1JwKoQDJ0tDzV0pJFKwTyKmmuYzi/dCj5NfuDUJNqtu5APZNF0iUGaD1QR4ebJNd96q eIz73jjEHxK3EKoTBpmNi3T2nrzRxFImcMHQGJZrtNzqLx3s9pTtYVLrW/d8GCMVfx+5ObY +Ey9ubRBWPkvaVBriETTg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:aOW1o4495w4=:4xt+KY53R1EMhpa+y314vE m06UrA2lKj0MEQVDnRdzjtMxM4Rlp9FURWR9McoI1UuOz/2lgheRSl4iwWRbEMntry9vSwSg3 LBEXOuH2xbhyBUGn8MXU7rUl/YzGuKDE1U5RiTW7jp+NfM0ZWQ7CdHPpmchMFWvvtlRYYvd5w G489LweSFCIfWPlps/+sN1+rfK99gLmtDYInYze6/UPRYbZFVE0BEUY7ifyNnyuFAKTFg9myg CSMsYiJeJNh2BS1In7vOnY/fhctF+0DoMvftbWgFrKaQSixmTTAni5MlCQxxLdroem0q9YoC2 i42q7iWLFH1bpvSPx78ZlhxuuKbhodXlGZ2Xui87SRW8As9FE4B1jES9NjOUes7MyTJG+YH4W jtwgz2rlqmKy4kEC9Ij2gHCzGYotO90ru8MvzgEG0HOXIBP0Dwdix64cBDObDd9zMvsTBQHYg gpjx7LSADbkgMDAQqr4+6rAoCwioIVCplqVduTj4+e0k1/Z9BGGwcIN1lQNWBlfcHou49evjF Vc9bdX/JDSxXbGnkSLXMVRofigyBKhwmOsa3ffH/4U2HvAGYjsxNDcBcZ19brivXlV/9fYyMB +r4MEHMph2FgXFCjQmDR4mvQUxPnoaNOItd+QXoS+OIGx9c3e87VRS49eYd9H0ngGhDe5ORIG JnqgwBZMY49DCUXhsrKBd8w7UpDU+e/DzcIp3+L7LZ20Jp1k/mlYbrB9/jgxh1bF8h8+eLEQi 2d5nfFmGCkca5WFUwUFKiT4og7Fsd/xRFej9FborftnlBlbjWhdF6+HweF6KMQlrHS8PjlBc2 Iyu6Khw/Aam1YnXZQkq4QJtYS70f4LBSapmtsIU8S9JNUOS4IDqBaeu2i098OZ5/xALgrFkgV ApqoecxRiC5zO65w5PWpa2gAMlmeSHntN+rj54YxpEKXgLdFDPBlHF0iln1rHOAX9Tiw6dvBv zvLTTtphSrseoauuhZTqu1K2X/8AhasBvIPQEP2ZMSE5TP4ggBmPfT6g0O95zHbNlXQVYalql +Yh5LXp+Dj0FBxodyXpF2pV/n49bXdOK3LILawzbQTY8hHhYzbUBVwFrLcwM5jnRVxugvhbmN 1wlASCP1Jtppxo4scG1cdtej2n7fk0e+2LDsRp3C1I2BRdAiz2JHAF00caoXBViBtmyPDSjRi ce/9j+bw3h20KRCEfgwj04l2zI Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2022 07:09:55 -0000 Hi Gene, > On Sep 27, 2022, at 05:50, Eugene Y Chang = wrote: >=20 > Of course=E2=80=A6. But the ISP=E2=80=99s maxim is "don=E2=80=99t = believe the customer=E2=80=99s speedtest numbers if it is not with their = host". [SM] In an act of reasonable regulation did the EU give national = regulatory agencies the right to define and set-up national = "speed-tests" (located outside of the access ISPs networks) which ISPs = effectively need to accept. In Germany the local NRA = (Bundes-Netz-Agentur, BNetzA) created a speedtest application against = servers operated on their behalf and will, if a customers demonstrates = the ISP falling short of the contracted rates (according to a somewhat = complicated definition and measurement rules). convince ISPs to follow = the law and either release customers from their contracts immediately or = lower the price in relation to the under-fulfillment. (Unfortunately all = related official web pages are in German only). This put a hold on the practice of gaming speedtests, like = DOCSIS ISPs measuring an in-segment speedtest server which conveniently = hides if a segment's uplink is congested... Not all ISPs gamed their = speedtests, and even the in-segment speedtest can be justified for some = measurements (e.g. when trying to figure out whether congestion is = in-segment or out-of-segment), but the temptation must have been large = to not set-up the most objective spedtest. (We have a saying along the = lines of "making a goat your gardener" which generally is considered a = sub-optimal approach). Regards Sebastian >=20 >=20 > Gene > ---------------------------------------------- > Eugene Chang > IEEE Senior Life Member > eugene.chang@ieee.org > 781-799-0233 (in Honolulu) >=20 >=20 >=20 >> On Sep 26, 2022, at 11:44 AM, Bruce Perens wrote: >>=20 >> That's a good maxim: Don't believe a speed test that is hosted by = your own ISP. >>=20 >> On Mon, Sep 26, 2022 at 2:36 PM Eugene Y Chang via Starlink = wrote: >> Thank you for the dialog,. >> This discussion with regards to Starlink is interesting as it = confirms my guesses about the gap between Starlinks overly simplified, = over optimistic marketing and the reality as they acquire subscribers. >>=20 >> I am actually interested in a more perverse issue. I am seeing = latency and bufferbloat as a consequence from significant under = provisioning. It doesn=E2=80=99t matter that the ISP is selling a fiber = drop, if (parts) of their network is under provisioned. Two end points = can be less than 5 mile apart and realize 120+ ms latency. Two Labor = Days ago (a holiday) the max latency was 230+ ms. The pattern I see = suggest digital redlining. The older communities appear to have much = more severe under provisioning.=20 >>=20 >> Another observation. Running speedtest appears to go from the edge of = the network by layer 2 to the speedtest host operated by the ISP. Yup, = bypasses the (suspected overloaded) routers. >>=20 >> Anyway, just observing. >>=20 >> Gene >> ---------------------------------------------- >> Eugene Chang >> IEEE Senior Life Member >> eugene.chang@ieee.org >> 781-799-0233 (in Honolulu) >>=20 >>=20 >>=20 >>> On Sep 26, 2022, at 11:20 AM, Sebastian Moeller = wrote: >>>=20 >>> Hi Gene, >>>=20 >>>=20 >>>> On Sep 26, 2022, at 23:10, Eugene Y Chang = wrote: >>>>=20 >>>> Comments inline below. >>>>=20 >>>> Gene >>>> ---------------------------------------------- >>>> Eugene Chang >>>> IEEE Senior Life Member >>>> eugene.chang@ieee.org >>>> 781-799-0233 (in Honolulu) >>>>=20 >>>>=20 >>>>=20 >>>>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>>>>=20 >>>>> Hi Eugene, >>>>>=20 >>>>>=20 >>>>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = wrote: >>>>>>=20 >>>>>> Ok, we are getting into the details. I agree. >>>>>>=20 >>>>>> Every node in the path has to implement this to be effective. >>>>>=20 >>>>> Amazingly the biggest bang for the buck is gotten by fixing = those nodes that actually contain a network path's bottleneck. Often = these are pretty stable. So yes for fully guaranteed service quality all = nodes would need to participate, but for improving things noticeably it = is sufficient to improve the usual bottlenecks, e.g. for many internet = access links the home gateway is a decent point to implement better = buffer management. (In short the problem are over-sized and = under-managed buffers, and one of the best solution is better/smarter = buffer management). >>>>>=20 >>>>=20 >>>> This is not completely true. >>>=20 >>> [SM] You are likely right, trying to summarize things leads to = partially incorrect generalizations. >>>=20 >>>=20 >>>> Say the bottleneck is at node N. During the period of congestion, = the upstream node N-1 will have to buffer. When node N recovers, the = bufferbloat at N-1 will be blocking until the bufferbloat drains. Etc. = etc. Making node N better will reduce the extent of the backup at N-1, = but N-1 should implement the better code. >>>=20 >>> [SM] It is the node that builds up the queue that profits most = from better queue management.... (again I generalize, the node with the = queue itself probably does not care all that much, but the endpoints = will profit if the queue experiencing node deals with that queue more = gracefully). >>>=20 >>>=20 >>>>=20 >>>>=20 >>>>>=20 >>>>>> In fact, every node in the path has to have the same = prioritization or the scheme becomes ineffective. >>>>>=20 >>>>> Yes and no, one of the clearest winners has been flow queueing, = IMHO not because it is the most optimal capacity sharing scheme, but = because it is the least pessimal scheme, allowing all (or none) flows = forward progress. You can interpret that as a scheme in which flows = below their capacity share are prioritized, but I am not sure that is = the best way to look at these things. >>>>=20 >>>> The hardest part is getting competing ISPs to implement and = coordinate.=20 >>>=20 >>> [SM] Yes, but it turned out even with non-cooperating ISPs there = is a lot end-users can do unilaterally on their side to improve both = ingress and egress congestion. Admittedly especially ingress congestion = would be even better handled with cooperation of the ISP. >>>=20 >>>> Bufferbloat and handoff between ISPs will be hard. The only way to = fix this is to get the unwashed public to care. Then they can say =E2=80=9C= we don=E2=80=99t care about the technical issues, just fix it.=E2=80=9D = Until then =E2=80=A6.. >>>=20 >>> [SM] Well we do this one home network at a time (not because = that is efficient or ideal, but simply because it is possible). Maybe, = if you have not done so already try OpenWrt with sqm-scripts (and maybe = cake-autorate in addition) on your home internet access link for say a = week and let us know ih/how your experience changed? >>>=20 >>> Regards >>> Sebastian >>>=20 >>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>>=20 >>>>> Regards >>>>> Sebastian >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>> Gene >>>>>> ---------------------------------------------- >>>>>> Eugene Chang >>>>>> IEEE Senior Life Member >>>>>> eugene.chang@ieee.org >>>>>> 781-799-0233 (in Honolulu) >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Sep 26, 2022, at 10:48 AM, David Lang wrote: >>>>>>>=20 >>>>>>> software updates can do far more than just improve recovery. >>>>>>>=20 >>>>>>> In practice, large data transfers are less sensitive to latency = than smaller data transfers (i.e. downloading a CD image vs a video = conference), software can ensure better fairness in preventing a bulk = transfer from hurting the more latency sensitive transfers. >>>>>>>=20 >>>>>>> (the example below is not completely accurate, but I think it = gets the point across) >>>>>>>=20 >>>>>>> When buffers become excessivly large, you have the situation = where a video call is going to generate a small amount of data at a = regular interval, but a bulk data transfer is able to dump a huge amount = of data into the buffer instantly. >>>>>>>=20 >>>>>>> If you just do FIFO, then you get a small chunk of video call, = then several seconds worth of CD transfer, followed by the next small = chunk of the video call. >>>>>>>=20 >>>>>>> But the software can prevent the one app from hogging so much of = the connection and let the chunk of video call in sooner, avoiding the = impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>>>>=20 >>>>>>> The one thing that Cake needs to work really well is to be able = to know what the data rate available is. With Starlink, this changes = frequently and cake integrated into the starlink dish/router software = would be far better than anything that can be done externally as the = rate changes can be fed directly into the settings (currently they are = only indirectly detected) >>>>>>>=20 >>>>>>> David Lang >>>>>>>=20 >>>>>>>=20 >>>>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>>>>=20 >>>>>>>> You already know this. Bufferbloat is a symptom and not the = cause. Bufferbloat grows when there are (1) periods of low or no = bandwidth or (2) periods of insufficient bandwidth (aka network = congestion). >>>>>>>>=20 >>>>>>>> If I understand this correctly, just a software update cannot = make bufferbloat go away. It might improve the speed of recovery (e.g. = throw away all time sensitive UDP messages). >>>>>>>>=20 >>>>>>>> Gene >>>>>>>> ---------------------------------------------- >>>>>>>> Eugene Chang >>>>>>>> IEEE Senior Life Member >>>>>>>> eugene.chang@ieee.org >>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>>>>=20 >>>>>>>>> Please help to explain. Here's a draft to start with: >>>>>>>>>=20 >>>>>>>>> Starlink Performance Not Sufficient for Military Applications, = Say Scientists >>>>>>>>>=20 >>>>>>>>> The problem is not availability: Starlink works where nothing = but another satellite network would. It's not bandwidth, although others = have questions about sustaining bandwidth as the customer base grows. = It's latency and jitter. As load increases, latency, the time it takes = for a packet to get through, increases more than it should. The = scientists who have fought bufferbloat, a major cause of latency on the = internet, know why. SpaceX needs to upgrade their system to use the = scientist's Open Source modifications to Linux to fight bufferbloat, and = thus reduce latency. This is mostly just using a newer version, but = there are some tunable parameters. Jitter is a change in the speed of = getting a packet through the network during a connection, which is = inevitable in satellite networks, but will be improved by making use of = the bufferbloat-fighting software, and probably with the addition of = more satellites. >>>>>>>>>=20 >>>>>>>>> We've done all of the work, SpaceX just needs to adopt it by = upgrading their software, said scientist Dave Taht. Jim Gettys, Taht's = collaborator and creator of the X Window System, chimed in: >>>>>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's = latency and jitter make it inadequate to remote-control my ham radio = station. But the military is experimenting with remote-control of = vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>>>>=20 >>>>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>>>>> The key issue is most people don=E2=80=99t understand why = latency matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>>>>=20 >>>>>>>>> First, we have to help people see the symptoms of latency and = how it impacts something they care about. >>>>>>>>> - gamers care but most people may think it is frivolous. >>>>>>>>> - musicians care but that is mostly for a hobby. >>>>>>>>> - business should care because of productivity but they = don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>>>>=20 >>>>>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing = the action of latency all this time and never knew it! I was being = shafted.=E2=80=9D Once you have this awakening, you can get all the = press you want for free. >>>>>>>>>=20 >>>>>>>>> Most of the time when business apps are developed, =E2=80=9Cwe=E2= =80=9D hide the impact of poor performance (aka latency) or they hide = from the discussion because the developers don=E2=80=99t have a way to = fix the latency. Maybe businesses don=E2=80=99t care because any = employees affected are just considered poor performers. (In bad economic = times, the poor performers are just laid off.) For employees, if they = happen to be at a location with bad latency, they don=E2=80=99t know = that latency is hurting them. Unfair but most people don=E2=80=99t know = the issue is latency. >>>>>>>>>=20 >>>>>>>>> Talking and explaining why latency is bad is not as effective = as showing why latency is bad. Showing has to be with something that has = a person impact. >>>>>>>>>=20 >>>>>>>>> Gene >>>>>>>>> ----------------------------------- >>>>>>>>> Eugene Chang >>>>>>>>> eugene.chang@alum.mit.edu >>>>>>>>> +1-781-799-0233 (in Honolulu) >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>>>>=20 >>>>>>>>>> If you want to get attention, you can get it for free. I can = place articles with various press if there is something interesting to = say. Did this all through the evangelism of Open Source. All we need to = do is write, sign, and publish a statement. What they actually write is = less relevant if they publish a link to our statement. >>>>>>>>>>=20 >>>>>>>>>> Right now I am concerned that the Starlink latency and jitter = is going to be a problem even for remote controlling my ham station. The = US Military is interested in doing much more, which they have = demonstrated, but I don't see happening at scale without some technical = work on the network. Being able to say this isn't ready for the = government's application would be an attention-getter. >>>>>>>>>>=20 >>>>>>>>>> Thanks >>>>>>>>>>=20 >>>>>>>>>> Bruce >>>>>>>>>>=20 >>>>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>>>>> These days, if you want attention, you gotta buy it. A 50k = half page >>>>>>>>>> ad in the wapo or NYT riffing off of It's the latency, = Stupid!", >>>>>>>>>> signed by the kinds of luminaries we got for the fcc wifi = fight, would >>>>>>>>>> go a long way towards shifting the tide. >>>>>>>>>>=20 >>>>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht = > wrote: >>>>>>>>>>>=20 >>>>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>>>>> > wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> The awareness & understanding of latency & impact on QoE is = nearly unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise reporters will = just continue to focus on what they know... >>>>>>>>>>>=20 >>>>>>>>>>> That's a great idea. I have visions of crashing the = washington >>>>>>>>>>> correspondents dinner, but perhaps >>>>>>>>>>> there is some set of gatherings journalists regularly = attend? >>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave = Taht via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> I still find it remarkable that reporters are still = missing the >>>>>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> -- >>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> -- >>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>> _______________________________________________ >>>>>>>>>> Starlink mailing list >>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> -- >>>>>>>>>> Bruce Perens K6BP >>>>>>>>>> _______________________________________________ >>>>>>>>>> Starlink mailing list >>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> -- >>>>>>>>> Bruce Perens K6BP >>>>>>=20 >>>>>> _______________________________________________ >>>>>> Starlink mailing list >>>>>> Starlink@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/starlink >>=20 >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink >>=20 >>=20 >> --=20 >> Bruce Perens K6BP >=20