From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 77D8F3B2A4 for ; Tue, 27 Sep 2022 12:15:02 -0400 (EDT) Received: by mail-wm1-x32b.google.com with SMTP id z13-20020a7bc7cd000000b003b5054c6f9bso9232356wmk.2 for ; Tue, 27 Sep 2022 09:15:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date; bh=S/U60oA9BDaMxGS7FIUTkXdZFVmjDNxA690vPr3+iZk=; b=Ui4tCJzjTD2/De8rmz1Bolp/cqTALhPEombu4pfmlTOl11huvC64e9hfeyDxC8spKo cD6zHfgQ0iRPxkrOIDsdcagOBXHvssX/jFN6q1ZU3sHY40HmNDXSRJaSLlefij9hKaI+ 54lBL2zvN0JSNCd97BnR98UisYlsVi5T/cmxkEr6z7v+xEgAfHJ7v9Q9X0W0IcXeZUQu kCpEXMyNNSaKOlh1LIqNaT1gOh1hZTwXAVBFifVJRRwjNjhQAAJwiCbVTo7j80b4UU1D kIOniBpk6TRE6n/DMuOIRwpvnUpiudgO/iRcJIlF4NuRlO5xTpUcZu9MLDyu5BmB+Hg8 Fx2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date; bh=S/U60oA9BDaMxGS7FIUTkXdZFVmjDNxA690vPr3+iZk=; b=nkZ7pAnBMI3cB3sFXlYvaofaYgUhWEiVPhe7qw2JA/FrZgOvLxe8Z20AajbrVWojX7 OgA7mgBgX8EDoV0gumF+gZQ1+Kf1gha3uOKi7wyNIID4GIYqU0MgHlM15hTx9+Abajjg HuSCiPa19xQI0393XyYMuBvAMmhefY9I1/bmHOvbYAE70GTMnTixA/xT6HXecJr5E6AL D+AjFoBNtL050A416YTYGpF+rxLGskTfaVS565WiJC496K8Ro8TVL+gyko+zHD8HArEg bOC0BFZYvbjWD51sPp4J5mEvbWK0LfNZkv97O4e1FYjSkBlXZxtAPHU7qZLl0sgxOTFT QnNw== X-Gm-Message-State: ACrzQf0NWpjS+GBVQlM6ewMxkJL+s8wpUq7UByf7RCTP4BysMtLNkrId Q6xmGVEAQMBG6l7pIwrG7LT7ZibklHzrMrpgzH8= X-Google-Smtp-Source: AMsMyM5WDZRcd2CqtRs8drBc3rpwtbDAAwJ6qdU60Veg8CWRA2zYI5IfSXuYNuweDzAfnCuvQlCA1Jljf6eelxELHoA= X-Received: by 2002:a05:600c:1da2:b0:3b4:856a:162c with SMTP id p34-20020a05600c1da200b003b4856a162cmr3266242wms.28.1664295300860; Tue, 27 Sep 2022 09:15:00 -0700 (PDT) MIME-Version: 1.0 References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> In-Reply-To: From: Dave Taht Date: Tue, 27 Sep 2022 09:14:43 -0700 Message-ID: To: warren ponder Cc: Eugene Y Chang , Dave Taht via Starlink Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2022 16:15:02 -0000 On Mon, Sep 26, 2022 at 2:28 PM warren ponder wrote: > > Dave what do you need in order to add sites to the data collection. Feel = free to reply separate or link to a previous thread Hey, thx for being willing to wade in! Simplest place to start is with packet captures of a long running iperf/iperf3/netperf, or even speedtest.net, up and own, and tear 'em apart in wireshark or tcptrace -G & xplot.org Here's some history for y'all: https://gettys.wordpress.com/2010/12/06/whose-house-is-of-glasse-must-not-t= hrow-stones-at-another/ We've discussed a lot of tools on this list, my preferred one is flent, which lets you see details without needing packet captures. We have some great plots of irtt with a 3ms interval that I'd love to be doing more regularly, worldwide. Over here I'd listed a bunch of (to me at least) useful research topics that I'd hoped would attract a student or three to pursue more fully. This was the draft plan we'd had tackling ALL the problems as we saw then. Finding some aspect of this we could function together as a group on, would be great. https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2y= BYtw/edit# > Thx > > WP > > On Mon, Sep 26, 2022, 2:10 PM Dave Taht via Starlink wrote: >> >> I tend to cite rfc7567 (published 2015) a lot, which replaces rfc2309 >> (published 1992!). >> >> Thing is, long before that, I'd come to the conclusion that fair >> queuing was a requirement for >> sustaining the right throughput for low rate flows in wildly variable >> bandwidth. At certain places in >> LTE/5g/starlink networks the payload is encrypted and the header info >> required unavailable, and my advocacy of fq is certainly not shared by >> everyone. >> >> We don't know enough about the actual points of congestion in starlink >> to know if fq could be applied, >> and although aqm is a very good idea everywhere, is also largely >> undeployed where it would matter most. >> >> I focused my initial analysis of starlink on just uplink congestion, >> which I believe can be easily improved given about 20 minutes with a >> cross compiler for the dishy. We have a very good proof of concept as >> to how to improve starlinks behavior over here: >> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/87 and >> ironically the same script could be run on their router as it is based >> on a 6 year old version of openwrt in the first place. >> >> I have plenty of data later than this ( >> https://docs.google.com/document/d/1puRjUVxJ6cCv-rgQ_zn-jWZU9ae0jZbFATLf= 4PQKblM/edit >> ) but I would like to be collecting it from at least six sites around >> the world. >> >> On Mon, Sep 26, 2022 at 1:54 PM Eugene Y Chang via Starlink >> wrote: >> > >> > Ok, we are getting into the details. I agree. >> > >> > Every node in the path has to implement this to be effective. >> > In fact, every node in the path has to have the same prioritization or= the scheme becomes ineffective. >> > >> > Gene >> > ---------------------------------------------- >> > Eugene Chang >> > IEEE Senior Life Member >> > eugene.chang@ieee.org >> > 781-799-0233 (in Honolulu) >> > >> > >> > >> > On Sep 26, 2022, at 10:48 AM, David Lang wrote: >> > >> > software updates can do far more than just improve recovery. >> > >> > In practice, large data transfers are less sensitive to latency than s= maller data transfers (i.e. downloading a CD image vs a video conference), = software can ensure better fairness in preventing a bulk transfer from hurt= ing the more latency sensitive transfers. >> > >> > (the example below is not completely accurate, but I think it gets the= point across) >> > >> > When buffers become excessivly large, you have the situation where a v= ideo call is going to generate a small amount of data at a regular interval= , but a bulk data transfer is able to dump a huge amount of data into the b= uffer instantly. >> > >> > If you just do FIFO, then you get a small chunk of video call, then se= veral seconds worth of CD transfer, followed by the next small chunk of the= video call. >> > >> > But the software can prevent the one app from hogging so much of the c= onnection and let the chunk of video call in sooner, avoiding the impact to= the real time traffic. Historically this has required the admin classify a= ll traffic and configure equipment to implement different treatment based o= n the classification (and this requires trust in the classification process= ), the bufferbloat team has developed options (fq_codel and cake) that can = ensure fairness between applications/servers with little or no configuratio= n, and no trust in other systems to properly classify their traffic. >> > >> > The one thing that Cake needs to work really well is to be able to kno= w what the data rate available is. With Starlink, this changes frequently a= nd cake integrated into the starlink dish/router software would be far bett= er than anything that can be done externally as the rate changes can be fed= directly into the settings (currently they are only indirectly detected) >> > >> > David Lang >> > >> > >> > On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >> > >> > You already know this. Bufferbloat is a symptom and not the cause. Buf= ferbloat grows when there are (1) periods of low or no bandwidth or (2) per= iods of insufficient bandwidth (aka network congestion). >> > >> > If I understand this correctly, just a software update cannot make buf= ferbloat go away. It might improve the speed of recovery (e.g. throw away a= ll time sensitive UDP messages). >> > >> > Gene >> > ---------------------------------------------- >> > Eugene Chang >> > IEEE Senior Life Member >> > eugene.chang@ieee.org >> > 781-799-0233 (in Honolulu) >> > >> > >> > >> > On Sep 26, 2022, at 10:04 AM, Bruce Perens wrote: >> > >> > Please help to explain. Here's a draft to start with: >> > >> > Starlink Performance Not Sufficient for Military Applications, Say Sci= entists >> > >> > The problem is not availability: Starlink works where nothing but anot= her satellite network would. It's not bandwidth, although others have quest= ions about sustaining bandwidth as the customer base grows. It's latency an= d jitter. As load increases, latency, the time it takes for a packet to get= through, increases more than it should. The scientists who have fought buf= ferbloat, a major cause of latency on the internet, know why. SpaceX needs = to upgrade their system to use the scientist's Open Source modifications to= Linux to fight bufferbloat, and thus reduce latency. This is mostly just u= sing a newer version, but there are some tunable parameters. Jitter is a ch= ange in the speed of getting a packet through the network during a connecti= on, which is inevitable in satellite networks, but will be improved by maki= ng use of the bufferbloat-fighting software, and probably with the addition= of more satellites. >> > >> > We've done all of the work, SpaceX just needs to adopt it by upgrading= their software, said scientist Dave Taht. Jim Gettys, Taht's collaborator = and creator of the X Window System, chimed in: >> > Open Source luminary Bruce Perens said: sometimes Starlink's latency a= nd jitter make it inadequate to remote-control my ham radio station. But th= e military is experimenting with remote-control of vehicles on the battlefi= eld and other applications that can be demonstrated, but won't happen at sc= ale without adoption of bufferbloat-fighting strategies. >> > >> > On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang > wrote: >> > The key issue is most people don=E2=80=99t understand why latency matt= ers. They don=E2=80=99t see it or feel it=E2=80=99s impact. >> > >> > First, we have to help people see the symptoms of latency and how it i= mpacts something they care about. >> > - gamers care but most people may think it is frivolous. >> > - musicians care but that is mostly for a hobby. >> > - business should care because of productivity but they don=E2=80=99t = know how to =E2=80=9Csee=E2=80=9D the impact. >> > >> > Second, there needs to be a =E2=80=9COMG, I have been seeing the actio= n of latency all this time and never knew it! I was being shafted.=E2=80=9D= Once you have this awakening, you can get all the press you want for free. >> > >> > Most of the time when business apps are developed, =E2=80=9Cwe=E2=80= =9D hide the impact of poor performance (aka latency) or they hide from the= discussion because the developers don=E2=80=99t have a way to fix the late= ncy. Maybe businesses don=E2=80=99t care because any employees affected are= just considered poor performers. (In bad economic times, the poor performe= rs are just laid off.) For employees, if they happen to be at a location wi= th bad latency, they don=E2=80=99t know that latency is hurting them. Unfai= r but most people don=E2=80=99t know the issue is latency. >> > >> > Talking and explaining why latency is bad is not as effective as showi= ng why latency is bad. Showing has to be with something that has a person i= mpact. >> > >> > Gene >> > ----------------------------------- >> > Eugene Chang >> > eugene.chang@alum.mit.edu >> > +1-781-799-0233 (in Honolulu) >> > >> > >> > >> > >> > >> > On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink > wrote: >> > >> > If you want to get attention, you can get it for free. I can place art= icles with various press if there is something interesting to say. Did this= all through the evangelism of Open Source. All we need to do is write, sig= n, and publish a statement. What they actually write is less relevant if th= ey publish a link to our statement. >> > >> > Right now I am concerned that the Starlink latency and jitter is going= to be a problem even for remote controlling my ham station. The US Militar= y is interested in doing much more, which they have demonstrated, but I don= 't see happening at scale without some technical work on the network. Being= able to say this isn't ready for the government's application would be an = attention-getter. >> > >> > Thanks >> > >> > Bruce >> > >> > On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink > wrote: >> > These days, if you want attention, you gotta buy it. A 50k half page >> > ad in the wapo or NYT riffing off of It's the latency, Stupid!", >> > signed by the kinds of luminaries we got for the fcc wifi fight, would >> > go a long way towards shifting the tide. >> > >> > On Mon, Sep 26, 2022 at 8:29 AM Dave Taht > wrote: >> > >> > >> > On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >> > > wro= te: >> > >> > >> > The awareness & understanding of latency & impact on QoE is nearly unk= nown among reporters. IMO maybe there should be some kind of background bri= efings for reporters - maybe like a simple YouTube video explainer that is = short & high level & visual? Otherwise reporters will just continue to focu= s on what they know... >> > >> > >> > That's a great idea. I have visions of crashing the washington >> > correspondents dinner, but perhaps >> > there is some set of gatherings journalists regularly attend? >> > >> > >> > =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave Taht via Starl= ink" on behalf of starlink@lists.bufferbloat.net > wrote: >> > >> > I still find it remarkable that reporters are still missing the >> > meaning of the huge latencies for starlink, under load. >> > >> > >> > >> > >> > -- >> > FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq= _codel/ >> > Dave T=C3=A4ht CEO, TekLibre, LLC >> > >> > >> > >> > >> > -- >> > FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq= _codel/ >> > Dave T=C3=A4ht CEO, TekLibre, LLC >> > _______________________________________________ >> > Starlink mailing list >> > Starlink@lists.bufferbloat.net >> > https://lists.bufferbloat.net/listinfo/starlink >> > >> > >> > -- >> > Bruce Perens K6BP >> > _______________________________________________ >> > Starlink mailing list >> > Starlink@lists.bufferbloat.net >> > https://lists.bufferbloat.net/listinfo/starlink >> > >> > >> > >> > >> > -- >> > Bruce Perens K6BP >> > >> > >> > _______________________________________________ >> > Starlink mailing list >> > Starlink@lists.bufferbloat.net >> > https://lists.bufferbloat.net/listinfo/starlink >> >> >> >> -- >> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_c= odel/ >> Dave T=C3=A4ht CEO, TekLibre, LLC >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink --=20 FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_code= l/ Dave T=C3=A4ht CEO, TekLibre, LLC