From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 087FE3B2A4 for ; Tue, 27 Sep 2022 09:56:07 -0400 (EDT) Received: by mail-wr1-x42f.google.com with SMTP id t14so15110344wrx.8 for ; Tue, 27 Sep 2022 06:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date; bh=/UJEIv2Cr6g7lOVEPzWd4lmxpTlxer86pFHA49AVmDY=; b=fEVSwruictjxD3HpL8IWAUYlCHhnxQwKJXSMEg2hErkzjPIWTSsGuT0NPNX1LfbvlF wES9XGTIfvYhrJiAfrc4pfs+0U0GigtCZPQdzV+UrLqGpeXXHABXVlrzNqDLnM+UCVlD LGVxtW8By5Fbjh8BGfwXI0IbVLRUIcdUT0P/xYtOPSLL4n4N1kewFSVVvrzvMsgAO/vn ngHfaZ6+ktHtEXhwdVe07LkR4iv/2cxUwCzDGS9FORMQl7NjauNZFSn8Ns36gr0C/M/r k7jfFB+uMxltD6Jn/9qXXTro6NWtbG5/hr/tEshbgY1r64ilcbTYA5JxQ5iqvB9CfLR9 247A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date; bh=/UJEIv2Cr6g7lOVEPzWd4lmxpTlxer86pFHA49AVmDY=; b=Z+pzAlI9D1HY/ZjZUWIUTLTtsHFTDGf/DMcGFXTA8weA5lmkMjLx60ticnOg5V7mGe n1D5amJtKAE6ursUun8nR69xRIxiH/y/RGmmsG6y5HTC4GJ8IcW1xz5hWWIEYxafM+Fp hPSz8YJpPijt+hKEgBpyBonHjwG5uvapagg3vtojzjKdM7HuBS2hEXP7GBQaqOxbA3ZE T/FgIJxV+/a76nGLXKbueESBGPH14CtaOcspD75n57lZ7pcDDaZoYLlJQprlSbafZFxZ QMCAkNRch5QozG/ZnT59PoQUcKTQE39DMOkJHdIB/96H8ILnU2RBJMdNH9YGkBPfOFtu t+wg== X-Gm-Message-State: ACrzQf3uUws3w2VHBRR8muI/OZAPtcig62RuvF251bW0JMP25VSbMzfe U3UCnnbg2KdaSwehHhd+U+qUPFC+DwJp7N4hLIQ= X-Google-Smtp-Source: AMsMyM7iM2GtN24vY2sGpyIvN+ULhTxuJWNHp6nJ/UGDcVs/sHpyduXWKyD08VIlRKxGUl1DUCXdJyKfqOO1sVCkrp0= X-Received: by 2002:a05:6000:184:b0:22a:cb6f:bb52 with SMTP id p4-20020a056000018400b0022acb6fbb52mr16567306wrx.500.1664286966577; Tue, 27 Sep 2022 06:56:06 -0700 (PDT) MIME-Version: 1.0 References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> <2086C010-B91E-450A-A77F-D0840BC5FCC1@gmx.de> <8E6A87E1-8424-44BE-B582-DF3C8424F584@ieee.org> In-Reply-To: From: Dave Taht Date: Tue, 27 Sep 2022 06:55:52 -0700 Message-ID: To: Sebastian Moeller Cc: Eugene Y Chang , Dave Taht via Starlink Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2022 13:56:08 -0000 On Mon, Sep 26, 2022 at 11:36 PM Sebastian Moeller via Starlink wrote: > Worst, I waited 9 months for them to resolve why I was only getting 300M= bps on my 1Gbps service (oops, sorry my 700Mbps service). Nothing I've seen to date below $400 dollars can actually push a gbit in both directions at the same time, with NAT, firewalling, etc. A lot of gear can't even push a gbit in one directio= n. The long accepted fallacy that a gigabit "router", is just a gigabit switch, bothers me, but it is just one piece of a long line of orwellian doublethink about how the net works but is so pervasive we ended up publishing: https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-rout= er-read-this-first/90305 I've seen an ostensibly serious proposal that budgeted about $4k/sub for a symmetric gig fiber rollout that budgeted... wait for it... $75 dollars for the router, and other proposals that wanted to punt the firewalling etc to a more centralized server, and just put a switch at the sub. > [SM] I guess mass market internet access is a low margin business, where = expert debugging time is precious. I hope they at least gave you a rebate f= or that time. Why is it so few take packet captures anymore? (I think I will rant on that at more length elsewhere) > > Regards > Sebastian > > > > > > > > >Gene > >---------------------------------------------- > >Eugene Chang > >IEEE Senior Life Member > >eugene.chang@ieee.org > >781-799-0233 (in Honolulu) > > > > > > > >> On Sep 26, 2022, at 11:29 AM, Sebastian Moeller wrot= e: > >> > >> Hi David, > >> > >>> On Sep 26, 2022, at 23:22, David Lang wrote: > >>> > >>> On Mon, 26 Sep 2022, Eugene Y Chang wrote: > >>> > >>>>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller w= rote: > >>>>> > >>>>> Hi Eugene, > >>>>> > >>>>> > >>>>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink > wrote: > >>>>>> > >>>>>> Ok, we are getting into the details. I agree. > >>>>>> > >>>>>> Every node in the path has to implement this to be effective. > >>>>> > >>>>> Amazingly the biggest bang for the buck is gotten by fixing those= nodes that actually contain a network path's bottleneck. Often these are p= retty stable. So yes for fully guaranteed service quality all nodes would n= eed to participate, but for improving things noticeably it is sufficient to= improve the usual bottlenecks, e.g. for many internet access links the hom= e gateway is a decent point to implement better buffer management. (In shor= t the problem are over-sized and under-managed buffers, and one of the best= solution is better/smarter buffer management). > >>>>> > >>>> > >>>> This is not completely true. Say the bottleneck is at node N. During= the period of congestion, the upstream node N-1 will have to buffer. When = node N recovers, the bufferbloat at N-1 will be blocking until the bufferbl= oat drains. Etc. etc. Making node N better will reduce the extent of the b= ackup at N-1, but N-1 should implement the better code. > >>> > >>> only if node N and node N-1 handle the same traffic with the same lin= k speeds. In practice this is almost never the case. > >> > >> [SM] I note that typically for ingress shaping a post-true-bottle= neck shaper will not work unless we create an artificial bottleneck by shap= ing the traffic to below true bottleneck (thereby creating a new true but a= rtificial bottleneck so the queue develops at a point where we can control = it). > >> Also if the difference between "true structural" and artificial b= ottleneck is small in comparison to the traffic inrush we can see "traffic = back-spill" into the typically oversized and under-managed upstream buffers= , but for reasonably well behaved that happens relatively rarely. Rarely en= ough that ingress traffic shaping noticeably improves latency-under-load in= spite of not beeing a guranteed solution. > >> > >> > >>> Until you get to gigabit last-mile links, the last mile is almost alw= ays the bottleneck from both sides, so implementing cake on the home router= makes a huge improvement (and if you can get it on the last-mile ISP route= r, even better). Once you get into the Internet fabric, bottlenecks are fai= rly rare, they do happen, but ISPs carefully watch for those and add additi= onal paths and/or increase bandwith to avoid them. > >> > >> [SM] Well, sometimes such links are congested too for economic re= asons... > >> > >> Regards > >> Sebastian > >> > >> > >>> > >>> David Lang > >>> > >>>>> > >>>>>> In fact, every node in the path has to have the same prioritizatio= n or the scheme becomes ineffective. > >>>>> > >>>>> Yes and no, one of the clearest winners has been flow queueing, I= MHO not because it is the most optimal capacity sharing scheme, but because= it is the least pessimal scheme, allowing all (or none) flows forward prog= ress. You can interpret that as a scheme in which flows below their capacit= y share are prioritized, but I am not sure that is the best way to look at = these things. > >>>> > >>>> The hardest part is getting competing ISPs to implement and coordina= te. Bufferbloat and handoff between ISPs will be hard. The only way to fix = this is to get the unwashed public to care. Then they can say =E2=80=9Cwe d= on=E2=80=99t care about the technical issues, just fix it.=E2=80=9D Until t= hen =E2=80=A6.. > >>>> > >>>> > >>>> > >>>>> > >>>>> Regards > >>>>> Sebastian > >>>>> > >>>>> > >>>>>> > >>>>>> Gene > >>>>>> ---------------------------------------------- > >>>>>> Eugene Chang > >>>>>> IEEE Senior Life Member > >>>>>> eugene.chang@ieee.org > >>>>>> 781-799-0233 (in Honolulu) > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On Sep 26, 2022, at 10:48 AM, David Lang wrote: > >>>>>>> > >>>>>>> software updates can do far more than just improve recovery. > >>>>>>> > >>>>>>> In practice, large data transfers are less sensitive to latency t= han smaller data transfers (i.e. downloading a CD image vs a video conferen= ce), software can ensure better fairness in preventing a bulk transfer from= hurting the more latency sensitive transfers. > >>>>>>> > >>>>>>> (the example below is not completely accurate, but I think it get= s the point across) > >>>>>>> > >>>>>>> When buffers become excessivly large, you have the situation wher= e a video call is going to generate a small amount of data at a regular int= erval, but a bulk data transfer is able to dump a huge amount of data into = the buffer instantly. > >>>>>>> > >>>>>>> If you just do FIFO, then you get a small chunk of video call, th= en several seconds worth of CD transfer, followed by the next small chunk o= f the video call. > >>>>>>> > >>>>>>> But the software can prevent the one app from hogging so much of = the connection and let the chunk of video call in sooner, avoiding the impa= ct to the real time traffic. Historically this has required the admin class= ify all traffic and configure equipment to implement different treatment ba= sed on the classification (and this requires trust in the classification pr= ocess), the bufferbloat team has developed options (fq_codel and cake) that= can ensure fairness between applications/servers with little or no configu= ration, and no trust in other systems to properly classify their traffic. > >>>>>>> > >>>>>>> The one thing that Cake needs to work really well is to be able t= o know what the data rate available is. With Starlink, this changes frequen= tly and cake integrated into the starlink dish/router software would be far= better than anything that can be done externally as the rate changes can b= e fed directly into the settings (currently they are only indirectly detect= ed) > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>> > >>>>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: > >>>>>>> > >>>>>>>> You already know this. Bufferbloat is a symptom and not the caus= e. Bufferbloat grows when there are (1) periods of low or no bandwidth or (= 2) periods of insufficient bandwidth (aka network congestion). > >>>>>>>> > >>>>>>>> If I understand this correctly, just a software update cannot ma= ke bufferbloat go away. It might improve the speed of recovery (e.g. throw = away all time sensitive UDP messages). > >>>>>>>> > >>>>>>>> Gene > >>>>>>>> ---------------------------------------------- > >>>>>>>> Eugene Chang > >>>>>>>> IEEE Senior Life Member > >>>>>>>> eugene.chang@ieee.org > >>>>>>>> 781-799-0233 (in Honolulu) > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens w= rote: > >>>>>>>>> > >>>>>>>>> Please help to explain. Here's a draft to start with: > >>>>>>>>> > >>>>>>>>> Starlink Performance Not Sufficient for Military Applications, = Say Scientists > >>>>>>>>> > >>>>>>>>> The problem is not availability: Starlink works where nothing b= ut another satellite network would. It's not bandwidth, although others hav= e questions about sustaining bandwidth as the customer base grows. It's lat= ency and jitter. As load increases, latency, the time it takes for a packet= to get through, increases more than it should. The scientists who have fou= ght bufferbloat, a major cause of latency on the internet, know why. SpaceX= needs to upgrade their system to use the scientist's Open Source modificat= ions to Linux to fight bufferbloat, and thus reduce latency. This is mostly= just using a newer version, but there are some tunable parameters. Jitter = is a change in the speed of getting a packet through the network during a c= onnection, which is inevitable in satellite networks, but will be improved = by making use of the bufferbloat-fighting software, and probably with the a= ddition of more satellites. > >>>>>>>>> > >>>>>>>>> We've done all of the work, SpaceX just needs to adopt it by up= grading their software, said scientist Dave Taht. Jim Gettys, Taht's collab= orator and creator of the X Window System, chimed in: > >>>>>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's la= tency and jitter make it inadequate to remote-control my ham radio station.= But the military is experimenting with remote-control of vehicles on the b= attlefield and other applications that can be demonstrated, but won't happe= n at scale without adoption of bufferbloat-fighting strategies. > >>>>>>>>> > >>>>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang > wrote: > >>>>>>>>> The key issue is most people don=E2=80=99t understand why laten= cy matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. > >>>>>>>>> > >>>>>>>>> First, we have to help people see the symptoms of latency and h= ow it impacts something they care about. > >>>>>>>>> - gamers care but most people may think it is frivolous. > >>>>>>>>> - musicians care but that is mostly for a hobby. > >>>>>>>>> - business should care because of productivity but they don=E2= =80=99t know how to =E2=80=9Csee=E2=80=9D the impact. > >>>>>>>>> > >>>>>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing th= e action of latency all this time and never knew it! I was being shafted.= =E2=80=9D Once you have this awakening, you can get all the press you want = for free. > >>>>>>>>> > >>>>>>>>> Most of the time when business apps are developed, =E2=80=9Cwe= =E2=80=9D hide the impact of poor performance (aka latency) or they hide fr= om the discussion because the developers don=E2=80=99t have a way to fix th= e latency. Maybe businesses don=E2=80=99t care because any employees affect= ed are just considered poor performers. (In bad economic times, the poor pe= rformers are just laid off.) For employees, if they happen to be at a locat= ion with bad latency, they don=E2=80=99t know that latency is hurting them.= Unfair but most people don=E2=80=99t know the issue is latency. > >>>>>>>>> > >>>>>>>>> Talking and explaining why latency is bad is not as effective a= s showing why latency is bad. Showing has to be with something that has a p= erson impact. > >>>>>>>>> > >>>>>>>>> Gene > >>>>>>>>> ----------------------------------- > >>>>>>>>> Eugene Chang > >>>>>>>>> eugene.chang@alum.mit.edu > >>>>>>>>> +1-781-799-0233 (in Honolulu) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink > wrote: > >>>>>>>>>> > >>>>>>>>>> If you want to get attention, you can get it for free. I can p= lace articles with various press if there is something interesting to say. = Did this all through the evangelism of Open Source. All we need to do is wr= ite, sign, and publish a statement. What they actually write is less releva= nt if they publish a link to our statement. > >>>>>>>>>> > >>>>>>>>>> Right now I am concerned that the Starlink latency and jitter = is going to be a problem even for remote controlling my ham station. The US= Military is interested in doing much more, which they have demonstrated, b= ut I don't see happening at scale without some technical work on the networ= k. Being able to say this isn't ready for the government's application woul= d be an attention-getter. > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> > >>>>>>>>>> Bruce > >>>>>>>>>> > >>>>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink > wrote: > >>>>>>>>>> These days, if you want attention, you gotta buy it. A 50k hal= f page > >>>>>>>>>> ad in the wapo or NYT riffing off of It's the latency, Stupid!= ", > >>>>>>>>>> signed by the kinds of luminaries we got for the fcc wifi figh= t, would > >>>>>>>>>> go a long way towards shifting the tide. > >>>>>>>>>> > >>>>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht > wrote: > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason > >>>>>>>>>>> > wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> The awareness & understanding of latency & impact on QoE is = nearly unknown among reporters. IMO maybe there should be some kind of back= ground briefings for reporters - maybe like a simple YouTube video explaine= r that is short & high level & visual? Otherwise reporters will just contin= ue to focus on what they know... > >>>>>>>>>>> > >>>>>>>>>>> That's a great idea. I have visions of crashing the washingto= n > >>>>>>>>>>> correspondents dinner, but perhaps > >>>>>>>>>>> there is some set of gatherings journalists regularly attend? > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave Taht= via Starlink" on behalf of starlink@lists.bufferbloat.net > wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I still find it remarkable that reporters are still missing = the > >>>>>>>>>>>> meaning of the huge latencies for starlink, under load. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> FQ World Domination pending: https://blog.cerowrt.org/post/st= ate_of_fq_codel/ > >>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> FQ World Domination pending: https://blog.cerowrt.org/post/sta= te_of_fq_codel/ > >>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> Starlink mailing list > >>>>>>>>>> Starlink@lists.bufferbloat.net > >>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Bruce Perens K6BP > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> Starlink mailing list > >>>>>>>>>> Starlink@lists.bufferbloat.net > >>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Bruce Perens K6BP > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Starlink mailing list > >>>>>> Starlink@lists.bufferbloat.net > >>>>>> https://lists.bufferbloat.net/listinfo/starlink > > > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --=20 FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_code= l/ Dave T=C3=A4ht CEO, TekLibre, LLC