From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp85.iad3a.emailsrvr.com (smtp85.iad3a.emailsrvr.com [173.203.187.85]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 343DB3B2A4 for ; Sat, 17 Jul 2021 14:36:36 -0400 (EDT) Received: from app38.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp3.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 9BA2023A1D; Sat, 17 Jul 2021 14:36:35 -0400 (EDT) Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app38.wa-webapps.iad3a (Postfix) with ESMTP id 8AB34E199E; Sat, 17 Jul 2021 14:36:35 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Sat, 17 Jul 2021 14:36:35 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Sat, 17 Jul 2021 14:36:35 -0400 (EDT) From: "David P. Reed" To: "Nathan Owens" Cc: "Mike Puchol" , "David Lang" , "starlink@lists.bufferbloat.net" MIME-Version: 1.0 Content-Type: text/plain;charset=UTF-8 Content-Transfer-Encoding: quoted-printable Importance: Normal X-Priority: 3 (Normal) X-Type: plain In-Reply-To: References: <1625856001.74681750@apps.rackspace.com> X-Client-IP: 209.6.168.128 Message-ID: <1626546995.56721936@apps.rackspace.com> X-Mailer: webmail/19.0.7-RC X-Classification-ID: d57aff6c-246a-46f9-8e40-47ba57c2787b-1-1 Subject: Re: [Starlink] =?utf-8?q?Starlink_and_bufferbloat_status=3F?= X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Jul 2021 18:36:36 -0000 Just to clarify, Starlink does NOT do inter-satellite data transfer. Each s= atellite is a "bent-pipe" which just reflects data transmitted to it back d= own.=0A=0ANY-Tokyo isn't line of sight, even at LEO. This is why GEO satell= ites are used.=0A=0AIridium had the ability to inter-satellite routing. How= ever, still, multiple hops are needed, and tracking all the other satellite= s in view is not easy, and stable routing management would be non-trivial i= ndeed. (not counting the much lower power/bit available between battery pow= ered satellites that aren't in the sun half the time. That was Iridium's ba= sic technical issue - battery storage. 60 satellites were barely tractable = for routing.=0A=0AThere seems to be a lot of imagination by the credulous t= echnical community going into dreaming about what Starlink actually can ach= ieve. Those who have actually worked with satellite systems know there is n= o magical genius that Musk has. He just launches lots of simple orbiting "m= irrors" (active repeaters) close to the Earth's surface. Pragmatic engineer= ing, exploiting better digital signal processing, lower power digital syste= ms, better antenna systems that use MIMO (or phased array, or whatever you = want to call it).=0A=0A=0AThe reason it is all so hush-hush, to this old en= gineer, anyway, is the usual attempt to exploit the Wizard of Oz effect. (T= rade secrets don't work well, so Musk would have filed patents worldwide if= he had any novel special engineering magic). Musk is great at exploiting p= ublicity to get people to think there's an all-powerful wizard behind the s= cenes. There isn't. This is like the belief that there is lots of highly cl= assified technology that scientists don't understand unless cleared. There = really isn't that much - the classification is about hiding all the weaknes= ses in military systems. A good thing, but not proof of mysterious magical = science beyond commercial practice. Star Wars was this kind of exploitation= of Magical Thinking in the public. A marketing blitz to con people who pro= jected their imagination on a fairly mundane engineering project with lots = of weaknesses.=0A=0ANow I'm happy to see Musk lose money hand over fist to = show us how bent-pipe LEO systems can work somewhat reliably. What else sho= uld a billionaire do to make his money useful? We learn stuff that way. And= issues like bufferbloat get rediscovered, and fixed, if his team pays atte= ntion. This is good in the long run.=0A=0ABut customers who bought Tesla's = expecting Autopilot to become self-driving? Or people who bought Tesla stoc= k thinking no other companies could compete? They are waiting for something= that is not real.=0A=0AOn Friday, July 16, 2021 1:35pm, "Nathan Owens" said:=0A=0A> The other case where they could provide benefi= t is very long distance paths=0A> --- NY to Tokyo, Johannesburg to London, = etc... but presumably at high=0A> cost, as the capacity will likely be much= lower than submarine cables.=0A> =0A> On Fri, Jul 16, 2021 at 10:31 AM Mik= e Puchol wrote:=0A> =0A>> Satellite optical links are us= eful to extend coverage to areas where you=0A>> don=E2=80=99t have gateways= - thus, they will introduce additional latency=0A>> compared=0A>> to two s= pace segment hops (terminal to satellite -> satellite to gateway).=0A>> If = you have terminal to satellite, two optical hops, then final satellite=0A>>= to gateway, you will have more latency, not less.=0A>>=0A>> We are being = =E2=80=9Csold=E2=80=9D optical links for what they are not IMHO.=0A>>=0A>> = Best,=0A>>=0A>> Mike=0A>> On Jul 16, 2021, 19:29 +0200, Nathan Owens , wrote:=0A>>=0A>> > As there are more satellites, the up down = time will get closer to 4-5ms=0A>> rather then the ~7ms you list=0A>>=0A>> = Possibly, if you do steering to always jump to the lowest latency=0A>> sate= llite.=0A>>=0A>> > with laser relays in orbit, and terminal to terminal rou= ting in orbit,=0A>> there is the potential for the theoretical minimum to t= end lower=0A>> Maybe for certain users really in the middle of nowhere, but= I did the=0A>> best-case math for "bent pipe" in Seattle area, which is as= good as it gets.=0A>>=0A>> On Fri, Jul 16, 2021 at 10:24 AM David Lang wrote:=0A>>=0A>>> hey, it's a good attitude to have :-)=0A>>>= =0A>>> Elon tends to set 'impossible' goals, miss the timeline a bit, and c= ome=0A>>> very=0A>>> close to the goal, if not exceed it.=0A>>>=0A>>> As th= ere are more staellites, the up down time will get closer to 4-5ms=0A>>> ra= ther=0A>>> then the ~7ms you list, and with laser relays in orbit, and term= inal to=0A>>> terminal=0A>>> routing in orbit, there is the potential for t= he theoretical minimum to=0A>>> tend=0A>>> lower, giving some headroom for = other overhead but still being in the 20ms=0A>>> range.=0A>>>=0A>>> David L= ang=0A>>>=0A>>> On Fri, 16 Jul 2021, Nathan Owens wrote:=0A>>>=0A>>> > El= on said "foolish packet routing" for things over 20ms! Which seems=0A>>> cr= azy=0A>>> > if you do some basic math:=0A>>> >=0A>>> > - Sat to User Term= inal distance: 550-950km air/vacuum: 1.9 - 3.3ms=0A>>> > - Sat to GW dist= ance: 550-950km air/vacuum: 1.9 - 3.3ms=0A>>> > - GW to PoP Distance: 50-= 800km fiber: 0.25 - 4ms=0A>>> > - PoP to Internet Distance: 50km fiber: 0= .25 - 0.5ms=0A>>> > - Total one-way delay: 4.3 - 11.1ms=0A>>> > - Theor= etical minimum RTT: 8.6ms - 22.2ms, call it 15.4ms=0A>>> >=0A>>> > This inc= ludes no transmission delay, queuing delay,=0A>>> > processing/fragmentatio= n/reassembly/etc, and no time-division=0A>>> multiplexing.=0A>>> >=0A>>> > = On Fri, Jul 16, 2021 at 10:09 AM David Lang wrote:=0A>>> >= =0A>>> >> I think it depends on if you are looking at datacenter-to-datacen= ter=0A>>> >> latency of=0A>>> >> home to remote datacenter latency :-)=0A>>= > >>=0A>>> >> my rule of thumb for cross US ping time has been 80-100ms lat= ency (but=0A>>> >> it's been=0A>>> >> a few years since I tested it).=0A>>>= >>=0A>>> >> I note that an article I saw today said that Elon is saying th= at=0A>>> latency=0A>>> >> will=0A>>> >> improve significantly in the near f= uture, that up/down latency is ~20ms=0A>>> >> and the=0A>>> >> additional d= elays pushing it to the 80ms range are 'stupid packet=0A>>> routing'=0A>>> = >> problems that they are working on.=0A>>> >>=0A>>> >> If they are still i= n that level of optimization, it doesn't surprise me=0A>>> >> that=0A>>> >>= they haven't really focused on the bufferbloat issue, they have more=0A>>>= >> obvious=0A>>> >> stuff to fix first.=0A>>> >>=0A>>> >> David Lang=0A>>>= >>=0A>>> >>=0A>>> >> On Fri, 16 Jul 2021, Wheelock, Ian wrote:=0A>>> >>= =0A>>> >>> Date: Fri, 16 Jul 2021 10:21:52 +0000=0A>>> >>> From: "Wheelock,= Ian" =0A>>> >>> To: David Lang = , David P. Reed =0A>>> >>> Cc: "starlink@lists.bufferb= loat.net" =0A>>> >>> Subject: Re: [Starlink= ] Starlink and bufferbloat status?=0A>>> >>>=0A>>> >>> Hi David=0A>>> >>> I= n terms of the Latency that David (Reed) mentioned for California to=0A>>> = >> Massachusetts of about 17ms over the public internet, it seems a bit=0A>= >> faster=0A>>> >> than what I would expect. My own traceroute via my VDSL = link shows 14ms=0A>>> >> just to get out of the operator network.=0A>>> >>>= =0A>>> >>> https://www.wondernetwork.com is a handy tool for checking=0A>>= > geographic=0A>>> >> ping perf between cities, and it shows a min of about= 66ms for pings=0A>>> >> between Boston and San Diego=0A>>> >> https://wond= ernetwork.com/pings/boston/San%20Diego (so about 33ms for=0A>>> >> 1-way tr= ansfer).=0A>>> >>>=0A>>> >>> Distance wise this is about 4,100 KM (2,500 M)= , and @2/3 speed of=0A>>> light=0A>>> >> (through a pure fibre link of that= distance) the propagation time is=0A>>> just=0A>>> >> over 20ms. If the ne= twork equipment between the Boston and San Diego is=0A>>> >> factored in, w= ith some buffering along the way, 33ms does seem quite=0A>>> >> reasonable = over the 20ms for speed of light in fibre for that 1-way=0A>>> transfer=0A>= >> >>>=0A>>> >>> -Ian Wheelock=0A>>> >>>=0A>>> >>> From: Starlink on behalf of=0A>>> >> David Lang =0A>>> >>> Date: Friday 9 July 2021 at 23:59=0A>>> >>> To: "David P. R= eed" =0A>>> >>> Cc: "starlink@lists.bufferbloat.net" <= starlink@lists.bufferbloat.net>=0A>>> >>> Subject: Re: [Starlink] Starlink = and bufferbloat status?=0A>>> >>>=0A>>> >>> IIRC, the definition of 'low la= tency' for the FCC was something like=0A>>> >> 100ms, and Musk was predicti= ng <40ms. roughly competitive with=0A>>> landlines,=0A>>> >> and worlds bet= ter than geostationary satellite (and many=0A>>> >>> External (mailto:david= @lang.hm)=0A>>> >>>=0A>>> >>=0A>>> https://shared.outlook.inky.com/report?i= d=3DY29tbXNjb3BlL2lhbi53aGVlbG9ja0Bjb21tc2NvcGUuY29tL2I1MzFjZDA4OTZmMWI0Yzc= 5NzdiOTIzNmY3MTAzM2MxLzE2MjU4NzE1NDkuNjU=3D#key=3D19e8545676e28e577c813de83= a4cf1dc=0A>>> >> https://www.inky.com/banner-faq/ https://www.inky.com=0A= >>> >>>=0A>>> >>> IIRC, the definition of 'low latency' for the FCC was som= ething like=0A>>> >> 100ms, and=0A>>> >>> Musk was predicting <40ms.=0A>>> = >>>=0A>>> >>> roughly competitive with landlines, and worlds better than=0A= >>> geostationary=0A>>> >>> satellite (and many wireless ISPs)=0A>>> >>>=0A= >>> >>> but when doing any serious testing of latency, you need to be wired= to=0A>>> >> the=0A>>> >>> router, wifi introduces so much variability that= it swamps the signal.=0A>>> >>>=0A>>> >>> David Lang=0A>>> >>>=0A>>> >>> O= n Fri, 9 Jul 2021, David P. Reed wrote:=0A>>> >>>=0A>>> >>>> Date: Fri, 9 J= ul 2021 14:40:01 -0400 (EDT)=0A>>> >>>> From: David P. Reed =0A>>> >>>> To: starlink@lists.bufferbloat.net=0A>>> >>>> Subject: [S= tarlink] Starlink and bufferbloat status?=0A>>> >>>>=0A>>> >>>>=0A>>> >>>> = Early measurements of performance of Starlink have shown significant=0A>>> = >> bufferbloat, as Dave Taht has shown.=0A>>> >>>>=0A>>> >>>> But... Starl= ink is a moving target. The bufferbloat isn't a hardware=0A>>> >> issue, it= should be completely manageable, starting by simple firmware=0A>>> >> chan= ges inside the Starlink system itself. For example, implementing=0A>>> >> f= q_codel so that bottleneck links just drop packets according to the=0A>>> B= est=0A>>> >> Practices RFC,=0A>>> >>>>=0A>>> >>>> So I'm hoping this has im= proved since Dave's measurements. How much=0A>>> has=0A>>> >> it improved? = What's the current maximum packet latency under full=0A>>> >> load, Ive he= ard anecdotally that a friend of a friend gets 84 msec.=0A>>> *ping=0A>>> >= > times under full load*, but he wasn't using flent or some other=0A>>> mea= surement=0A>>> >> tool of good quality that gives a true number.=0A>>> >>>>= =0A>>> >>>> 84 msec is not great - it's marginal for Zoom quality experienc= e (you=0A>>> >> want latencies significantly less than 100 msec. as a rule = of thumb for=0A>>> >> teleconferencing quality). But it is better than Dave= 's measurements=0A>>> showed.=0A>>> >>>>=0A>>> >>>> Now Musk bragged that h= is network was "low latency" unlike other high=0A>>> >> speed services, whi= ch means low end-to-end latency. That got him=0A>>> >> permission from the= FCC to operate Starlink at all. His number was, I=0A>>> >> think, < 5 msec= . 84 is a lot more than 5. (I didn't believe 5, because=0A>>> he=0A>>> >> p= robably meant just the time from the ground station to the terminal=0A>>> >= > through the satellite. But I regularly get 17 msec. between California=0A= >>> and=0A>>> >> Massachusetts over the public Internet)=0A>>> >>>>=0A>>> >= >>> So 84 might be the current status. That would mean that someone at=0A>>= > >> Srarlink might be paying some attention, but it is a long way from wha= t=0A>>> >> Musk implied.=0A>>> >>>>=0A>>> >>>>=0A>>> >>>> PS: I forget the = number of the RFC, but the number of packets queued=0A>>> on=0A>>> >> an eg= ress link should be chosen by taking the hardware bottleneck=0A>>> >> throu= ghput of any path, combined with an end-to-end Internet underlying=0A>>> >>= delay of about 10 msec. to account for hops between source and=0A>>> desti= nation.=0A>>> >> Lets say Starlink allocates 50 Mb/sec to each customer, pa= ckets are=0A>>> limited=0A>>> >> to 10,000 bits (1500 * 8), so the outbound= queues should be limited to=0A>>> >> about 0.01 * 50,000,000 / 10,000, whi= ch comes out to about 250 packets=0A>>> from=0A>>> >> each terminal of buff= ering, total, in the path from terminal to public=0A>>> >> Internet, assumi= ng the connection to the public Internet is not a=0A>>> problem.=0A>>> >>> = _______________________________________________=0A>>> >>> Starlink mailing = list=0A>>> >>> Starlink@lists.bufferbloat.net=0A>>> >>>=0A>>> >>=0A>>> http= s://secure-web.cisco.com/1sNc_-1HhGCW7xdirt_lAoAy5Nn5T6UA85Scjn5BR7QHXtumhr= f6RKn78SuRJG7DUKI3duggU9g6hJKW-Ze07HTczYqB9mBpIeALqk5drQ7nMvM8K7JbWfUbPR7JS= NrI75UjiNXQk0wslBfoOTvkMlRj5eMOZhps7DMGBRQTVAeTd5vwXoQtDgS6zLCcJkrcO2S9MRSC= C4f1I17SzgQJIwqo3LEwuN6lD-pkX0MFLqGr2zzsHw5eapd-VBlHu5reC4-OEn2zHkb7HNzS1pc= ueF6tsUE1vFRsWs2SIOwU5MvbKe3J3Q6NRQ40cHI1AGd-i/https://lists.bufferbloat.ne= t/listinfo/starlink=0A>>> >>>=0A>>> >>> ___________________________________= ____________=0A>>> >> Starlink mailing list=0A>>> >> Starlink@lists.bufferb= loat.net=0A>>> >> https://lists.bufferbloat.net/listinfo/starlink=0A>>> >>= =0A>>> >=0A>>>=0A>> _______________________________________________=0A>> St= arlink mailing list=0A>> Starlink@lists.bufferbloat.net=0A>> https://lists.= bufferbloat.net/listinfo/starlink=0A>>=0A>>=0A> =0A