From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.lang.hm (unknown [66.167.227.145]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 54D7F3CB37 for ; Sun, 18 Jul 2021 21:20:13 -0400 (EDT) Received: from dlang-laptop.local (unknown [10.2.0.162]) by mail.lang.hm (Postfix) with ESMTP id E1CF2FF769; Sun, 18 Jul 2021 18:20:11 -0700 (PDT) Date: Sun, 18 Jul 2021 18:20:06 -0700 (PDT) From: David Lang X-X-Sender: dlang@dlang-laptop To: Nick Buraglio cc: Jonathan Bennett , starlink@lists.bufferbloat.net, "David P. Reed" In-Reply-To: Message-ID: References: <1625856001.74681750@apps.rackspace.com> User-Agent: Alpine 2.21.1 (DEB 209 2017-03-23) MIME-Version: 1.0 Content-Type: multipart/mixed; BOUNDARY="===============4865789427217445894==" Subject: Re: [Starlink] Starlink and bufferbloat status? X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2021 01:20:13 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --===============4865789427217445894== Content-Type: text/plain; format=flowed; charset=UTF-8 Content-Transfer-Encoding: 8BIT Elon is talking about a viable path in the future being dishy - sat - sat - dishy They aren't there yet, but they are sure planning on it David Lang On Sun, 18 Jul 2021, Nick Buraglio wrote: > We keep saying “route”. What do we actually mean from a network stack > perspective? Are we talking about relaying light / frames / electric or do > we mean actual packet routing, because there are obviously a lot of > important distinctions there. > I’m willing to bet that there is no routing (as in layer 3 packet routing) > at all except the Dish NAT all the way into their peering data center. The > ground stations are very likely RF to fiber wave division back to a carrier > hotel with no L3 buffering at all. That keeps latency very low (think O-E-O > and E-O transitions) and moves L3 buffering to two locations and keeps the > terrestrial network very easy to make redundant (optical protection, etc.). > > nb > > On Fri, Jul 16, 2021 at 12:39 PM Jonathan Bennett < > jonathanbennett@hackaday.com> wrote: > >> >> >> On Fri, Jul 16, 2021, 12:35 PM Nathan Owens wrote: >> >>> The other case where they could provide benefit is very long distance >>> paths --- NY to Tokyo, Johannesburg to London, etc... but presumably at >>> high cost, as the capacity will likely be much lower than submarine cables. >>> >>>> >> Or traffic between Starlink customers. A video call between me and someone >> else on the Starlink network is going to be drastically better if it can >> route over the sats. >> >>> >>>> On Fri, Jul 16, 2021 at 10:31 AM Mike Puchol wrote: >>> >>>> Satellite optical links are useful to extend coverage to areas where you >>>> don’t have gateways - thus, they will introduce additional latency compared >>>> to two space segment hops (terminal to satellite -> satellite to gateway). >>>> If you have terminal to satellite, two optical hops, then final satellite >>>> to gateway, you will have more latency, not less. >>>> >>>> We are being “sold” optical links for what they are not IMHO. >>>> >>>> Best, >>>> >>>> Mike >>>> On Jul 16, 2021, 19:29 +0200, Nathan Owens , wrote: >>>> >>>>> As there are more satellites, the up down time will get closer to >>>> 4-5ms rather then the ~7ms you list >>>> >>>> Possibly, if you do steering to always jump to the lowest latency >>>> satellite. >>>> >>>>> with laser relays in orbit, and terminal to terminal routing in orbit, >>>> there is the potential for the theoretical minimum to tend lower >>>> Maybe for certain users really in the middle of nowhere, but I did the >>>> best-case math for "bent pipe" in Seattle area, which is as good as it gets. >>>> >>>> On Fri, Jul 16, 2021 at 10:24 AM David Lang wrote: >>>> >>>>> hey, it's a good attitude to have :-) >>>>> >>>>> Elon tends to set 'impossible' goals, miss the timeline a bit, and come >>>>> very >>>>> close to the goal, if not exceed it. >>>>> >>>>> As there are more staellites, the up down time will get closer to 4-5ms >>>>> rather >>>>> then the ~7ms you list, and with laser relays in orbit, and terminal to >>>>> terminal >>>>> routing in orbit, there is the potential for the theoretical minimum to >>>>> tend >>>>> lower, giving some headroom for other overhead but still being in the >>>>> 20ms >>>>> range. >>>>> >>>>> David Lang >>>>> >>>>> On Fri, 16 Jul 2021, Nathan Owens wrote: >>>>> >>>>>> Elon said "foolish packet routing" for things over 20ms! Which seems >>>>> crazy >>>>>> if you do some basic math: >>>>>> >>>>>> - Sat to User Terminal distance: 550-950km air/vacuum: 1.9 - 3.3ms >>>>>> - Sat to GW distance: 550-950km air/vacuum: 1.9 - 3.3ms >>>>>> - GW to PoP Distance: 50-800km fiber: 0.25 - 4ms >>>>>> - PoP to Internet Distance: 50km fiber: 0.25 - 0.5ms >>>>>> - Total one-way delay: 4.3 - 11.1ms >>>>>> - Theoretical minimum RTT: 8.6ms - 22.2ms, call it 15.4ms >>>>>> >>>>>> This includes no transmission delay, queuing delay, >>>>>> processing/fragmentation/reassembly/etc, and no time-division >>>>> multiplexing. >>>>>> >>>>>> On Fri, Jul 16, 2021 at 10:09 AM David Lang wrote: >>>>>> >>>>>>> I think it depends on if you are looking at datacenter-to-datacenter >>>>>>> latency of >>>>>>> home to remote datacenter latency :-) >>>>>>> >>>>>>> my rule of thumb for cross US ping time has been 80-100ms latency >>>>> (but >>>>>>> it's been >>>>>>> a few years since I tested it). >>>>>>> >>>>>>> I note that an article I saw today said that Elon is saying that >>>>> latency >>>>>>> will >>>>>>> improve significantly in the near future, that up/down latency is >>>>> ~20ms >>>>>>> and the >>>>>>> additional delays pushing it to the 80ms range are 'stupid packet >>>>> routing' >>>>>>> problems that they are working on. >>>>>>> >>>>>>> If they are still in that level of optimization, it doesn't surprise >>>>> me >>>>>>> that >>>>>>> they haven't really focused on the bufferbloat issue, they have more >>>>>>> obvious >>>>>>> stuff to fix first. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>> On Fri, 16 Jul 2021, Wheelock, Ian wrote: >>>>>>> >>>>>>>> Date: Fri, 16 Jul 2021 10:21:52 +0000 >>>>>>>> From: "Wheelock, Ian" >>>>>>>> To: David Lang , David P. Reed >>>>>>>> Cc: "starlink@lists.bufferbloat.net" < >>>>> starlink@lists.bufferbloat.net> >>>>>>>> Subject: Re: [Starlink] Starlink and bufferbloat status? >>>>>>>> >>>>>>>> Hi David >>>>>>>> In terms of the Latency that David (Reed) mentioned for California >>>>> to >>>>>>> Massachusetts of about 17ms over the public internet, it seems a bit >>>>> faster >>>>>>> than what I would expect. My own traceroute via my VDSL link shows >>>>> 14ms >>>>>>> just to get out of the operator network. >>>>>>>> >>>>>>>> https://www.wondernetwork.com is a handy tool for checking >>>>> geographic >>>>>>> ping perf between cities, and it shows a min of about 66ms for pings >>>>>>> between Boston and San Diego >>>>>>> https://wondernetwork.com/pings/boston/San%20Diego (so about 33ms >>>>> for >>>>>>> 1-way transfer). >>>>>>>> >>>>>>>> Distance wise this is about 4,100 KM (2,500 M), and @2/3 speed of >>>>> light >>>>>>> (through a pure fibre link of that distance) the propagation time is >>>>> just >>>>>>> over 20ms. If the network equipment between the Boston and San Diego >>>>> is >>>>>>> factored in, with some buffering along the way, 33ms does seem quite >>>>>>> reasonable over the 20ms for speed of light in fibre for that 1-way >>>>> transfer >>>>>>>> >>>>>>>> -Ian Wheelock >>>>>>>> >>>>>>>> From: Starlink on behalf >>>>> of >>>>>>> David Lang >>>>>>>> Date: Friday 9 July 2021 at 23:59 >>>>>>>> To: "David P. Reed" >>>>>>>> Cc: "starlink@lists.bufferbloat.net" < >>>>> starlink@lists.bufferbloat.net> >>>>>>>> Subject: Re: [Starlink] Starlink and bufferbloat status? >>>>>>>> >>>>>>>> IIRC, the definition of 'low latency' for the FCC was something like >>>>>>> 100ms, and Musk was predicting <40ms. roughly competitive with >>>>> landlines, >>>>>>> and worlds better than geostationary satellite (and many >>>>>>>> External (mailto:david@lang.hm) >>>>>>>> >>>>>>> >>>>> https://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL2lhbi53aGVlbG9ja0Bjb21tc2NvcGUuY29tL2I1MzFjZDA4OTZmMWI0Yzc5NzdiOTIzNmY3MTAzM2MxLzE2MjU4NzE1NDkuNjU=#key=19e8545676e28e577c813de83a4cf1dc >>>>>>> https://www.inky.com/banner-faq/ https://www.inky.com >>>>>>>> >>>>>>>> IIRC, the definition of 'low latency' for the FCC was something like >>>>>>> 100ms, and >>>>>>>> Musk was predicting <40ms. >>>>>>>> >>>>>>>> roughly competitive with landlines, and worlds better than >>>>> geostationary >>>>>>>> satellite (and many wireless ISPs) >>>>>>>> >>>>>>>> but when doing any serious testing of latency, you need to be wired >>>>> to >>>>>>> the >>>>>>>> router, wifi introduces so much variability that it swamps the >>>>> signal. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> On Fri, 9 Jul 2021, David P. Reed wrote: >>>>>>>> >>>>>>>>> Date: Fri, 9 Jul 2021 14:40:01 -0400 (EDT) >>>>>>>>> From: David P. Reed >>>>>>>>> To: starlink@lists.bufferbloat.net >>>>>>>>> Subject: [Starlink] Starlink and bufferbloat status? >>>>>>>>> >>>>>>>>> >>>>>>>>> Early measurements of performance of Starlink have shown >>>>> significant >>>>>>> bufferbloat, as Dave Taht has shown. >>>>>>>>> >>>>>>>>> But... Starlink is a moving target. The bufferbloat isn't a >>>>> hardware >>>>>>> issue, it should be completely manageable, starting by simple >>>>> firmware >>>>>>> changes inside the Starlink system itself. For example, implementing >>>>>>> fq_codel so that bottleneck links just drop packets according to the >>>>> Best >>>>>>> Practices RFC, >>>>>>>>> >>>>>>>>> So I'm hoping this has improved since Dave's measurements. How >>>>> much has >>>>>>> it improved? What's the current maximum packet latency under full >>>>>>> load, Ive heard anecdotally that a friend of a friend gets 84 msec. >>>>> *ping >>>>>>> times under full load*, but he wasn't using flent or some other >>>>> measurement >>>>>>> tool of good quality that gives a true number. >>>>>>>>> >>>>>>>>> 84 msec is not great - it's marginal for Zoom quality experience >>>>> (you >>>>>>> want latencies significantly less than 100 msec. as a rule of thumb >>>>> for >>>>>>> teleconferencing quality). But it is better than Dave's measurements >>>>> showed. >>>>>>>>> >>>>>>>>> Now Musk bragged that his network was "low latency" unlike other >>>>> high >>>>>>> speed services, which means low end-to-end latency. That got him >>>>>>> permission from the FCC to operate Starlink at all. His number was, I >>>>>>> think, < 5 msec. 84 is a lot more than 5. (I didn't believe 5, >>>>> because he >>>>>>> probably meant just the time from the ground station to the terminal >>>>>>> through the satellite. But I regularly get 17 msec. between >>>>> California and >>>>>>> Massachusetts over the public Internet) >>>>>>>>> >>>>>>>>> So 84 might be the current status. That would mean that someone at >>>>>>> Srarlink might be paying some attention, but it is a long way from >>>>> what >>>>>>> Musk implied. >>>>>>>>> >>>>>>>>> >>>>>>>>> PS: I forget the number of the RFC, but the number of packets >>>>> queued on >>>>>>> an egress link should be chosen by taking the hardware bottleneck >>>>>>> throughput of any path, combined with an end-to-end Internet >>>>> underlying >>>>>>> delay of about 10 msec. to account for hops between source and >>>>> destination. >>>>>>> Lets say Starlink allocates 50 Mb/sec to each customer, packets are >>>>> limited >>>>>>> to 10,000 bits (1500 * 8), so the outbound queues should be limited >>>>> to >>>>>>> about 0.01 * 50,000,000 / 10,000, which comes out to about 250 >>>>> packets from >>>>>>> each terminal of buffering, total, in the path from terminal to >>>>> public >>>>>>> Internet, assuming the connection to the public Internet is not a >>>>> problem. >>>>>>>> _______________________________________________ >>>>>>>> Starlink mailing list >>>>>>>> Starlink@lists.bufferbloat.net >>>>>>>> >>>>>>> >>>>> https://secure-web.cisco.com/1sNc_-1HhGCW7xdirt_lAoAy5Nn5T6UA85Scjn5BR7QHXtumhrf6RKn78SuRJG7DUKI3duggU9g6hJKW-Ze07HTczYqB9mBpIeALqk5drQ7nMvM8K7JbWfUbPR7JSNrI75UjiNXQk0wslBfoOTvkMlRj5eMOZhps7DMGBRQTVAeTd5vwXoQtDgS6zLCcJkrcO2S9MRSCC4f1I17SzgQJIwqo3LEwuN6lD-pkX0MFLqGr2zzsHw5eapd-VBlHu5reC4-OEn2zHkb7HNzS1pcueF6tsUE1vFRsWs2SIOwU5MvbKe3J3Q6NRQ40cHI1AGd-i/https://lists.bufferbloat.net/listinfo/starlink >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>>> >>>>>> >>>>> >>>> _______________________________________________ >>>> Starlink mailing list >>>> Starlink@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/starlink >>>> >>>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/starlink >>> >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink >> > --===============4865789427217445894== Content-Type: text/plain; CHARSET=utf-8 Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: INLINE X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KU3Rhcmxpbmsg bWFpbGluZyBsaXN0ClN0YXJsaW5rQGxpc3RzLmJ1ZmZlcmJsb2F0Lm5ldApodHRwczovL2xpc3Rz LmJ1ZmZlcmJsb2F0Lm5ldC9saXN0aW5mby9zdGFybGluawo= --===============4865789427217445894==--