Re: [Starlink] Starlink and bufferbloat status?

[Nathan Owens <nathan@nathan.io>]

[-- Attachment #1: Type: text/plain, Size: 16497 bytes --]

Check out the NASA TBIRD mission coming up, 2x 100GbE coherent optics and
an EDFA, beaming back to a 12in telescope on earth — hopefully it works!
Basically all COTS components.

On Fri, Jul 16, 2021 at 3:40 PM Jeremy Austin <jeremy@aterlo.com> wrote:

> I agree that RF is constrained in total capacity compared to optical
> frequencies. At the risk of showing my ignorance by quoting from Wikipedia,
> “Atmospheric and fog attenuation, which are exponential in nature, limit
> practical range of FSO devices to several kilometres.”
>
> Is there a safe and legal FSO mechanism that works over these distances
> through atmosphere shell-to-ground? And the power requirements suitable for
> a StarLink-sized single, solar-powered system?
>
> Optics through a vacuum (or near vacuum) are a totally different story.
> Intra-satellite links make perfect sense once the cost comes down.
>
> Willing to be corrected but skeptical of the sky-to-ground link budget,
>
> Jeremy Austin
>
> On Fri, Jul 16, 2021 at 1:40 PM Mike Puchol <mike@starlink.sx> wrote:
>
>> If we understand shell as “group of satellites at a certain altitude
>> range”, there is not much point in linking between shells if you can link
>> within one shell and orbital plane, and that plane has at least one
>> satellite within range of a gateway. I could be proven wrong, but IMHO the
>> first generation of links are meant of intra-plane, and maybe at a stretch
>> cross-plane to the next plane East or West.
>>
>> The only way to eventually go is optical links to the ground too, as RF
>> will only get you so far. At that stage, every shell will have its own
>> optical links to the ground, with gateways placed in areas with little
>> average cloud cover.
>>
>> Best,
>>
>> Mike
>> On Jul 16, 2021, 23:30 +0200, David Lang <david@lang.hm>, wrote:
>>
>> at satellite distances, you need to adjust your vertical direction
>> depending on
>> how far away the satellite you are talking to is, even if it's at the same
>> altitude
>>
>> the difference between shells that are only a few KM apart is less than
>> the
>> angles that you could need to satellites in the same shell further away.
>>
>> David Lang
>>
>> On Fri, 16 Jul 2021, Mike Puchol wrote:
>>
>> Date: Fri, 16 Jul 2021 22:57:14 +0200
>> From: Mike Puchol <mike@starlink.sx>
>> To: David Lang <david@lang.hm>
>> Cc: Nathan Owens <nathan@nathan.io>,
>> "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>,
>> David P. Reed <dpreed@deepplum.com>
>> Subject: Re: [Starlink] Starlink and bufferbloat status?
>>
>> Correct. A mirror tracking head that turns around the perpendicular to
>> the satellite path allows you to track satellites in the same plane, in
>> front or behind, when they change altitude by a few kilometers as part of
>> orbital adjustments or collision avoidance. To have a fully gimbaled head
>> that can track any satellite in any direction (and at any relative
>> velocity!) is a totally different problem. I could see satellites linked to
>> the next longitudinal plane apart from those on the same plane, but
>> cross-plane when one is ascending and the other descending is way harder.
>> The next shells will be at lower altitudes, around 300-350km, and they have
>> also stated they want to go for higher shells at 1000+ km.
>>
>> Best,
>>
>> Mike
>> On Jul 16, 2021, 20:48 +0200, David Lang <david@lang.hm>, wrote:
>>
>> I expect the lasers to have 2d gimbles, which lets them track most things
>> in
>> their field of view. Remember that Starlink has compressed their orbital
>> planes,
>> they are going to be running almost everything in the 550km range
>> (500-600km
>> IIRC) and have almost entirely eliminated the ~1000km planes
>>
>> David Lang
>>
>> On Fri, 16 Jul 2021,
>> Mike Puchol wrote:
>>
>> Date: Fri, 16 Jul 2021 19:42:55 +0200
>> From: Mike Puchol <mike@starlink.sx>
>> To: David Lang <david@lang.hm>
>> Cc: Nathan Owens <nathan@nathan.io>,
>> "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>,
>> David P. Reed <dpreed@deepplum.com>
>> Subject: Re: [Starlink] Starlink and bufferbloat status?
>>
>> True, but we are then assuming that the optical links are a mesh between
>> satellites in the same plane, plus between planes. From an engineering
>> problem point of view, keeping optical links in-plane only makes the system
>> extremely simpler (no full FOV gimbals with the optical train in them, for
>> example), and it solves the issue, as it is highly likely that at least one
>> satellite in any given plane will be within reach of a gateway.
>>
>> Routing to an arbitrary gateway may involve passing via intermediate
>> gateways, ground segments, and even using terminals as a hopping point.
>>
>> Best,
>>
>> Mike
>> On Jul 16, 2021, 19:38 +0200, David Lang <david@lang.hm>, wrote:
>>
>> the speed of light in a vaccum is significantly better than the speed of
>> light
>> in fiber, so if you are doing a cross country hop, terminal -> sat -> sat
>> -> sat
>> -> ground station (especially if the ground station is in the target
>> datacenter)
>> can be faster than terminal -> sat -> ground station -> cross-country
>> fiber,
>> even accounting for the longer distance at 550km altitude than at ground
>> level.
>>
>> This has interesting implications for supplementing/replacing undersea
>> cables as
>> the sats over the ocean are not going to be heavily used, dedicated ground
>> stations could be setup that use sats further offshore than normal (and
>> are
>> shielded from sats over land) to leverage the system without interfering
>> significantly with more 'traditional' uses
>>
>> David Lang
>>
>> On Fri, 16 Jul 2021, Mike Puchol wrote:
>>
>> Date: Fri, 16 Jul 2021 19:31:37 +0200
>> From: Mike Puchol <mike@starlink.sx>
>> To: David Lang <david@lang.hm>, Nathan Owens <nathan@nathan.io>
>> Cc: "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>,
>> David P. Reed <dpreed@deepplum.com>
>> Subject: Re: [Starlink] Starlink and bufferbloat status?
>>
>> Satellite optical links are useful to extend coverage to areas where you
>> don’t have gateways - thus, they will introduce additional latency compared
>> to two space segment hops (terminal to satellite -> satellite to gateway).
>> If you have terminal to satellite, two optical hops, then final satellite
>> to gateway, you will have more latency, not less.
>>
>> We are being “sold” optical links for what they are not IMHO.
>>
>> Best,
>>
>> Mike
>> On Jul 16, 2021, 19:29 +0200, Nathan Owens <nathan@nathan.io>, wrote:
>>
>> As there are more satellites, the up down time will get closer to 4-5ms
>> rather then the ~7ms you list
>>
>>
>> Possibly, if you do steering to always jump to the lowest latency
>> satellite.
>>
>> with laser relays in orbit, and terminal to terminal routing in orbit,
>> there is the potential for the theoretical minimum to tend lower
>>
>> Maybe for certain users really in the middle of nowhere, but I did the
>> best-case math for "bent pipe" in Seattle area, which is as good as it gets.
>>
>> On Fri, Jul 16, 2021 at 10:24 AM David Lang <david@lang.hm> wrote:
>>
>> hey, it's a good attitude to have :-)
>>
>> Elon tends to set 'impossible' goals, miss the timeline a bit, and come
>> very
>> close to the goal, if not exceed it.
>>
>> As there are more staellites, the up down time will get closer to 4-5ms
>> rather
>> then the ~7ms you list, and with laser relays in orbit, and terminal to
>> terminal
>> routing in orbit, there is the potential for the theoretical minimum to
>> tend
>> lower, giving some headroom for other overhead but still being in the 20ms
>> range.
>>
>> David Lang
>>
>>   On Fri, 16 Jul 2021, Nathan Owens wrote:
>>
>> Elon said "foolish packet routing" for things over 20ms! Which seems crazy
>> if you do some basic math:
>>
>>    - Sat to User Terminal distance: 550-950km air/vacuum: 1.9 - 3.3ms
>>    - Sat to GW distance: 550-950km air/vacuum: 1.9 - 3.3ms
>>    - GW to PoP Distance: 50-800km fiber: 0.25 - 4ms
>>    - PoP to Internet Distance: 50km fiber: 0.25 - 0.5ms
>>    - Total one-way delay: 4.3 - 11.1ms
>>    - Theoretical minimum RTT: 8.6ms - 22.2ms, call it 15.4ms
>>
>> This includes no transmission delay, queuing delay,
>> processing/fragmentation/reassembly/etc, and no time-division
>> multiplexing.
>>
>> On Fri, Jul 16, 2021 at 10:09 AM David Lang <david@lang.hm> wrote:
>>
>> I think it depends on if you are looking at datacenter-to-datacenter
>> latency of
>> home to remote datacenter latency :-)
>>
>> my rule of thumb for cross US ping time has been 80-100ms latency (but
>> it's been
>> a few years since I tested it).
>>
>> I note that an article I saw today said that Elon is saying that latency
>> will
>> improve significantly in the near future, that up/down latency is ~20ms
>> and the
>> additional delays pushing it to the 80ms range are 'stupid packet routing'
>> problems that they are working on.
>>
>> If they are still in that level of optimization, it doesn't surprise me
>> that
>> they haven't really focused on the bufferbloat issue, they have more
>> obvious
>> stuff to fix first.
>>
>> David Lang
>>
>>
>>    On Fri, 16 Jul 2021, Wheelock, Ian wrote:
>>
>> Date: Fri, 16 Jul 2021 10:21:52 +0000
>> From: "Wheelock, Ian" <ian.wheelock@commscope.com>
>> To: David Lang <david@lang.hm>, David P. Reed <dpreed@deepplum.com>
>> Cc: "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>
>> Subject: Re: [Starlink] Starlink and bufferbloat status?
>>
>> Hi David
>> In terms of the Latency that David (Reed) mentioned for California to
>>
>> Massachusetts of about 17ms over the public internet, it seems a bit
>> faster
>> than what I would expect. My own traceroute via my VDSL link shows 14ms
>> just to get out of the operator network.
>>
>>
>> https://www.wondernetwork.com  is a handy tool for checking geographic
>>
>> ping perf between cities, and it shows a min of about 66ms for pings
>> between Boston and San Diego
>> https://wondernetwork.com/pings/boston/San%20Diego (so about 33ms for
>> 1-way transfer).
>>
>>
>> Distance wise this is about 4,100 KM (2,500 M), and @2/3 speed of light
>>
>> (through a pure fibre link of that distance) the propagation time is just
>> over 20ms. If the network equipment between the Boston and San Diego is
>> factored in, with some buffering along the way, 33ms does seem quite
>> reasonable over the 20ms for speed of light in fibre for that 1-way
>> transfer
>>
>>
>> -Ian Wheelock
>>
>> From: Starlink <starlink-bounces@lists.bufferbloat.net> on behalf of
>>
>> David Lang <david@lang.hm>
>>
>> Date: Friday 9 July 2021 at 23:59
>> To: "David P. Reed" <dpreed@deepplum.com>
>> Cc: "starlink@lists.bufferbloat.net" <starlink@lists.bufferbloat.net>
>> Subject: Re: [Starlink] Starlink and bufferbloat status?
>>
>> IIRC, the definition of 'low latency' for the FCC was something like
>>
>> 100ms, and Musk was predicting <40ms. roughly competitive with landlines,
>> and worlds better than geostationary satellite (and many
>>
>> External (mailto:david@lang.hm)
>>
>>
>> https://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL2lhbi53aGVlbG9ja0Bjb21tc2NvcGUuY29tL2I1MzFjZDA4OTZmMWI0Yzc5NzdiOTIzNmY3MTAzM2MxLzE2MjU4NzE1NDkuNjU=#key=19e8545676e28e577c813de83a4cf1dc
>>   https://www.inky.com/banner-faq/  https://www.inky.com
>>
>>
>> IIRC, the definition of 'low latency' for the FCC was something like
>>
>> 100ms, and
>>
>> Musk was predicting <40ms.
>>
>> roughly competitive with landlines, and worlds better than geostationary
>> satellite (and many wireless ISPs)
>>
>> but when doing any serious testing of latency, you need to be wired to
>>
>> the
>>
>> router, wifi introduces so much variability that it swamps the signal.
>>
>> David Lang
>>
>> On Fri, 9 Jul 2021, David P. Reed wrote:
>>
>> Date: Fri, 9 Jul 2021 14:40:01 -0400 (EDT)
>> From: David P. Reed <dpreed@deepplum.com>
>> To: starlink@lists.bufferbloat.net
>> Subject: [Starlink] Starlink and bufferbloat status?
>>
>>
>> Early measurements of performance of Starlink have shown significant
>>
>> bufferbloat, as Dave Taht has shown.
>>
>>
>> But...  Starlink is a moving target. The bufferbloat isn't a hardware
>>
>> issue, it should be completely manageable, starting by simple firmware
>> changes inside the Starlink system itself. For example, implementing
>> fq_codel so that bottleneck links just drop packets according to the Best
>> Practices RFC,
>>
>>
>> So I'm hoping this has improved since Dave's measurements. How much has
>>
>> it improved? What's the current maximum packet latency under full
>> load,  Ive heard anecdotally that a friend of a friend gets 84 msec. *ping
>> times under full load*, but he wasn't using flent or some other
>> measurement
>> tool of good quality that gives a true number.
>>
>>
>> 84 msec is not great - it's marginal for Zoom quality experience (you
>>
>> want latencies significantly less than 100 msec. as a rule of thumb for
>> teleconferencing quality). But it is better than Dave's measurements
>> showed.
>>
>>
>> Now Musk bragged that his network was "low latency" unlike other high
>>
>> speed services, which means low end-to-end latency.  That got him
>> permission from the FCC to operate Starlink at all. His number was, I
>> think, < 5 msec. 84 is a lot more than 5. (I didn't believe 5, because he
>> probably meant just the time from the ground station to the terminal
>> through the satellite. But I regularly get 17 msec. between California and
>> Massachusetts over the public Internet)
>>
>>
>> So 84 might be the current status. That would mean that someone at
>>
>> Srarlink might be paying some attention, but it is a long way from what
>> Musk implied.
>>
>>
>>
>> PS: I forget the number of the RFC, but the number of packets queued on
>>
>> an egress link should be chosen by taking the hardware bottleneck
>> throughput of any path, combined with an end-to-end Internet underlying
>> delay of about 10 msec. to account for hops between source and
>> destination.
>> Lets say Starlink allocates 50 Mb/sec to each customer, packets are
>> limited
>> to 10,000 bits (1500 * 8), so the outbound queues should be limited to
>> about 0.01 * 50,000,000 / 10,000, which comes out to about 250 packets
>> from
>> each terminal of buffering, total, in the path from terminal to public
>> Internet, assuming the connection to the public Internet is not a problem.
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>>
>>
>> https://secure-web.cisco.com/1sNc_-1HhGCW7xdirt_lAoAy5Nn5T6UA85Scjn5BR7QHXtumhrf6RKn78SuRJG7DUKI3duggU9g6hJKW-Ze07HTczYqB9mBpIeALqk5drQ7nMvM8K7JbWfUbPR7JSNrI75UjiNXQk0wslBfoOTvkMlRj5eMOZhps7DMGBRQTVAeTd5vwXoQtDgS6zLCcJkrcO2S9MRSCC4f1I17SzgQJIwqo3LEwuN6lD-pkX0MFLqGr2zzsHw5eapd-VBlHu5reC4-OEn2zHkb7HNzS1pcueF6tsUE1vFRsWs2SIOwU5MvbKe3J3Q6NRQ40cHI1AGd-i/https://lists.bufferbloat.net/listinfo/starlink
>>
>>
>> _______________________________________________
>>
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>> _______________________________________________
>> Starlink mailing list
>> Starlink@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
> --
>
>
> <https://preseem.com/>
>   <https://www.facebook.com/preseem/> <https://twitter.com/preseem>
> <https://www.youtube.com/channel/UCLGfpuWwXcimzpxK3IIDgzg/videos>
> <https://www.linkedin.com/company/4287641/>
>      Jeremy Austin
>
>       Sr. Product Manager
>
>       *Preseem | Aterlo Networks*
>
>       Book a Call: https://app.hubspot.com/meetings/jeremy548
>
>
>
>
> 1 833 773 7336 x718 <1+833+773+7336+718>
> jeremy@preseem.com
> preseem.com
>          <https://preseem.com/stay-connected/>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>

[-- Attachment #2: Type: text/html, Size: 31951 bytes --]