[Starlink] Starlink hidden buffers

[Ulrich Speidel]

On 25/05/2023 3:18 am, Mark Handley wrote:
>
>
> On Wed, 24 May 2023, at 1:55 PM, Ulrich Speidel via Starlink wrote:
>>
>> Dishy tracks most satellites for significantly less than 15 minutes, 
>> and for a relatively small part of their orbit. Let me explain:
>>
>>
>>
>> This is an obstruction map obtained with starlink-grpc-tools 
>> (https://github.com/sparky8512/starlink-grpc-tools 
>> <https://github.com/sparky8512/starlink-grpc-tools>). 
>> The way to read this is in polar coordinates: The centre of the image 
>> is the dishy boresight (direction of surface normal), distance from 
>> the centre is elevation measured as an angle from the surface normal, 
>> and direction from the centre is essentially the azimuth - top is 
>> north, left is west, bottom is south, and right is east. The white 
>> tracks are the satellites dishy uses, and a graph like this gets 
>> built up over time, one track at a time. Notice how short the tracks 
>> are - they don't follow the satellite for long - typically under a 
>> minute. The red bits are satellites getting obscured by the edge of 
>> our roof.
>>
>> I've also attached a time lapse movie of how one of these graphs 
>> builds up - if I correctly remember (the script is on another 
>> machine), one frame in the video corresponds to 5 seconds.
>>
>> Conclusion: latency change from tracking one satellite is smaller 
>> than the latency difference as you jump between satellites. You could 
>> be looking at several 100 km of path difference here. In an instant. 
>> Even that, at 300,000 km/s of propagation speed, is only in the order 
>> of maybe 1 ms or so - peanuts compared to the RTTs in the dozens of 
>> ms that we're seeing. But if you get thrown from one queue onto 
>> another as you get handed over - what does that do to the remote TCP 
>> stack that's serving you?
>>
>
> Interesting video.  From eyeballing it, it seems that when it changes 
> satellite, it's most often changing between satellites that are a 
> similar distance from boresight.  When it does this, the difference in 
> propogation delay from dishy to satellite will be minimal.  It's 
> possible it's even switching when the latency matches - I can't really 
> tell from the video.
Qualified "maybe" here ... most of Starlink still runs on bent pipe 
topology, and we don't know how or why a particular satellite is chosen, 
of for that matter where that choice is made. The video was produced in 
Auckland, within relatively close proximity (23.15 km) to Starlink's 
Clevedon ground station. So there would have been quite a few satellites 
to choose from that were in sight of both ends. Also, on our deck (where 
the measurement was taken), there are obstructions in pretty much all 
directions on the lower horizon. That's not necessarily the situation 
you'd get on the ridgeline of a farmhouse roof 300 km away from a 
gateway. So that "similar distance from boresight" might be a location 
artefact.
>
> Of course you can't tell from just one end of the connection whether 
> starlink is switching satellite just when overall ground-to-ground 
> path latency of the current path drops below the path latency of the 
> next path.  For that we'd need to see what happened at the 
> groundstation too.  But if you were trying to optimize things to 
> minimize reordering, you might try something like this.  As you point 
> out, you've still got variable uplink queue sizes to handle as you 
> switch, but there's no fundamental reason why path switches *always* 
> need to result in latency discontinuities.
Yes, although with slot assignments (which they can't really avoid I 
guess), satellite capacity would be the primary criterion I suppose. The 
effect of reordering is mostly that it drives up the amount of buffer 
memory needed for reassembly at the receiving end, which is not much of 
an issue nowadays with sufficient receiver socket memory. In this sort 
of scenario, delays from reordering to the application reading from the 
socket are no worse than delays from not switching until a bit later.
>
>
> If you did decide to switch when the underlying path latency matches, 
> and thinking more about those uplink queues: when you switch a path 
> from a smaller uplink queue (at a groundstation) to a larger one, 
> there's no reordering, so TCP should be happy(ish).  When switching 
> from a larger uplink queue to a smaller one, you can cause reordering, 
> but it's easy enough to hide by adding an earliest release time to any 
> new packets (based on the last time a packet from that flow was (or 
> will be) last sent on the old path), and not release the packets from 
> the new queue to send to the satellite before that time.  I've no idea 
> if anyone cares enough to implement such a scheme though.
Case in point: This discussion started because we were wondering why 
Starlink had so much buffer in the system. That adding of earliest 
release time means that you are buffering, so it'd be exactly the thing 
that started this mailing list!
>
> Not saying any of this is what Starlink does - just idle speculation 
> as to how you might minimize reordering if it was enough of a 
> problem.  And of course I'm ignoring any queues in satellites...

We know that we're seeing RTTs into the hundreds of ms in scenarios 
where we have physical path latencies of at most a couple of dozen ms. 
So, yes, speculation, but ...

Also, I don't get the impression that path latency minimisation is top 
priority for Starlink. My impression is that as long as RTT is what you 
might see on a terrestrial connection to the other side of the globe, 
it's good enough for Starlink.

Cheers,

Ulrich

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel at auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/b3163ada/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obstructions_pos2_762.png
Type: image/png
Size: 1289 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/b3163ada/attachment-0001.png>