[Starlink] Starlink hidden buffers

[Mark Handley]

On Wed, 24 May 2023, at 1:55 PM, Ulrich Speidel via Starlink wrote:
> 
> Dishy tracks most satellites for significantly less than 15 minutes, and for a relatively small part of their orbit. Let me explain:
> 
> 
> 
> 
> This is an obstruction map obtained with starlink-grpc-tools (https://github.com/sparky8512/starlink-grpc-tools). The way to read this is in polar coordinates: The centre of the image is the dishy boresight (direction of surface normal), distance from the centre is elevation measured as an angle from the surface normal, and direction from the centre is essentially the azimuth - top is north, left is west, bottom is south, and right is east. The white tracks are the satellites dishy uses, and a graph like this gets built up over time, one track at a time. Notice how short the tracks are - they don't follow the satellite for long - typically under a minute. The red bits are satellites getting obscured by the edge of our roof.
> 
> I've also attached a time lapse movie of how one of these graphs builds up - if I correctly remember (the script is on another machine), one frame in the video corresponds to 5 seconds.
> 
> Conclusion: latency change from tracking one satellite is smaller than the latency difference as you jump between satellites. You could be looking at several 100 km of path difference here. In an instant. Even that, at 300,000 km/s of propagation speed, is only in the order of maybe 1 ms or so - peanuts compared to the RTTs in the dozens of ms that we're seeing. But if you get thrown from one queue onto another as you get handed over - what does that do to the remote TCP stack that's serving you? 
> 

Interesting video.  From eyeballing it, it seems that when it changes satellite, it's most often changing between satellites that are a similar distance from boresight.  When it does this, the difference in propogation delay from dishy to satellite will be minimal.  It's possible it's even switching when the latency matches - I can't really tell from the video.  

Of course you can't tell from just one end of the connection whether starlink is switching satellite just when overall ground-to-ground path latency of the current path drops below the path latency of the next path.  For that we'd need to see what happened at the groundstation too.  But if you were trying to optimize things to minimize reordering, you might try something like this.  As you point out, you've still got variable uplink queue sizes to handle as you switch, but there's no fundamental reason why path switches *always* need to result in latency discontinuities.  

If you did decide to switch when the underlying path latency matches, and thinking more about those uplink queues: when you switch a path from a smaller uplink queue (at a groundstation) to a larger one, there's no reordering, so TCP should be happy(ish).  When switching from a larger uplink queue to a smaller one, you can cause reordering, but it's easy enough to hide by adding an earliest release time to any new packets (based on the last time a packet from that flow was (or will be) last sent on the old path), and not release the packets from the new queue to send to the satellite before that time.  I've no idea if anyone cares enough to implement such a scheme though.

Not saying any of this is what Starlink does - just idle speculation as to how you might minimize reordering if it was enough of a problem.  And of course I'm ignoring any queues in satellites...

Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230524/7deb16e0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obstructions_pos2_762.png
Type: image/png
Size: 1289 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230524/7deb16e0/attachment-0001.png>