[Starlink] Starlink hidden buffers

[David Lang]

On Thu, 25 May 2023, Ulrich Speidel wrote:

> On 15/05/2023 3:33 pm, David Lang wrote:
>> On Mon, 15 May 2023, Ulrich Speidel wrote:
>> 
>> > On 14/05/2023 9:00 pm, David Lang wrote:
>> >> On Sun, 14 May 2023, Ulrich Speidel wrote:
>> >>
>> >> >> I just discovered that someone is manufacturing an adapter so you no
>> >> >> longer have to cut the cable
>> >> >>
>> >> >>
>> >> 
>> https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P 
>> <https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P>
>> 
>> >> >>
>> >> > I'll see whether I can get hold of one of these. Cutting a cable on a
>> >> > university IT asset as an academic is not allowed here, except if it
>> >> > doesn't meet electrical safety standards.
> OK, we have one on order, along with PoE injector and power supply. Don't 
> hold your breath, though, I'll be out of the country when it arrives and 
> it'll be late July before I get to play with it.

I've got a couple on order, but they won't arrive for 1-3 more weeks :-(

>> >> >> But remember that the system does know how much usage there is in the 
>> >> >> cell before they do the handoff. It's unknown if they do anything with 
>> >> >> that, or if they are just relaying based on geography. We also don't 
>> >> >> know what the bandwidth to the ground stations is compared to the 
>> >> >> dishy.
>> >>
>> >> > Well, we do know for NZ, sort of, based on the licences Starlink has 
>> >> > here.
>> >>
>> >> what is the ground station bandwith?
>> I was asking more in terms of Gb/s rather than MHz of bandwidth. Dedicated
>> ground stations with bigger antennas, better filters, more processing and
>> overall a much higher budget can get much better data rates out of a given
>> amount of bandwidth than the user end stations will.
>> 
>> it's also possible (especially with bigger antennas) for one ground station 
>> location to talk to multiple different satellites at once (the aiming of the 
>> antennas can isolate the signals from each other)
>
> Well, the Gb/s is what a link budget would give you if you knew the 
> modulation scheme(s) and any FEC used. The ground station antennas are normal 
> parabolic dishes in radomes for all we can tell, and are all the same size, 
> so you can kind of estimate aperture and hence gain reasonably well. Path 
> loss depends a little on distance and path quality (weather / rain fade), and 
> we don't really know in how far their modems use adaptive rates to cope with 
> this (my Ookla tests during Cyclone Gabrielle most certainly don't rule this 
> out - rates went down both ways during the thick of it). I guess we know 
> relatively little about the on-board phased array on the satellites (apart 
> from very loose bounds), which restricts our ability to say much about gain 
> there (and potential for spatial separation / re-use of beam frequencies). We 
> also don't know how Starlink manages its frequency allocation across its 
> ground stations.

I'll also note that in the last launch of the v2 mini satellites, they mentioned 
that those now supported E band backhaul to handle 4x the bandwidth of the 
earlier satellites

> It's certainly noticeable here that they seem to have sets of three grouped 
> together in a relatively compact geographical area (you could visit all NZ 
> North Island ground stations in a day by car from Auckland, Auckland traffic 
> notwithstanding, and at a stretch could do the same down south from Hinds to 
> Awarua if you manage to ignore the scenery, but getting from the southernmost 
> North Island ground station to the northernmost South Island one is basically 
> a two day drive plus ferry trip).

I lived in Wanganui for a few years, including one RV trip down the South 
Island. I know what you mean about needing to ignore the scenery :-)

>> 
>> >> As latency changes, figuring out if it's extra distance that must be 
>> >> traveled, or buffering is hard. does the latency stay roughly the same 
>> >> until the next satellite change? or does it taper off?
>> 
>> > Good question. You would expect step changes in physical latency between 
>> > satellites, but also gradual change related to satellite movement. Plus of 
>> > course any rubble thrown into any queue by something suddenly turning up on 
>> > that path. Don't forget that it's not just cells now, we're also talking 
>> > up- and downlink for the laser ISLs, at least in some places.
>> 
>> how far do the satellites move in 15 min and what effect would that have on 
>> latency (I would assume that most of the time, the satellites are switched to 
>> as they are getting nearer the two stations, so most of the time, I would 
>> expect a slight reduction in latency for ~7 min and then a slight increase 
>> for ~7 min, but I would not expect that this would be a large variation
>
> Dishy tracks most satellites for significantly less than 15 minutes, and for 
> a relatively small part of their orbit. Let me explain:

Ok, I thought I had heard they switched every 15 min, so it's every 5 min 
instead?

> Conclusion: latency change from tracking one satellite is smaller than the 
> latency difference as you jump between satellites. You could be looking at 
> several 100 km of path difference here. In an instant. Even that, at 300,000 
> km/s of propagation speed, is only in the order of maybe 1 ms or so - peanuts 
> compared to the RTTs in the dozens of ms that we're seeing. But if you get 
> thrown from one queue onto another as you get handed over - what does that do 
> to the remote TCP stack that's serving you?

yes, the point I thought that I was trying to make was that the latency change 
from satellite movement was not very significant

>> >> If it stays the same, I would suspect that you are actually hitting a 
>> >> different ground station and there is a VPN backhaul to your egress point 
>> >> to the regular Internet (which doesn't support mobile IP addresses) for 
>> >> that cycle. If it tapers off, then I could buy bufferbloat that gets 
>> >> resolved as TCP backs off.
>> >
>> > Yes, quite sorting out which part of your latency is what is the million
>> > dollar question here...
>> >
>> > We saw significant RTT changes here during the recent cyclone over periods 
>> > of several hours, and these came in steps (see below), with the initial 
>> > change being a downward one. Averages are over 60 pings (the time scale 
>> > isn't 100% true as we used "one ping, one second" timing) here.
>> >
>> >
>> > We're still not sure whether to attribute this to load change or ground 
>> > station changes. There were a lot of power outages, especially in 
>> > Auckland's lifestyle block belt, which teems with Starlink users, but all 
>> > three North Island ground stations were also in areas affected by power 
>> > outages (although the power companies concerned don't provide the level of 
>> > detail to establish whether they were affected). It's also not clear what, 
>> > if any, backup power arrangements they have). At ~25 ms, the step changes 
>> > in RTT are too large be the result of a switch in ground stations, though, 
>> > the path differences just aren't that large. You'd also expect a ground 
>> > station outage to result in longer RTTs, not shorter ones, if you need to 
>> > re-route via another ground station. One explanation might be users getting 
>> > cut off if they relied on one particular ground station for bent pipe ops - 
>> > but that would not explain this order of magnitude effect as I'd expect 
>> > that number to be small. So maybe power outages at the user end after all. 
>> > But that would then tell us that these are load-dependent queuing delays. 
>> > Moreover, since those load changes wouldn't have involved the router at our 
>> > site, we can conclude that these are queue sojourn times in the Starlink 
>> > network.

remember that SpaceX controlls the ground stations as well, so if they are doing 
any mobile IP trickery to redirect traffic from one ground station to another, 
they can anticipate the shift or move the queue for the user or other trickery 
like this (probably aren't yet, they seem to be in the early days here, focusing 
on keeping things working and improving on the space side more than anything 
else)

>> I have two starlink dishes in the southern california area, I'm going to put 
>> one on the low-priority mobile plan shortly. These are primarily used for 
>> backup communication, so I would be happy to add something to them to do 
>> latency monitoring. In looking at what geo-location reports my location as, I 
>> see it wander up and down the west coast, from the Los Angeles area all the 
>> way up to Canada.

> Would be worthwhile to also do traceroutes to various places to see where you 
> emerge from the satellite side of things.
>> 
>> >> I think that active queue management on the sending side of the bottleneck 
>> >> will handle it fairly well. It doesn't have to do calculations based on 
>> >> what the bandwidth is, it just needs to know what it has pending to go 
>> >> out.
>> 
>> > Understood - but your customer for AQM is the sending TCP client, and there 
>> > are two questions here: (a) Does your AQM handle rapid load changes and (b) 
>> > how do your TCP clients actually respond to your AQM's handling?
>> 
>> AQM allocates the available bandwidth between different connections (usually 
>> different users)
> But it does this under the assumption that the vector for changes in bandwidth 
> availability is the incoming traffic, which AQM gives (indirect) feedback to, 
> right?

no, this is what I'm getting at below

>> When it does this indirectly for inbound traffic by delaying acks, the 
>> results depend on the senders handling of these indirect signals that were 
>> never intended for this purpose.

This is what you are thinking of, where it's providing indirect feedback to an 
unknowable inbound queue on a remote system

>> But when it does this directly on the sending side, it doesn't matter what 
>> the senders want, their data WILL be managed to the priority/bandwidth that 
>> the AQM sets, and eventually their feedback is dropped packets, which 
>> everyone who is legitimate responds to.

when the AQM in on the sending side of the bottleneck, it now has direct control 
over the queue, and potentially has information over the available bandwidth as 
it changes. But even if it doesn't know what the available bandwidth is, it 
still can dispatch the data in it's queues 'fairly' (whatever that means to the 
particulat AQM algorithm), changes in the data rate just change how fast the 
queue drains.

> Understood. You build a control loop, where the latency is the delay in the 
> control signal.
>
> Classically, you have a physical bottleneck that the AQM manages, where the 
> physical bandwidth doesn't change.
>
> The available bandwidth changes, (mostly) as a result of TCP connections (or 
> similarly behaved UDP applications) joining in slow start, or disappearing.
>
> Basically, your queues grow and shrink one packet at a time.
>
> Your control signal allows you (if they're well behaved) throttle / 
> accelerate senders.
>
> What you don't get are quantum jumps in queue occupancy, jump changes in 
> underlying physical bandwidth, or a whole set of new senders that are 
> completely oblivious to any of your previous control signals. But you get all 
> that with satellite handovers like these.

for a single TCP session,it has slow-start, but if you suddently start dozens or 
hundreds of TCP sessions, (bittorrent, other file transfer protocols, or just a 
website with hundreds of sub-elements), I think it's a bigger step than you are 
thinking.

And again, I think the same issue exists on cell sites as users move from one 
cell to another.

> So what if the response you elicit in this way is to a queue scenario that no 
> longer applies?

you run the risk of under-utilizing the link for a short time (which may mean 
that you decide to run the queues a little bigger than with fixed links, so that 
when a chunk of data disappears from your queue, you still will keep utilization 
up, sacraficing some latency to improve overall throughput)

David Lang
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obs_map20-2-23.mp4
Type: video/mp4
Size: 169217 bytes
Desc: 
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230524/d0fc694c/attachment-0001.mp4>